2024-07-28

The last two weeks I improved ident86 and used it to identicalise the Enhanced DR-DOS drdos module (ported to JWasm) as well as a new port of MSDebug to NASM. This new port re-used the fixmem.pl script from the WarpLink port to NASM.

This repo contains a test case for assembling data in a segment that belongs to a group, created from the MSDebug sources. The NASM manual states that references to such data should default to use the group as a reference. But if the data is assembled using the old Microsoft Macro Assembler from the 2018 free software release of MS-DOS v2 then NASM's references to the data are by default addressed using the segment as a reference, not the group.

I used this test repo to report the problem to the NASM bugtracker.

I worked around this problem by porting debconst and debdata in the MSDebug sources first, so that they are assembled by NASM. In that case the references do work as expected.

This repo had fixmem.pl and nasm.mac added from the WarpLink repo. Both were enhanced to work with the old MASM sources of MSDebug. Some of the changes include:

• Replace SIZE keyword followed by structure name
• Replace IF and IF NOT directives
• Replace = expressions by equates
• Ignore subttl directive
• Allow for differing capitalisation in equate names
• Allow uncapitalised include directives
• Add a group macro
• Swap two-letter text bytes in word immediates
• Replace operator keywords in calculations
• Extend supported calculations to detect numeric or equate operands
• Add org macro
• Accept DG: keyword and translate it to a WRT clause
• Accept segment overide before BYTE PTR
• Add the addequate function to add aliases for differently-capitalised equates
• Call addequate if equate occurs in an equate
• Hide segment macros from trace listing file to avoid confusing convlist.pl
• Add macro to translate int 3 to int3
• Several fixes to not mix up linebreaks (1, 2, 3, 4)
• Detect wait and default as disallowed equate and label names, prepend dollar sign for NASM
• Allow BYTE PTR in parens
• Allow WRT term in parens
• Allow push or pop memory operand without a size
• Detect dot followed by a label
• Parse single operand with brackets but lacking size
• Call addequates more
• Fix mixup of all-caps equates or labels
• Make sure rerunning on the same file is allowed (sometimes passing the same file twice is useful)
• Fix so left shift operator in an operand isn't misdetected as a structure instance
• Fix operating on the same file twice in a row (this used to use an inferior file change detection based on the name, leading to an infinite loop)
• Delete size keywords for lds and les
• Fix so labels starting with db won't confuse the replacement of not, or, xor keywords that's only supposed to occur in expressions
• Accept BYTE PTR without brackets before a plain number, does not indicate memory access
• Use EXTRN directive with :abs size as an equate
• Fix misdetection of labels ending in db (require either colon or a blank to separate label from a db directive)

The final changes were just maintenance:

• Replace exe2bin by x2b2
• Build with NASM files instead of original
• Drop obsolete files
• Bump release number

Most of these changes were picked from the SvarDOS repo.

Pick all SvarDOS changes to the drdos module to port it to JWasm. Identicalised using ident86 at every step.

Added cfg.bat and ovr.bat to select or deselect compression of the drdos and drbio modules. Allows to optimise the edrpack build a little by disabling all original compression and dropping the code used to uncompress these modules.

Align the stack in biosinit.asm.

In DIR command show accumulated size of listed files.

Fix DIR /2 command if a directory entry has no date time stamp.

Add zerocomp tool that compresses a file without any framing data. Used for inicomp.

A bug was fixed thrice in linfo.eld and set.eld (1, 2) concerning the re-use of multi-purpose puts handlers. When re-using them, their downlink needs to be initialised anew to avoid the possibility of using a stale downlink (if the downlink was modified from the original extcall to puts_ext_done).

This bug was reported by a user via email.

Add support for Enhanced DR-DOS zerocomp as a compression method.

Fix a mistake in a size of the xchg instruction.

Add explanation that swapping operands for test and xchg instructions results in the same meaning of the instruction. Particularly double-register instructions may be encoded two different ways. In test an immediate operand always must come last but other than that operand order doesn't matter. (This I found out while improving ident86.)

In convlist.pl support hex dump continuation lines in NASM listings that do not fill the entire 40 columns of the dump area.

• In side by side mode do same or samesame replacement in the next line after re-sync
• Display error if empty range selected
• For -M switch accept leading plus sign to add to -m number
• Display counters of differences detected
• Allow to re-sync side by side view with multiple one-sided lines
• Add -o switch to change .tls offset (MSDebug needs -o -256, a negative number to the -o switch)
• Allow to swap operands of xchg or test if this yields a no difference (only relevant for reg,reg encodings)
• Delete size keyword within brackets
• Expand a16 byte displacements to unsigned words
• Delete a16 displacement equal to zero
• Allow segment override for a16 edits
• Put hg revision hash of ident86 script
• Add -v and -V switches (show only version and quit, or do not show version)
• Enable line buffering for use with tee
• Allow -v to work without filenames
• Drop empty initial line from disassembly result (did no harm but would have had to be handled specifically later)
• Repeat disassembly if malformed line detected (previously would just raise on an assert, aborting the run)
• Limit repetition of disassembly

Introduce fuzzy logic comparison to mark lines that differ only in the immediate numbers within a certain range. To do this, all numbers behind the instruction mnemonic are stored into a list and replaced in the text by placeholders starting with a string of Zs and continuing with a string of Ys next for a second immediate. Then the text after replacement is compared with the other instruction. If they now match, next the numbers in the lists are compared. Their delta is calculated, fed to the absolute function, and then it is checked that the delta does not exceed 32. If all of this is matched then the lines are considered to be "fuzzy same". This is only used if side by side mode is used.

• Add display of earliest difference (first line that is not no difference, nor samesame, nor fuzzy same). Requires side by side mode
• -d switch to limit disassembly after earliest difference
• List earliest difference address at the very end
• Extract function swapoperands, and then call it during side by side view
• Extract function markearliest, call it also when re-syncing side by side view
• Display a hint of how many NOPs still expected but missing
• Record the seek offset of the best matching line in the trace listing (.tls) file
• When .tls source indicates that source used define data directives, display the data using the same directive rather than trying to disassemble
• Add support for WarpLink extended (/mx) .map files to detect alignment bytes and dump them using db directives rather than trying to disassemble
• Fix so that a shorter db range doesn't get expanded to the trace listing hex dump length
• Extract function handlenops and call it during side by side view
• Fix to correctly clear replace variable so a multi-line db command dump doesn't lead to repeating data
• Use fuzzy comparison in detectnops
• Note which file is missing a NOP in the detectnops hint
• Add a hint if one file contains a NOP and the other contains another instruction at this address
• Do not repeat disassembly on obviously bad match, which is a D command dump with Xs in a data element. This is a stopgap solution to ease debugging. It doesn't detect if an element other than the first starts with Xs.
• Round up ranges of dw and dd data items so no partial items are listed. (Changeset message incorrectly lists "db and dw".)
• Give a hint if a segment override mismatch is detected in a different line

Add -e switch. This can be set to 0 (default, as before) or 1 or 2. If nonzero, all the hints are worded so as to specify what edits are needed in the specified file to match the other file. We'll usually use -e 2 for now.

Add -c switch. This specifies a filename to use as a cookie file. On initialisation, if the file exists then the last line is read. This line specifies a minimum offset like for the -m switch. On finding a definite difference (not no difference, nor samesame, nor fuzzy same) the offset of the current range's start is appended to the cookie file, except if the file already exists and the last line matches the new line to write already.

The idea is to automatically skip content that is likely already identicalised after finding and fixing a definite difference. I used to do this manually to lower the time needed to run ident86. And when you do something robotic and simple manually, repeatedly, why not automate it?

The idea is to work on a file with the -c switch for step by step work, then run a full comparison without -c, -m, or -M to ensure the entire file is identicalised.

The hints are fairly specific for a number of differences already. It is possible to automatically read the hints, relate them to the trace listing, relate that to the original listing file, and then try to heuristically find the corresponding spot in the original source text file. This would allow to programmatically apply the edit to the source file without user intervention, then loop back to assembling and comparing the file again.