2024-12-03

This Sunday I was sick at home so I didn't get to write a blog post. Luckily I found some time today (Tuesday).

Some older changesets uploaded now are about comments in the line assembler source texts.

The only other changes are to introduce an option that causes an error when a resident debugger (application mode after TSR command or device mode before ATTACH command) tries to do program-loading L, and documenting the change in the news-r9 section of the manual. This is needed because specifying a flat binary file on the device debugger command line would lead to a crash as it overwrote a random address. An earlier changeset (2024-02-21) already prohibited loading executables (.COM or .EXE files) when resident.

These changes were mostly applied to all our TSRs, that is RxANSI, SEEKEXT, lClock, FDAPM's IDLEDPMS, KEEPHOOK, D2FCBFIX, and the TSR example (simply called "tsr").

• Update attribution
• Move reads of the resident size into a function to simplify further patches
• Allow a near jump in the uninstaller code, and optimise one jump to be very short
• Display the multiplex number upon successful removal of a resident TSR.
• Support process protection choices.
• RxANSI only: Fix a warning promoted to error for a byte sized relocation. This is actually valid in this case.
• KEEPHOOK only: Fix a bug on nullified critical install error (int 2Dh hook fails after successful simulation) because the code expected a pointer to the interrupt list but didn't get one.
• TSR example only: Update attribution in manual.

During my work on MS-DOS v4's SHARE I noticed that the sharer may walk the MCB chain in order to find processes. It needs this because there are several functions that are to close files found in the System File Tables (SFTs), but SFT entries do not refer back to the Process Handle Tables (PHTs) in which they are referenced. So the sharer needs to enumerate processes and close all the Process Handles referencing a particular SFT.

(This is still somewhat error prone because an application could expect a Process Handle to still be valid if it didn't close it on its own. So it could open a new file and the new Process Handle used could unexpectedly equal the one used for the file that was closed by the sharer.)

The problem is that the sharer expects any MCB that is self-owned (the MCB owner field is equal to the segment of the MCB itself plus one) to contain a process, an assumption which I famously decided to break in my TSRs. Therefore, in the update I provide new switches to choose a form of "process protection" so that SHARE (and other interested parties) will be less likely to misdetect the TSR as a process when there isn't a process there.

Unfortunately, this also makes other programs (like MEM) less likely to detect the program as a process, which we wanted to have in the first place. The different choices implemented so far are:

• /J0, no overhead, old style (unsafe) self-owned MCB without a process (still the default for now, but likely not for long)
• /J1, 1 paragraph overhead, double MCB. The real MCB refers to the fake MCB as its parent, and the fake MCB appears to be a self-owned (process) MCB with the resident program name. As the sharer doesn't follow the owner field of the real MCB it never detects the fake MCB as a process.
• /J2, 1 paragraph overhead, SD MCB. The real MCB is set up with owner = 8 and name = "SD", and the fake MCB after it is set up as an SD sub-MCB with type = "D" and the resident program name. This could lead to someone misdetecting us as a device, but this seems less likely. MEM may still show the "device" name as desired. Another disadvantage is that someone could try to merge the SD MCB with a prior or subsequent SD MCB, invalidating our expectation that freeing this SD MCB will free exactly and only our resident program.
• /J3, 6 paragraphs overhead, pseudo PSP. The real MCB is self-owned, but we construct a passable process in the first 6 paragraphs. During its construction we re-initialise the PHT pointer, PHT count, and sharer "next PSP" links to the defaults, and all PHT entries to -1 (closed).
• /J4, 6 paragraphs overhead, "true" PSP. Similar to /J3 but the PSP is created using int 21h service 55h. After its creation, service 50h is used to change the active PSP back to the installer's (possibly relocated) PSP. Also, all PHT entries of the created PSP are poked into the PHT of the installer's PSP and closed one by one (using service 3Eh).

DEVLOAD is affected by the same problem, as it currently creates a single self-owned MCB for the device driver that it loads.

Ignore a section directive with "NASM port directive" comment. This is used by the msbio port to indicate to fixmem.pl that some variables live in a certain section for assumptions, but the section directive is commented out (using NASM %if 0) and doesn't actually create any section in the executable – and consequently, in the map file. Therefore convlist.pl should also ignore this directive.

The other change is to drop the UTF-8 encoding previously used. By defaulting to perl's latin-1, all inputs in the listing files are allowed. In practice this is easier to work with than requiring UTF-8. (TracList may require UTF-8 however. In that case iconv must be used.)

Several significant changes to ident86 happened. Short ones only are:

• A magic fix to getting start below zero. Probably wrong but doesn't really hurt.
• If relocation count is mismatched, print rather than raise. This is after a much earlier change (2024-07-12) that changed the comparison of the exact relocations from an assert to a print.
• In the message displayed for relocation count mismatch display the counts for both files.
• Add some debugging output.
• Fix a bogus trailer length calculation. This is the first time that the calculation of the trailer (part of one file where the other already is at EOF) correctly does what seems to have been intended. The error was introduced during the very earliest set of changes (2024-07-09).
• Add trailing bytes to different bytes, and observe -M (but not -m).
• Add trailing bytes to different lines.
• Harden directive ranges calculation and add more debugging output.
• Fix some indentation in the .tls directives loop.
• Move inversion of map ranges to later in the main function.
• Add some more debugging output.
• Observe the header size in map ranges. (Wrong for non-MZ executables? No, for those the header size is detected as zero.)
• Recognise push and pop tls source file directives if they occur with a NASM source listing level indicator.
• Open .tls file in latin-1 encoding to be consistent and allow any byte values.

The script newly lists relocations that only occur in one of the two files.

A new switch is added, -Z, which specifies a relocation segment to apply to the MZ executable image. This helps ident86 to detect missing relocations that may otherwise not result in any detectable differences. In particular, the DD directive with a label to a DW directive with the label, and a second time with SEG label.

The relocation is applied to the file buffers after reading. To facilitate this, the data is converted to bytearray variables. The relocations are also applied when handleranges needs to reload its disassembly buffers. The relocations are not applied to trailing data.

In a subsequent changeset, it is checked that none of the relocations overlap, as this may invalidate the carry calculations for the relocations. There shouldn't be a practical need for overlapping relocations.

To help the relocations being applied to disassembly buffers, the carry in to the relocating function is optional. Because relocations don't overlap, the carry for the first needed byte can be obtained by reading one byte more than needed at the beginning. This is done by the disassembly buffer reloading code. This byte is discarded after applying relocations.

The -I switch allows to specify include paths. This is used for edits to find their respective files, making obsolete the fiddling with symlinks to make all files reachable.

Two relevant changes: First, PROC FAR and PROC NEAR can be detected as two separate labels now.

Second, duplicate labels handling. If the same label occurs multiple times in a file, it is recorded how often the label has been seen. The search of the source file will likewise decrement a counter if a duplicate label is to be matched.

Further, the switch -x is added to always make findsourceline start its second (complete) run immediately. This seemed to be needed for some runs to succeed.

• Replace double diamond operator by manual open and readline calls
• Replace bareword filehandles by variables.
• Introduce an array of files. Push the outermost file's info into it.
• Follow include directives instead of appending files to the list to be processed afterwards. This preserves the exact source order of the files for trivial cases. This involves changing the file handles used for input and output when entering and exiting an include file.
• Drop unneeded variable used for repetition.
• Record istruc base label section, avoiding a rerun difference.
• Accept a debug flag for getpossibleequates as the last parameter.
• Record sections for different label types, with colon, LABEL, or PROC directive.
• Add more debugging output with $debuglevel
• For equates check whether not a label by getpossibleequates indicating an equate.
• Detect trailing backslashes only in comments.
• Fix structure members being detected with name DB, DW, or DD and size unknown if a member is not named with a label.
• Handle istruc instance with initialiser and an equate for the DUP amount
• Fix trailing backslash detection after prior change.
• Avoid an error if file not found.
• Display our own error if file not found.
• Normalise section names to allcaps.
• Add line numbers to file info and add --debug-sections switch. The line numbers are of course approximate as sometimes one source line is expanded to multiple destination lines.
• Do not access active sections array if there is no active section, do not display misleading message.
• Pop the active section array upon reading as input the output of a prior run, fixing reruns.
• Add the section for an equate that is defined to be equal to a label.
• Only read label section in prior changeset if it is defined.
• Fix labelsize detection of OFFSET keyword to be cap-insensitive.
• Add the switch --quotemacros to change macro parameters for certain mmacros to quoted strings. Also strips all angle brackets. This is needed for the MSG_SERVICES mmacro as it operates on keywords matching smacros that it uses.
• Add handling of structure member with a DUP for istruc instances.
• Unify detection of $strucdup for prior changeset.
• Allow istruc initialisers to be specified as multiple items for the same member if that member contains commas.
• Display error if istruc receives too many initialisers.
• Fix to variable use for prior changesets.
• Expand closing bracket followed by an opening bracket to a plus sign. This occurred with memory operands that had two a16 registers sometimes.
• Allow a lone dollar sign as a label in getpossibleeqates.
• Allow istruc instance without a label prefixed.
• For prior changeset don't display error for BREAK mmacro use with angle brackets.