2024-05-05

This week I worked primarily on the newly released MS-DOS v4.00 sources, which are FLOSS except for the toolchain. The tools are included under the MIT license but as binaries only.

The devload help usage message was displayed twice, this is now fixed.

The partition table scanning code has been adjusted to work when included into an Extension for lDebug (ELD). Further, in the ELD build there exists a new variable to also call the partition handler for extended partitions (type 05h, 0Fh, or 85h) rather than only the logical partitions contained within them. (Extended partitions in the MBR's partition table do call the handler twice however.)

Do not store a trailing LF or NUL in the command line buffer if a device command line ends in one of these. Rather, overwrite them with a CR.

This is needed because MS-DOS v4.00 would terminate with an LF. Without the patch, the K and L commands would operate on a filename consisting of the LF.

This showed a different failure mode of the device driver debugger, not yet fixed: If a name with an unknown filename is specified (ie .BIN file handling) then the debugger will try to load this file to a random address. This is yet to be fixed. (In application mode the random address is always replaced by the code that creates a new empty process.)

Another bug in the device driver mode of the debugger was that the scan for an LDEBUGCONFIG variable would corrupt a random byte within memory addressed using a PSP segment. This bug happened to make it so that our MS-DOS v4 build didn't find any files any longer after the lDebug init ran. The same bug did not affect lDDebug in the same way; this was likely due to random chance of what address was corrupted.

The fix was simply to add an ss: segment override prefix.

The interrupt loops were included even if the only interrupts to hook were int 6 (optional to be excluded on the HP 95LX) or int 0Ch or 0Dh (not actually hooked in the interrupt loops). Thus, when creating special builds akin to Debug/X's DebugD (let's call this lDebugD) then these loops could be entered with the CX register set to zero, causing them to wreak havoc. The solution was to add branches for this case to all the interrupt loops. (Technically, this can be optimised out if we know the counter is not ever going to be zero.)

The chstool.eld is a new ELD. It grew out of my frustration to grasp the correspondence of partition table values in the MBR and in EPBRs. The first mode it gained was to translate (for a given unit, or an invented geometry) a packed CHS tuple (eg cxdx on int 13h entry) into the component CHS values and further into the corresponding LBA value. The LBA is further used to display a byte size (assuming 512 Bytes per sector), both in IEC and SI units.

The second mode is implemented as the WALK command, which takes a unit number. In this mode, the scanptab code is used to walk the entire partition tables structure of a unit, and display the partition type, LBA start, LBA size, CHS start, and CHS end values of each encountered partition. It was relatively easy to adapt the debugger's code from boot.asm to calculate the correct LBA start for every partition. Crucially, this did not require me to recall the exact structure of the extended partition tables.

The LBA start and size values are displayed as byte sizes (again assuming 512 Bytes per sector) as well. The CHS values are actually decoded using the same code as the first mode.

Finally, the code in the ELD to read a sector (always used to read a partition table) was adjusted to also dump the in-sector offset and hexadecimal contents of every partition table entry within the sector it loaded. The format for this dump is one byte (boot indicator), one 3byte (CHS start tuple), one byte (partition type), one more 3byte (CHS end tuple), and finally two dwords (LBA start and LBA size).

The tool also gained a submode of the WALK command activated using an EXTENDED keyword in which scanptab also calls the each partition handler for every encountered extended partition.

It finally dawned on me: All CHS tuples in all EPBRs are unit-absolute values. Only the LBA start values are relative to the start of the EPBR in which they're contained. This may seem obvious to some but I hadn't grasped this until after studying the chstool.eld output.

Edited to add: Logical partitions have a start LBA relative to the innermost EPBR in which the partition table is. Nested extended partitions, however, have a start LBA relative to the outermost EPBR in which they're contained. I fixed this in scanptab for the Extended mode.

The IF commands would branch to cmd3_notblank to run their THEN clauses. However, that entry is too late to run the command handlers of installed ELDs. The sense of this choice was likely to avoid activating auto-repeat. However, auto-repeat is already disabled by the non-empty IF command itself.

The fix is to branch to cmd3_injected instead, which will run the ELD command handlers.

In lineio.asm's GOTO command there was also a use of cmd3_notblank. This seems to be used solely to pass the label line to the command execution, which is most likely useless. However, for consistency I changed this likewise.

The partition structures moved out of debug.mac into iniload.mac to allow use in an ELD.

The ELD makone.sh script and its companion cfg.sh now include the scanptab pathname so that scanptab.asm may be included.

As we discovered a while ago in the forum, the Book8088 combination of ROM-BIOS modules led to a memory corruption error when an unknown int 13h function was called. This included the int 13h functions 41h (detect LBA extensions) and 42h (LBA extensions read).

The error was that the second handler would write to offset 41h expecting to access the byte at linear 441h. However, it did not set up DS to equal 40h as expected.

In the FreeDOS kernel, the detection function apparently was called with DS equal to zero, so the corruption affected the int 10h handler in the Interrupt Vector Table.

After some consideration, I reasoned that we do not need any particular DS value to call the detection function. So we can temporarily set up the expected DS = 40h for the call, to avoid the ill effects of the bug.

It took me a while because I was considering the case of the "LBA skip check" detection in the lDOS boot sector loaders. In their case, we call int 13h function 42h with CY directly, and detect if an error 01h is returned. In this case, a certain DS value is required and it may be cumbersome or impossible to arrange for the DS value to equal 40h and still address the LBA packet using DS:SI.

Therefore, the original suggestion was to avoid any LBA function calls, either by using the FreeDOS kernel's SYS CONFIG utility or the lDOS instsect /L none switch along with the lDOS iniload query patch to force CHS access. While these ways are still valid, and indeed the instsect switch (or a boot.asm build option of _LBA=0) are still the only ways to get the boot sector loader to behave, I have now figured out that the less constrained environments of lDOS iniload, lDOS testwrit, and lDebug can use the workaround of setting DS = 40h instead of having to patch them to avoid calling the LBA detection function.

So that's what this patch does.

The only real change in ldosboot is the symmetrical change to the LBA detection, to work around the Book8088 bug. This required some more spaghettification to make some space within the first 512 Bytes of iniload. Other than that it is the same change as to lDebug's detection in its boot.asm.

A documentation change is to mention MS-DOS in the lDOS iniload signature list (as "lDMS") and for use with drkernpl. This new use actually required no code changes in drkernpl at all; it just so happened that the MSBIO stage originally loaded using MSLOAD is also to be loaded to linear 700h and entered with CS:IP = 70h:0, just like DRBIO in Enhanced DR-DOS.

The convmasm.pl script in the (obsoletely named) tractest repo was originally written to handle the listing and map file outputs of the MS-DOS v2 release of MS-DOS Debug, which is still the basis of my MSDebug fork. As such, it did handle multiple listing files but did not handle multiple sections (segments) in the program.

There was another script, convdebx.pl, which in turn was written to handle the (patched) JWasm output of assembling Debug/X directly to a binary, without an external linker. This supported multiple sections but only a single listing file.

This week's patches to convmasm.pl mainly involved a fusion of both approaches to allow processing a multi-file, multi-section situation. Each file has a base offset for each section which occurs in it. This is handled by a hash of hashes structure in the perl sources.

Further changes involved to not consider non-global symbols because they may clash.

Some smaller adjustments include to allow slashes after prefix opcodes, allow "---- E" or "---- R" relocations for segment relocations, support SEGMENT AT directives, and handle what appear to be macro level indicators consisting of one or two bytes almost all the way to the right in the opcode dump line.

Too many changes to describe in detail. Basics:

• Set up a Mercurial (hg) repo with files from the git repo
• Replace some UTF-8 sequences in several source files to allow building (albeit with some source files broken)
• Edit environment script to refer to actual paths of include and library directories
• Prepend to PATH variable rather than replacing it
• Determine current directory using batch scripting trickery
• Add mak scripts (one for a Linux host and the other for the DOS side)
• Call the CPY.BAT scripts to copy generated files to the root directory
• sysinit1: Avoid to corrupt code with relocated stack overlapping code yet to be executed
• Rename message file to all-lowercase to avoid Linux host's hg to lose track of the file when modified during the build
• sysinit1: Align stack on a word boundary
• msload: Align stack
• Try to check in processed message file (this failed eventually)
• msload: Move initial stack to overlap code, increasing the stack size in the process
• msload: Lessen stack size to 256 Bytes
• msinit: Add debugger detection like FreeDOS's and Enhanced DR-DOS's
• sysconf: Pass maximum break address allowed to device drivers (this allows to load lDebug, which needs this MS-DOS v5 level feature)
• sysconf: Fix a check of a device driver's returned break address (may be redundant)
• sysinit1: Set SC or SD names for system-owned memory blocks
• identicalise branch:
  • Identicalise select.exe, easy
  • Move message file to a .pre file and copy it to the .msg file during build using the makefile, to avoid changes to the message file being registered as changes to the repo
  • Identicalise graphics.pro and select.prt simply by using unix2dos
  • Add unpacker tool for select.dat
  • Identicalise select.dat with the help of the unpacker to compare to the uncompressed data of the release
• msinit: Only reject "MSDOS3.0" or "IBM 3.0" BPBs, as suggested by someone at the OS/2 museum site
• msinit: Fix Cover_FDISK_Bug to only activate for exactly 64 Ki sectors
• sysinit1: Do not display error if an INSTALL= command in config.sys succeeds to exec but its return code does not indicate TSR return
• Pass CX from redirector Extended Open/Create to int 21h caller
• create: Pass attribute to redirector Extended Open/Create
• create: Share IFS_extopen_shared
• tools.ini: Do not mark message file as read-only after processing it, which interfered with the copy
• tools.ini: Create assembler listing files (chiefly for TracList)
• mem:
  • Only display SD sub-MCBs if SD name matches
  • Display SC blocks as System Reserved
  • Fix if a Z block is an SD container, and deduplicate code
  • Display Bytes or KiB sizes along hexadecimal size
• msdos: Process source files with a scriptlet so that code segment directives are included in the listing (for TracList's convmasm.pl script)
• Normalise MSDOS, MSBIO, and COMMAND link files
• mak.sh: Call convmasm.pl to create trace listing (.tls) files
• setenv.bat: Add DEBUG variable to disable @echo off directive
• copy and type: Allow to operate when Extended Attributes functions are not supported (on dosemu2 redirected drives or on other kernels)
• copy: Delete code disallowing trailing backslashes in destination pathname (required to make the new setenv.bat work to self-host the DOS)
• msdisk: Add Hidden_Trks before CHS calculation, hardening this
• msload: Paragraph align MSBIO stage in the file
• msload: Optimisations, some for size and two rep movsw instructions for speed
• identicalise branch: Use hharte's patch to identicalise to v4.01 release
• msdisk: Extract functions lbatochs and chstotuple
• msdisk, msinit: Support 256 CHS Heads (untested)
• msdisk: Correct a CHS overflow check, should check for Cylinder >= 1024 rather than > 1024
• msinit: Use only the LBA start values from EPBRs, do not use the EPBR CHS start tuple any longer
• msinit: Disable Hidden_Trks handling as we never set it any longer
• msinit: Check that partition last sector is CHS-addressable
• msinit: Detect LBA extensions support
• msinit: Work around Book8088 ROM-BIOS bug (same as lDebug and ldosboot)
• support LBA access
  • msdisk: Fix, do not depend on UPB ds == cs
  • move readsector into msinit as it is not used by resident msdisk
• msload: Create trace listing file
• Implement single-file load, re-using lDOS iniload/drload/drkernpl and the scripting from my Enhanced DR-DOS single-file load patches
• Drop DOS loader from msinit
• msload: Drop from repo

Edited to add: The Extended partition use of LBA start offsets had to be fixed for units that contain more than two logical partitions, after I figured out I had an error in the chstool Extension for lDebug. This turned up during local testing of the LBA access, in which I happened to create 6 partitions (1 primary, 5 logical) on an 21 GiB disk image. (I used the MS-DOS v7.00 fdisk to create the partitions.)