2023-11-26

This week I worked some on lDebug, its ELDs, and a few patches to TracList and MetaKern.

MetaKern will now pass DS:SI -> 16 zero bytes, to aid lDOS boot loaders that try to detect a valid partition table entry for dynamic hidden sectors determination.

This is the simplest possible fix for this. Because MetaKern may be used for loading other partitions' boot sectors, I did not make it pass along the hidden sectors value that MetaKern itself unofficially received in the FreeDOS load protocol BPB.

The TracList script gained a new command. The A command is used to start an automatic Multi, Update, Unmulti cycle. This is useful to gain the better performance of Unmulti mode for a single file when a large number of files is open, but select the correct file quickly when switching listings is required.

It is possible to emulate the operation of the A command by sending two M commands, but the single A command is faster and more convenient.

• bootdir.eld: Do not scan entire directory for every sorted entry, rather end the scan when the sorted entry is found
• In SORTDEBUG display how many SFN entries would fit in the buffers
• Refer to "buffer" rather than auxiliary buffer to reflect the use of either type of buffer
• Implement DIRFIRST sort modifier
• Implement DATE sort order
• Implement SIZE sort order
• Add REVERSE sort modifier which affects the name, DATE, or SIZE sort order (but not DIRFIRST)

This pair of ELDs allows to inject debugger commands from one debugger to another. The amiscmd ELD provides the new private AMIS function 43h. This receives a counted-string command, which it stores in a 1024 Bytes queue. It installs a command inject handler which will inject the next command when it is called. The inject ELD is the counterpart that calls the function 43h.

During testing of these commands, it became apparent that command injection doesn't play nice with the Y command. If a Y command is injected and then another command, the other command will always run before any of the commands read from the Script for lDebug. If a change to this behaviour may come along later, I reserved the CX input for function 43h for flags. The handler installed by amiscmd.eld will check that CX is zero, for now.

Some of the more recent ELDs reintroduced a bug: Modifications of line_in can corrupt buffered input from a Script for lDebug file if they do not take care to call yy_reset_buf. I tested all affected ELDs and verified that they do fail in this way when called from a Script for lDebug.

In some cases, such as the EXTNAME command installed by extname.eld, it is important to put the command to test as the very first command in the Script for lDebug so that its limited (4-byte) expansion of the line_in content will indeed corrupt part of the buffered content. If an EXTNAME command appeared later, the remaining buffered content may start beyond the reach of the buffer overflow.

The ifext ELD's uinit_data_end label wasn't aligned on a paragraph boundary. This is harmless, but for consistency I added the alignment.

The path search ELD gained a mode to process a single N or K command as a transient ELD, then run that command using command injection. However, in this mode there is no way to deselect any of the three features of the ELD. (The three features are unknown filename extension warning, filename extension guessing, and path search.)

This is probably not needed, but ext_finish will now preserve the BP value set up by the file open handling.

The ELD loader has been extracted from lineio.asm to make it re-usable for library ELDs.

This ELD is a library of other ELDs. All the user-facing regular ELDs are packed into this ELD's file. It uses the file handle left open by the debugger's ELD loader, and the fact that this loader always leaves the file seek pointing behind the data block image of the first ELD. This allows the extlib ELD to find its library within its file. (It is possible to nest library-style ELDs, though there isn't really a reason to do so.)

The ELD contains a small help, as well as two ways of listing the contained ELDs. An empty command line tail or the keyword HELP will bring up the help. If HELP WIDE or HELP DESCRIBE are specified, then the help display will additionally list the contained ELDs.

The wide display is similar to a wide DOS DIR listing. It is also fast, as the library ELD only needs to list data included in its own ELD data block to create the wide display.

The describe display will list the contained ELDs with one line per ELD, and includes the description lines read from the ELDs. This listing is slower because extlib.eld needs to read the library files to find and read their description lines.

An interesting optimisation is the use of relative seeking on the bootloaded file system instead of absolute seeking. This is a win for forward seeking or limited backward seeking (within the current cluster). However, there is no harm in using it for longer backward seeking because the relative seek handling will correctly rewind the file to the starting cluster if needed.

Another part is that HELP commands will actually zero the offsets within the library table once they are done. This is to avoid a large number of false positives when testing the ELD using eldcomp.eld. The additional Metadata added for list.eld also has an offset of the library section itself, named eldltOffset, which is also zeroed by this code.

The get name parse loop in ifext.eld and extlib.eld would skip the initial byte of an unquoted ELD name. This did not seem to cause any problems but has been corrected.

The ELD loader didn't check that the 16-bit size registers wouldn't overflow due to the paragraph alignment addition. This only affected invalid ELDs, where the (code or data) image size plus allocation size is larger than 0FFF0h.

The ELD loader used by the extlib ELD was adapted to use the common DOS or boot I/O dispatch function, optionally. To support use in the debugger without changes, a build option was added to loadeld.asm that can enable this code. However, the code for non-ELD use misspelled the %assign directive that was meant to disable this option.

The extlib ELD's own library table was expanded with a small header of Metadata prepended to the table. This allows the list.eld to traverse the nested ELDs and display their description lines, and optionally the list.eld verbose output as well as the nested ELDs' help pages. The list ELD will only recurse into any libraries it encounters if the LIB keyword is specified on the command line.

The recently-added SEG display of the ldmem ELD was missing a getotherds for accessing the selectors of the other link debugger when used as ldmemoth.eld. This meant the selectors line displayed would contain the selectors of the host debugger rather than the other link debugger's.

The extpak.eld is a variant of the extlib.eld in which all the contained library ELDs are compressed in one solid block. The window size for the heatshrink compression is chosen as -w 12 to fit with the 4 KiB circular decompression buffer allocated in the ELD data block. A 256-byte file input buffer is used to avoid frequent file reads in the depacker. Whenever parts of the compressed ELDs are needed, the depacker is run until the desired fragment is read.

As a major optimisation, a prior run of the depacker may be continued if it ended up not reading past the next fragment to read. As most reads of the compressed ELDs are sequential in this way, the depacker only needs to be restarted from the beginning very seldom. The performance impact is obvious for some operations, such as the HELP DESCRIBE listing or the repeated comparisons of ELD instance names until extpak.eld decides that no ELD matches the specified name.

To support the continued depacking, I initially considered backing up and restoring its stack. However, it turned out to be simpler to switch stacks instead. Therefore, an additional 256-byte stack is allocated within the ELD data block. (It overlaps the help message though, as that is no longer needed once reading the compressed ELDs has begun.) This design also would easily work if offsets into the depacker's stack are used within the depacker. This is likely not the case, but was a consideration for the stack backup approach. (The depacker does point its copy_data function at its stack for 1-byte decompressed literals, but this likely would be of no concern for a stack backup as this pointer is not used after calling put_file_data.)

In addition to the depackskip variable which acts as a seek offset into the compressed image, the stack switching utilises a new depackseek variable to tell how far within the image the prior depack call ended up. If a prior depack call was paused in this way, and the seek is below-or-equal the next desired skip, then the read_and_depack function will continue the prior depack call rather than starting a new one from the beginning of the image. This involves calculating an adjusted skip by subtracting the current seek from the skip variable.

The stack-switching will then use the prior depack call's stack as it was saved by the put_file_data function, to return into the context of this very function again. All of the 16-bit 8086 registers needed by either depack or read_and_depack are saved on each of their stacks, though the only change needed for this was to save and restore BP. (The need for this change was indicated by a General Protection Fault on a les di instruction in depack when testing the partial depacker in DPMI protected mode in lCDebugX.)

The common setup done by every read_and_depack function call involves saving away the file handle (or bootloaded file structure offset) and setting up the depacked_buffer and depacked_length variables, which indicate where to read the desired fragment to. The setup that is only done for new depack calls consists of resetting depackseek to zero, seeking the input file to the offset of the compressed library image, clearing the file buffer's .next and .tail pointers to prepare for reading from the file, and finally the stack is switched to the top of the depack_stack.

Like extlib.eld before it, extpak.eld will zero the offsets in the library table after a help call. However, the compressed image length passed to the depack function is encoded in mov immediate instructions in the code section so does not easily lend itself to modification. Therefore, it is documented that this code section mismatch is expected.

Further, the 4 KiB circular decompression buffer is too large for eldcomp to successfully allocate the ELD data block buffers for comparison if a data mismatch is detected. It helps to edit the extlib.asm sources to use a 256-byte buffer instead, which won't succeed when actually trying to decompress anything but will play well with eldcomp.

The ELD library support code in list.eld does not yet support the compressed library image used by extpak.eld. Therefore, the help page display and verbose listing output of list.eld is not available for the extpak.eld library. The description lines of the library ELDs can be displayed using extpak's own HELP DESCRIBE command, but not using list.eld.