2023-10-03
This week I didn't get to prepare a blog post on Sunday. However, October 3rd is a public holiday in Germany, the Day of German Unity. So here's today's blog post.
This week I only worked on the debugger, specifically its support and selection of Extensions for lDebug (ELDs). Some commands were re-created as ELDs, other ELDs add completely new commands.
The first step along the way was to add the reclaim ELD. It would find unused memory at the end of the ELD code and data buffers and reclaim those by decreasing the "used" counters.
An important step was switching how unused memory is marked. Prior, zeroing the first byte of the ELD instance name would indicate it is no longer used. The problems with this are twofold:
First, ELDs should by default be marked unused so that interruption (such as by Control-C which is possible at any DOS I/O step, or if the debugger error handler is entered) will discard the remaining ELD. So we want to specifically mark ELDs as resident if they're hooked into the system somehow, and not keep them resident otherwise.
The second problem is that once the marker is set to NUL to indicate not resident, it is nontrivial to recover the letter overwritten by this marker in order to mark the ELD as resident again.
Both problems were fixed by allocating one bit from the 16 Bytes that had long since been reserved in the ELD instance structure. To harden the format, the ELD protocol revision was changed from ELD0 to ELD1 along with this.
The command handler allows an ELD to install itself residently and get to handle commands in the debugger's cmd3 command loop. The handler is called immediately after receiving a command from the getline function.
When the handler is run, it can decide how to handle the command. It can change the command, handle it completely (and return to the debugger cmd3 entrypoint), or pass it along back to the debugger to handle.
There is a single handler offset stored by the debugger in a variable. If a zero is stored, the debugger knows that there is no handler installed. Otherwise, it transfers control to the extension section.
The command handler has a special format, however. It starts with a short jump to the handler proper. After this, there must be an extcall to the debugger entrypoint for resuming normal command handling, cmd3_not_ext. There are 8 Bytes reserved for this extcall, for a total of 10 Bytes per entrypoint.
The special part is that this structure is well-known, and it is allowed that one ELD can refer to another ELD. To do so, it modifies the near call opcode (0E8h) that starts the extcall expansion and writes a near jump opcode (0E9h) instead, then writes a rel16 displacement that functions as a downlink to the next ELD in the handler chain.
Hooking and unhooking of the command handlers proceeds similarly to trivial interrupt handler chain modifications.
The reclaim ELD did mostly work well, but it had one serious flaw: As an ELD, it must be loaded into the remaining ELD code space itself. (Asides, the user has to remember to run it when appropriate.) The problem is that the ELD code space can become so full that the reclaim ELD itself cannot be loaded without reclaiming memory first.
There were several ways to solve this:
I chose the last option. It quickly turned out that the reclamation must happen in an EXT command before trying to load a new ELD:
cmd3 code.It cost about 200 Bytes, but the reclaim handling is now by default included into the debugger. (The uninstall messages still advise using the ELD, but this will be removed.)
The ELD linker faced a problem: If a data link or code link is not found, then the linker may itself not be finished being linked yet. This is a problem because basic message output requires linking in code like the puts family of functions and data like the line_out buffer.
The solution was to split all link tables in two: One table for the linker itself, and a second table for the ELD application.
Errors can be displayed in the second pass if the first pass succeeded in linking everything needed by the linker itself, including the error handling of the linker. This is a great quality of life improvement for developing ELDs, as it indicates exactly which links are missing rather than having to trace or guess.
Part of this change is to continue running the second pass upon errors after displaying the missing link. An error counter is kept to abort after the end of the linking. This allows to display errors for more than one link per run, also cutting down on the needed attempts at adding links. This is important because rebuilding the debugger can take minutes, so we want to lessen the amount of builds.
The linker can be optimised by setting the vstart attribute of the ELD data section and ELD code section to zero. However, there is an advantage to using a nonzero vstart: Missing or wrong relocations can be detected by comparing the ELD without the optimisation and the ELD (also called XLD) with the optimisation.
To this end, the eldcomp ELD runs another ELD twice and compares the memory it utilises in both instances. (To avoid a buffer allocation for now, it checksums the memory and stores only the length and the checksum.)
There are several macros used to assemble shared code snippets:
ELD define is assigned 0 in the debugger, 1 in ELDsrelocated smacro accepts one parameter. In the debugger, it expands to the parameter. In an ELD, it expands to the relocateddata label needed for link data relocations._PM=1 to include DPMI-specific code._PM=0 _EXTENSIONS=1 debugger with very small implementations that carry out the Real/Virtual 86 Mode tasks of these functions.
The only non-test user of this feature is eldcomp, yet. Command injection allows an ELD to run an entirely new iteration of the cmd3 loop and insert a new command instead of running the getline function.
The transfer happens similarly to command handler hooks, except for two parts: Every use of the inject handler also resets the inject handler field in anticipation of running without another injection next, and there is only ever one inject handler stored, not a chain of them. (When installing an inject handler, an ELD may choose to preserve the prior inject handler offset if any.)
The inject handler may branch either back into the cmd3 loop to the non-inject path (which will call getline) or instead to the injected path (after the getline callsite, which expects that skipwhite has been called). Of course, an injected command can also be processed by the command handlers set up by other ELDs (or even the same ELD, potentially).
Houdinis are conditional breakpoints. They can be disabled in several ways:
Houdinis are useful because it is not necessary to trace into the ELD anew for every run, rather, one can install houdini and then run the ELD. On the other hand, they don't interfere with usual use of an ELD.
Writing of which, it is not yet solved how to trace into an ELD with TracList. So far, the best workaround is to add a --local-offset 80h switch after the listing specified for the ELD. This depends on the ELD being the first ELD (other than the LINKCALL builtin ELD), however.
Described in vstart section. First user of command injection.
Re-creations of these debugger commands. RN notably ported the 386-related patching to the ELD application, to support the shared sources' use of it.
Displays install command nouns, including description, keywords, and current state (except for AREAS).