pushbx wiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== September October work on Extensions for lDebug ======


**2023-10-03**

This week I didn't get to prepare a blog post on Sunday. However, October 3rd is a public holiday in Germany, the Day of German Unity. So here's today's blog post.

This week I only worked on the debugger, specifically its support and selection of Extensions for lDebug (ELDs). Some commands were re-created as ELDs, other ELDs add completely new commands.


===== The current ELDs =====

  * ldmem: Display lDebug memory use. Can be installed residently, hooking into the command dispatch to add the LDMEM command.
  * reclaim: Reclaim unused memory at the end of the ELD buffers. No longer needed, code embedded into the EXT command.
  * eldcomp: Compare two ELDs for byte-perfect matches. This is used to check that different ELD linker options result in the same extension image, helping to identify errors in the ELD relocation. This ELD uses ELD command injection.
  * DI, DM, RN, RM: Re-create the original debugger commands. These can also be installed residently to handle the respective commands.
  * HISTORY utility: This allows to clear the command history or dump it to the screen. Can also be installed residently.
  * instnoun: Adds a command that lists all install nouns as well as the current state for most of them.


===== ELD interface progress =====

==== The reclaim ELD ====

The first step along the way was to add the reclaim ELD. It would find unused memory at the end of the ELD code and data buffers and reclaim those by decreasing the "used" counters.


==== ELD resident flag ====

An important step was switching how unused memory is marked. Prior, zeroing the first byte of the ELD instance name would indicate it is no longer used. The problems with this are twofold:

First, ELDs should by default be marked unused so that interruption (such as by Control-C which is possible at any DOS I/O step, or if the debugger error handler is entered) will discard the remaining ELD. So we want to specifically mark ELDs as resident if they're hooked into the system somehow, and not keep them resident otherwise.

The second problem is that once the marker is set to NUL to indicate not resident, it is nontrivial to recover the letter overwritten by this marker in order to mark the ELD as resident again.

Both problems were fixed by allocating one bit from the 16 Bytes that had long since been reserved in the ELD instance structure. To harden the format, the ELD protocol revision was changed from ELD0 to ELD1 along with this.


==== Command handler ====

The command handler allows an ELD to install itself residently and get to handle commands in the debugger's ''cmd3'' command loop. The handler is called immediately after receiving a command from the ''getline'' function.

When the handler is run, it can decide how to handle the command. It can change the command, handle it completely (and return to the debugger ''cmd3'' entrypoint), or pass it along back to the debugger to handle.

There is a single handler offset stored by the debugger in a variable. If a zero is stored, the debugger knows that there is no handler installed. Otherwise, it transfers control to the extension section.

The command handler has a special format, however. It starts with a short jump to the handler proper. After this, there must be an ''extcall'' to the debugger entrypoint for resuming normal command handling, ''cmd3_not_ext''. There are 8 Bytes reserved for this ''extcall'', for a total of 10 Bytes per entrypoint.

The special part is that this structure is well-known, and it is allowed that one ELD can refer to another ELD. To do so, it modifies the near call opcode (0E8h) that starts the ''extcall'' expansion and writes a near jump opcode (0E9h) instead, then writes a rel16 displacement that functions as a downlink to the next ELD in the handler chain.

Hooking and unhooking of the command handlers proceeds similarly to trivial interrupt handler chain modifications.


==== Reclaim function embedded into debugger EXT command ====

The reclaim ELD did mostly work well, but it had one serious flaw: As an ELD, it must be loaded into the remaining ELD code space itself. (Asides, the user has to remember to run it when appropriate.) The problem is that the ELD code space can become so full that the reclaim ELD itself cannot be loaded without reclaiming memory first.

There were several ways to solve this:

  * Expand resident ELD uninstall code to do the reclamation when appropriate.
  * Reserve enough space in the ELD code buffer to hold the reclaim ELD.
  * Embed the reclaim handling into the debugger.

I chose the last option. It quickly turned out that the reclamation must happen in an EXT command before trying to load a new ELD:

  * ELDs do not have to return to the EXT command handler, they can directly enter the ''cmd3'' code.
  * Resident ELDs can uninstall "at any time".
  * The eldcomp ELD wants ELDs to retain their allocations in the buffers after they finish. This can be accomplished with just one DCO7 flag checked by the ELDs, without a run time option in the debugger, if reclaim happens at the beginning of an EXT command.

It cost about 200 Bytes, but the reclaim handling is now by default included into the debugger. (The uninstall messages still advise using the ELD, but this will be removed.)


==== ELD two-pass linker ====

The ELD linker faced a problem: If a data link or code link is not found, then the linker may itself not be finished being linked yet. This is a problem because basic message output requires linking in code like the ''puts'' family of functions and data like the ''line_out'' buffer.

The solution was to split all link tables in two: One table for the linker itself, and a second table for the ELD application.

Errors can be displayed in the second pass if the first pass succeeded in linking everything needed by the linker itself, including the error handling of the linker. This is a great quality of life improvement for developing ELDs, as it indicates exactly which links are missing rather than having to trace or guess.

Part of this change is to continue running the second pass upon errors after displaying the missing link. An error counter is kept to abort after the end of the linking. This allows to display errors for more than one link per run, also cutting down on the needed attempts at adding links. This is important because rebuilding the debugger can take minutes, so we want to lessen the amount of builds.


==== Variable vstart ====

The linker can be optimised by setting the vstart attribute of the ELD data section and ELD code section to zero. However, there is an advantage to using a nonzero vstart: Missing or wrong relocations can be detected by comparing the ELD without the optimisation and the ELD (also called XLD) with the optimisation.

To this end, the eldcomp ELD runs another ELD twice and compares the memory it utilises in both instances. (To avoid a buffer allocation for now, it checksums the memory and stores only the length and the checksum.)


==== Macros for sharing code with debugger and ELD ====

There are several macros used to assemble shared code snippets:

  * The ''ELD'' define is assigned 0 in the debugger, 1 in ELDs
  * The ''relocated'' smacro accepts one parameter. In the debugger, it expands to the parameter. In an ELD, it expands to the ''relocateddata'' label needed for link data relocations.
  * The relocation macros are defined as no-ops by the debugger, while being used to add relocations in ELDs.
  * ELDs may assign some values to certain defines, such as ''_PM=1'' to include DPMI-specific code.
  * Relatedly, some DPMI-specific functions are supported by a ''_PM=0 _EXTENSIONS=1'' debugger with very small implementations that carry out the Real/Virtual 86 Mode tasks of these functions.


==== Command injection ====

The only non-test user of this feature is eldcomp, yet. Command injection allows an ELD to run an entirely new iteration of the ''cmd3'' loop and insert a new command instead of running the ''getline'' function.

The transfer happens similarly to command handler hooks, except for two parts: Every use of the inject handler also resets the inject handler field in anticipation of running without another injection next, and there is only ever one inject handler stored, not a chain of them. (When installing an inject handler, an ELD may choose to preserve the prior inject handler offset if any.)

The inject handler may branch either back into the ''cmd3'' loop to the non-inject path (which will call ''getline'') or instead to the injected path (after the ''getline'' callsite, which expects that ''skipwhite'' has been called). Of course, an injected command can also be processed by the command handlers set up by other ELDs (or even the same ELD, potentially).


==== Houdinis ====

Houdinis are conditional breakpoints. They can be disabled in several ways:

  * At build time
  * By running in non-debuggable mode of lCDebug
  * By running in a non-debuggable build of lDebug
  * By clearing a DCO7 flag (install noun HOUDINI)

Houdinis are useful because it is not necessary to trace into the ELD anew for every run, rather, one can ''install houdini'' and then run the ELD. On the other hand, they don't interfere with usual use of an ELD.

Writing of which, it is not yet solved how to trace into an ELD with TracList. So far, the best workaround is to add a ''-%%%%-local-offset 80h'' switch after the listing specified for the ELD. This depends on the ELD being the first ELD (other than the LINKCALL builtin ELD), however.


===== Details on the ELD applications =====

==== ldmem ====

  * Displays memory use of the debugger, [[blog:pushbx:2023:0924_mid_september_work_on_toclip_warplink_ldebug#ldmemeld|as mentioned previously]]
  * Displays amount of history entries, bytes, and bytes per entry average
  * Displays stack use heuristic
  * Display ELD memory in ELD code and data buffers


==== History ====

  * Clear history
  * Display history contents
  * Only ELD with a help page so far


==== eldcomp ====

Described in vstart section. First user of command injection.


==== DI, DM, RN, RM ====

Re-creations of these debugger commands. RN notably ported the 386-related patching to the ELD application, to support the shared sources' use of it.


==== instnoun ====

Displays install command nouns, including description, keywords, and current state (except for AREAS).

{{tag>ldebug eld}}


~~DISCUSSION~~
pushbx wiki

User Tools

Site Tools

Page Tools
