2025-03-09

As mentioned, last week I didn't get to write a proper blog post. So this one's about two weeks' worth of changes.

Fix the error handling in realloc_ems_handle. EMS doesn't set an error code in BL, and do not modify the indirect pages variable in case of error.

Two changes to the test file program:

Align stack_top to a paragraph boundary. This doesn't modify the binary because it already happened to be aligned due to the stack size of 512 bytes.

Close the input file after reading all of it, but before creating the destination file. This would allow to specify the same file as both input and output, even with a sharer loaded. (But don't do that, on errors the file will be truncated.)

Detect the write_entrypoint_list mmacro like a DW directive, to handle its data as word data items. This was useful during lDOS development to compare two different revisions of the kernel, both of which were assembled using NASM. (A condition in msbio2.nas wasn't evaluated as intended because the ddataseg.nas include was missing. This led to a change between the binaries before and after this changeset.)

Add lkernpl.asm to load a single-file kernel at 00600h (like fdkernpl.asm) but pass along an lCFG block (like drkernpl.asm). Eventually all three of the *kernpl.asm files may be merged into one file, with the three existing filenames including this common file with appropriate options. However, this is of lower priority and can be done at a later time.

Add a variable called build_kernpl_module to allow using the new lkernpl.asm source file rather than drkernpl.asm.

• Fix variable.eld to reset ES after calling findvar in the length check. This led to memory corruption and checking the wrong variable on subsequent calls to findvar.
• Expand nonexistent variables to the empty string. Longer comment comparing the behaviour of different shells in the changeset message.
• In errfix.eld debug display indicate if the marker is behind the CR byte. This could happen due to a bug in printf.eld but at the time I believed that errfix.eld should handle this.
• Similarly, dump the content of the byte after the CR.
• In printf.eld do not overflow the input if no string END keyword is found.
• Harden several Extensions for lDebug (alias.eld, extname.eld, variable.eld) to handle the errfix array byte corresponding to the line_in byte after the CR. Likely useless but shouldn't cause any problems.
• In variable.eld when expanding a variable, move up the remainder of the errfix array to preserve the source contents for subseqent text. In a second changeset make it so that this only happens if the variable expansion is longer than the text it replaces. (Because we move downwards (DN) the move would corrupt data if the source is above the destination.)
• Add data links for priorrc and rc variables.
• Add new Extension for lDebug, rcexec.eld which adds the RC.EXECUTE command. This command replaces the RC buffer contents and immediately runs the RC buffer. This is intended to allow running multiple commands from a single-command alias.eld alias.
• Document rcexec.eld in the manual.
• Add S_INITCDS S MCB type and change spelling of S_INITIALISE
• Add S_INITSTACK S MCB type.
• Support _MCBPOSITION and _MCBLIMIT options for DM command (enabled by default only for the DM Extension for lDebug, not the internal DM command).
• Update dm.eld description in manual with new keywords.
• Add S_EXCLDALTAH S MCB type (not yet uploaded).

I developed the bigtest application as a test program for determining how much of the Low Memory Area is available to allocate to a single process memory block. I submitted the program and several results to the retrocomputing stackexchange site as a question and answer.

Of note is that bigtest.exe sets up an execution environment compatible to the debugger's Extension for lDebug architecture. It can and does in fact use an unmodified copy of dm.eld from the debugger to carry out the memory dump display.

• Initial commit: A simple program that doesn't do anything except allocate a large process memory block.
• Add a simple mak.sh script to build the test program.
• Add files to build dm.eld from the debugger.
• Disable building dm.xld by default.
• In repo root mak.sh call ELD makone.sh script to build dm.eld
• Include lmacros3.mac and add an include path to the nasm command.
• In mak.sh emit an assembler listing and pass map file and the listing to convlist.pl to create a trace listing file.
• Add loadeld.asm and eldapend.c from the debugger.
• Load an ELD and transfer to it. The ELD is loaded either from a filename specified on the command line (LFNs allowed), or from the program's own executable found from the pathname after the environment.
• In mak script compile eldapend and use it to append dm.eld as a trailing ELD to the executable.
• In mak script delete executable files before building.
• Add init_eld from debugger's init.asm
• Add linkinfo.asm file from the debugger.
• Disable most links in the link info file to enable building the test program with it included.
• Add some line I/O functions: puts, putsz, putc, putsline, putsline_crlf, and trimputs. These are much simpler than the debugger's, they always write to DOS standard output and do not do any pagination. Also adds the line_out buffer to the data segment.
• Add functions, variables, and links to allow running the ELD linker. This was done by manually checking eldlink.asm for every use of extcall and linkdatarelocation.
• Add all data links required for dm.eld (they were displayed by the linker on attempting to load the ELD)
• Add isstring.asm and options.mac from the debugger.
• Allow the ELD to run. This involves adding links to several new functions, including disp_dxax_times_cx_width_bx_size and getword. (The getword only accepts hexadecimal literals rather than adding a whole expression evaluator.)
• Fix getword function not to reset DX before every hexit.
• Set the dm.eld parameters for displaying position, limitting to up to the PSP, and table formatting.
• Rename test.asm to bigtest.asm
• Rename the output files to bigtest.*
• Add .hgignore file.

• Pass the dconfig environment to the shell, fixing the FreeCOM error prompt about not finding its executable. Unlike the prompt suggests, FreeCOM doesn't actually need the %COMSPEC% variable. It does need an environment and the trailing executable pathname however. I discussed this on the SvarDOS EDR-DOS tracker.
• Update lDOS version.
• Fix a crash after using an INSTALL= directive to install the DR-DOS sharer. I also discussed this on the SvarDOS tracker.

• Address most HMA allocations using segment FFFEh, except if the allocation stretches up to 0FFFFh:0FFEFh or 0FFFFh:0FFFFh. (DI after zeroing equals zero or 0FFF0h.) The idea is to be more conservative as in not using pointers with the segment as 0FFFFh, lest other code thinks the pointer is invalid.
• Create initial PSP at segment 61h.
• Put the first MCB at segment 60h, and create an S MCB for DOSENTRY + DOSDATA.
• In entryseg.nas avoid NASM warning on segment attribute redeclaration.
• Drop subtraction of initial PSP that used to be in front of SYSINIT CS.
• Move some device headers and variables in DOSENTRY:
  • Device headers (misspelled) up to LPT3 before ORIG13
  • COM2 to COM4 device headers before IRT
  • Several variables to in front of AUXNUM. (The diff shows AUXNUM and subsequent variables as moving to behind the other variables.) At this point I noted that the subsequent variables must be placed after AUXNUM. When I tried recreating the crashes due to this several days later, I was unable to reproduce however.
  • Move SM92 structure to before AUXNUM.
  • Put entry.asm before ALTAH.
  • Move msdisk.nas variables to before ALTAH.
• In msbio1.nas drop duplicate dosbiocode functions.
• Share code for Leave2F and Leave13.
• In msdisk.nas use Leave13 rather than retf 2
• Note that msdisk.nas uses Leave13.
• Move msclock.nas variables to before ALTAH.
• Move msbio2.nas variables to before ALTAH.
• Move around some data to prepare for next changeset.
• Add msbio.cl2 message before ALTAH.
• In mak script update the filenames to delete.
• In exec.nas fix an incorrect JB after a DEC, which would never branch. Use JZ instead.
• Relocate SYSINIT and DOSCODE from DOSENTRY, avoiding the hazard of overwriting the relocation code itself. This uses the movp function from other lDOS projects.
• Re-initialise DOS segment references soon after relocation.
• Do not re-init DOSDATA segment at this point as it doesn't change yet.
• Check that the relocation doesn't corrupt the relocation code. This was too early in the repo history, as at this point the relocation could corrupt the DOSDATA segment without this detection erroring out. This was automatically fixed later when DOSDATA was relocated here too.
• Before the relocation check for a borrow after subtracting from the memory top segment. This is unlikely to be needed, but more correct.
• Write a SYSINIT S MCB, not used yet.
• Enable passing config lines using PREPEND or APPEND even if no config file is found. This fix was actually incidental; I believed it to be needed but found that the actual error was due to running the lDebug command like boot protocol ldos; append … in what the last clause is parsed as a comment after the protocol specification. The fix is still correct however.
• Link the free MCB to the SYSINIT MCB. The latter is not freed yet.
• Round up the size of a buffer in a better way, not adding a paragraph if the buffer header exactly fits in a paragraph.
• Fix, TryQMemory (error allocating memory for country.sys buffer?) should set CY.
• Big change to drop all uses of the devmark SD MCB. (Only DEVICE= / DEVICEHIGH= still creates SD MCBs.) Adds the allocate_temporary_block function to allocate memory to an S MCB using last fit LMA only. The country block is temporarily allocated and then freed. The temporary CDS block is repeatedly allocated and freed. The config block and buffer block are allocated but not freed yet. The SYSINIT_BASE block is no longer used, INSTALL= executes from the SYSINIT block directly.
• In allocate_relocate_block first try UMA only best fit, then UMA then LMA first fit.
• Align some word variables.
• Free config block after last use (by INSTALL= execution). Could be improved to compact and relocate this some before this point.
• Free early buffer block after setting up final buffers.
• Change msbio/msinit.nas BOOTBIAS to 0.
• Allocate and free Init_BootSeg, instead of reserving part of the SYSINIT MCB.
• Relocate UPBs early, and discard their dskdrvs array (only used by INITDPB). All UPBs at this point are DOS-internal ones.
• Relocate DOS-internal UPBs late. This specifically does not include DRIVER.SYS UPBs added to the chain using the int 2Fh function 08h interface.
• Fix the DPB drive/unit number on DEVICE=, DX got corrupted. (Probably broken since the change to not use devmark SD MCB.)
• Fix a buffer overflow in INITDPB, need to grow the DPB S MCB first before writing to the next DPB's fields. Strangely enough this seems not to have caused any problems.
• Relocate DPBs early. All DPBs at this point are DOS-internal DPBs.
• Relocate DOS-internal DPBs late. This does not include those allocated by DEVICE= (appended at chain end). Relocation of DPBs at this point involves relocating references in buffers, CDSes, SFTs, and FCB SFTs.

Relocating the DOS-internal UPBs and DPBs twice each is to simplify the changes needed. The early, first allocation of both in INITUPB and INITDPB depends on allocating from the first free block so that the allocation can grow upwards whenever more blocks are needed. This would have to be changed to immediately allocate at the temporary location. The early relocations happen to be fairly easy; the DOS-internal DPBs aren't even referenced by any other data structures at the early relocation point.

• Split code.asm from entry.asm and include it late in msbdata.nas, this allows to put relocating entrypoints later in msbdata.nas
• Run shell exec call in DOSENTRY, and free the sysinit memory on successful exec.
  • This involves a backdoor variable to exec.nas which indicates to free memory after successful exec, before passing control to the new process but after loading the executable and building its environment.
  • This isn't fully utilised yet: After an unsuccessful exec, control flow could return into SYSINIT for another attempt, possibly prompting the user for an executable and command line to run. At this point the code just displays a message and halts.
• Use the DISKSECTOR in DOSENTRY as the SHELL= exec stack. Short lived hack.
• Allocate a block of 160 bytes for the init SHELL= exec stack. This can go into the UMA or even HMA: Whenever this stack is used in the HMA we have already entered DOS (also in the HMA) so A20 must be on.
• Add some memory critical sections.
• Be more strict about UPB/DPB relocation.
• Extract function relocate_dibx_dpb_from_dxbp_to_essi
• In prior changeset's function disable interrupts around the change
• Fix so RE_INIT is a near function in SYSINIT. (This seems to have happened to work but was actually running in free memory. Also, it was located in the resident BIOCODE so wasted some space there too.)
• Adjust the final DOS int 2Fh handler to use segment 26h for its entrypoint.
• Note on where dual area handling occurs in memory.asm
• In memory.asm use global, not extern (happened to work regardless)
• Comment out memory.asm function GetSizeInMCB.
• Place early DOSCODE in front of rather than behind SYSINIT. Give it its own S MCB as well. (Already freed by DOSCODE relocator.)
• In msdos/msinit.nas clear all fields of the free MCB.
• Relocate DOSDATA initially (along with DOSCODE and SYSINIT).
• Extract function DOSREINIT.
• Fix to write segments of swap areas in DOSREINIT, not NEARDOSINIT. This caused the MFS redirector of dosemu2 to read from the buffers in the early DOSDATA later than it should.
• Relocate DOSDATA in relocate end function, call DOSREINIT there.
• Optimise the DOSENTRY A20 and relocation code. Uses a far jump immediate for the XMS entrypoint, an SMC patch to enable the A20 handling rather than comparing the DOSCODE segment, and the DOSCODE segment is found from a new reference within a mov immediate instruction. (DOSCODE relocation needs to update this field too.) The A20 error messages are also split apart in a smarter way.
• Rename START section to DOSSTARTJUMP to avoid clash with msequ.mac's START equate.
• In ddataseg.nas and entryseg.nas use lmacros3.mac's sectioning macros.
• Define the DOSBIODATA section alias for DOSENTRY.
• Fix, adjust SPSAV for the pushed DS.
• Prepare to split DOSBIODATA from DOSENTRY. Debugging this change, which should cause no binary difference, required the ident86 change.
• Shuffle some variables.
• Fix: HARDNUM should be in SYSINIT section. If I recall, this happened to corrupt the final buffer chain leading to fun to debug infinite loops.
• Add DOSBIODATA section to the DOSDATA segment.
• Fix the Patch_DoubleWordMov code (disabled in prior changeset) to set a flag byte rather than patch code. This was likely broken earlier than this.
• Place initial PSP at top of available memory. (This temporarily allocated the first MCB to the backdoor owner, to avoid interfering with the UPB/DPB/config/DEVICE= allocations which expect the first free block to be large.)
• End relocate init PSP to the LMA or UMA. (Cannot be HMA as it needs to be addressed using a segment.) PSP relocation is based on the code from TSR example.
• Fix some DOSENTRY uses that want DOSBIODATA.
• Move DOSENTRY segment to 00600h. (The first paragraph is overwritten by the DOSENTRY S MCB, so the resident contents proper start at 00610h.)
• In iniloadc.mac set _PAYLOAD_FILE only for module lkernpl.
• Spaghettify some DOSENTRY contents.
• In msioctl.nas fix wrong section for some tables.
• Move two entrypoints to earlier in msbdata.nas
• In msvolid.nas fix wrong section for some variables.
• Address msbio device headers using segment 70h.
• In the makefiles make it so msbio1.obj depends on the segment include files.
• Insure that device headers and entries are high enough (>= 100h) to be addressed using segment 70h.
• In DOSREINIT detect sharer present, and scan for magic code that tells us where the MFT starts.
• In DOSREINIT relocate SFT references in sharer, if present. (Discussed on the SvarDOS EDR-DOS tracker.)
• Move int 6Ch handler and all clock-related code from DOSENTRY to DOSCODE, and data to DOSBIODATA (ie, the DOSDATA segment).
• Parse the config directive COMPAT=ALTAH (no-op).
• Disable ALTAH compatibility offset that used to hardcode the variable to 70h:61Bh.
• Allocate memory block for COMPAT=ALTAH option.
• In mscon.nas change uses of ALTAH to indirect through an offset stored in DOSBIODATA.
• Modify the indirect ALTAH offset to implement COMPAT=ALTAH fully.