Early mid February extensions to WarpLink

This is not actually a regulation Sunday blog post as I didn't prepare it on the HP 95LX, due to the place I usually write these being closed for the carnival days.

This week I prepared some new extensions to the WarpLink OMF linker, most of which was originally written in 1989 through 1993, and released to the public domain in 1999. I have been working on WarpLink since 2023.

The new extensions I did as yet are split into three categories:

• wlemitalign, sections which emit their alignment even if they are empty or part of the nobits trail.
• wlcalc, post-link calculations that are applied to the executable file after the main linking is done.
• wllist, allowing to collect patch lists pointing into certain segments without having to manually update the lists.

• msinit.nas:
  • Make use of the WarpLink r10 wlcalc segment reference.
  • Delete old code with breakpoints and prior calculations that was to be used for comparison.
  • Hardcode DiskSector segment in the Init_BootSeg variable, rather than writing to it at run time.

• lmacros1.mac: As stated in the changeset message, delete the bytes() smacro because it interfered with a warning message in lDebug. There are no known users of this smacro and its counterpart, frombytes(), which both were no-op passthroughs.

• Add wlcalc_word_seg_ operation, which creates a segment relocation. Note that this adds the relative segment value (build/link time) to the word value in the usual wlcalc step, but applies a segment relocation to this spot. The relocation is written in an earlier pass so that it gets included into the relocation table. However, the actual resolving of the relocation of course happens at run/load time. Because this relocation is always applied to a word, only a word-size variable is allowed for this type of calculation.
• Support wlcalc PASS after the size, because the SEG operation doesn't support putting the PASS keyword after the segment name. To avoid misinterpreting "ADD" operations as hexadecimal literals, this use of PASS only supports decimal numbers. Like the other PASS keyword, the pass number is limited to 16 bits and the default pass is pass 0.
• Increase ecm release to r10.
• Add a bunch of .COM file switches. These are intended to replace the x2b2 utility by specifying WarpLink's /C switch (create a .COM file) along with some of the following:
  • /CZ: Accept entry point of (only) 0:0 rather than 0:256, and do not cut the 256-byte prefix off the executable image.
  • /CS: Ignore a stack if present.
  • /CB: Allow a big file (exceeding 64 KiB).
  • /CL=: Allow relocations, with the given number as a relocation factor to apply.
• Add wlcalc SEGREL operation. This adds the relative segment value at link time, but does not create a relocation. This is expected to be needed for porting lDebug to build with WarpLink. Unlike the SEG operation, any variable size (1 to 4 bytes) is allowed.
• wlcalc MINUSSEGREL operation: This subtracts the relative segment value at link time. It would be possible to emulate this using SUBR 0, SEGREL, SUBR 0 but it is much nicer to add this operation type.
• Disable listing of files included by an %include directive. As stated in the changeset message, this shrinks the trace listing (.tls) file from 8.7 MB to 3.2 MB, and it makes it much less annoying to search for labels that occur in the macro files. Scriptlets to generate the change are included in the changeset message too.
• Drop list directives for machlink.nas, so that the mlequate.mac, mlglobal.mac, and mlerrmes.mac macro files are listed once in the .tls file.
• Put the wlcalc strings into the .CONST section. No need but it is neater.
• Check for a zero environment segment, which can occur to indicate the process has no environment.
• Prepend underscore for labels starting with "wlcalc" lest they be misinterpreted as wlcalc requests themselves, if the linker were to be linked with the /XC switch. Scriptlet to generate included in the changeset message.
• Begin preparations for wllist support. Increase ecm release to r10a. Create wllist temporary list file and object file, and save the temporary object file's pathname to the object name list. Write the wllist-prefixed labels to stdout (unconditionally). Write a MODEND record to the temporary object file, before it is opened in pass 1 of the linker.
• Add most I/O functions to the process_wllist function, most of which are not yet used. There's functions to seek, read, or write the temporary list file, and to seek or write the object file. ecm release increased to r10b.
• Fix check_tmp_var function so it points after the inserted backslash, not before.
• Fix the so_xl function, it shouldn't chain to so_xc. The so_xl code wasn't enabled yet and won't be for several dozen more changesets after this one.
• Add /XS switch, which instructs the linker not to delete the temporary files it creates, to help with debugging them. It also displays the pathnames used for the files, at the point it would otherwise delete them.
• Add the Translator comment record to the temporary object file. The record is stored within the mlcredit.nas source text file so as to simplify updating the release number string that's part of the record.
• Add the _WLLIST_ERR code.
• After the /XS wllist preserve messages display a linebreak if delete_wllist_temps was called from the mlerror.nas code. This fixes the display in case of errors detected by the linker.
• Detect segment LNAMES for the wllist N labels, and write unique new LNAMES to the object file. "N" stands for "New list" and an N label must include an "S" key letter followed by a segment name, then an "L" key letter followed by a list name. The segment name is terminated by NUL, ?, or _. The list name is terminated by NUL or ?. Also increments ecm release to r10c. Adds I/O read functions to read back from the object file, and makes it so all read functions come in two variants: One that treats EOF (short reads) as an error, and the other that allows the caller to decide how to treat EOF.
• Detect group LNAMES for wllist G labels. A G label must have a second "G" key letter followed by the group name, then a number of "S" key letters each followed by a segment name.
• Write an empty LNAME as the first LNAMES record to the object file.
• Create wllist SEGDEF records. So far these indicate a size of 0 bytes. The creation is managed by two nested loops. The outer loop finds temporary list file entries with twlListUsed of zero and a twlSegmentIndex that's nonzero. The inner loop starts behind the currently processed temporary list file entry, and finds entries with twlListUsed of zero and a twlSegmentIndex matching the LNAMES index of the currently processed entry. In both loops, the matching entries have their twlListUsed set to 1 and the twlSegmentIndex updated from LNAMES indices to SEGDEF indices.
• Create wllist GRPDEF records. This is the first user of the second part of the file buffer, above the TEMPWLLIST structure at offset zero. It matches temporary list file entries with twlGroupIndex nonzero and twlListUsed zero. An inner loop finds the SEGDEF records that correspond to the G label's "S" key letters, and adds their SEGDEF indices to the GRPDEF record being built in the second buffer part.
• Display a _WLLIST_ERR if a group's segment isn't found. This also adds checks to mlerror.nas to display a symbol name if a _WLLIST_ERR is being handled.
• Fix to add a cpu 8086 directive to the nasm.mac macro file. Avoids assembling near jcc into the 386+ single-instruction form, falling back to the 8086-compatible dual-instruction form.
• Fix, address second buffer part using TEMPWLLIST_size rather than TEMPWLLIST. The latter is always zero.
• Fix to not hardcode the size of a temporary list file entry. This fixes two early spots that still hardcoded the size as 4.
• Use equates for the twlListUsed status. This byte can take on the values twlUsedInitial (written during pass 1 processing of all non-temporary object files), twlUsedNewSeg (to indicate a new list specifier with a segment LNAMES or SEGDEF index), or twlUsedNewGroup (to indicate a group specifier with a group LNAMES or GRPDEF index).
• In GRPDEF processing, loop through all twlUsedNewSeg entries with matching SEGDEF index and enter the appropriate GRPDEF index. Errors out if a nonzero GRPDEF index has already been written into the twlUsedNewSeg entry.
• Move the TEMPWLLIST structure into mlequate.mac, and make use of it (size only) in mlpass1a.nas where the initial temporary list file entries are written.
• Add the twlSegdefSizeOffset dword field to the TEMPWLLIST structure. For any twlUsedNewSeg entry that refers to a SEGDEF, this points to the word-sized "segment size" field within the SEGDEF record in the object file. This is needed because the segment size must be updated by later loops.
• Make the cmsg and zmsg functions in mlimage2.nas global. These were originally created for the /XS wllist preserving messages.
• Enumerate the twlUsedInitial wllist entries matching the list names of all twlUsedNewSeg entries, and check that no twlUsedInitial entries remain unprocessed after that loop. During enumeration, the used labels are displayed to stdout with a "Used wllist label:" prefix. Increase ecm release to r10d.
• Finally add the switches /TO= and /TL= which force the pathnames used for the temporary object file and list file, helping with debugging.
• Support the TEMP= variable as a fallback if TMP= isn't defined, and set the default path for the temporary files to .\ (only used if no /TL=, /TO=, TMP=, or TEMP= applies). Increment ecm release to 10e.
• Collect size needed for the wllist patch tables. Uses twlSegdefSizeOffset to access the size field within the object file's SEGDEF records.
• Extract the similar collect loops into a function. This function, .inner_collect, accepts a callback pointer to handle the specific actions.
• Emit LEDATA records for the pointer lists. As yet they contain all-zeroes.
• Create EXTDEF records for all list items. List items are here identified by temporary list file entries that still have a twlUsedInitial value in the twlListUsed field. A single EXTDEF record can hold multiple EXTDEF names, up to the second buffer part size or 1024 bytes, whichever is smaller. The EXTDEF record is partially written within the second buffer part and flushed to the object file either when too full or after the loop finishes. Adds the twlExtdefIndex word field to the TEMPWLLIST structure, which overlaps with twlSegdefSizeOffset as they aren't used at the same time.
• Fix to preserve registers across the in-loop .dump_extdef call. While the buffer holding the label name isn't touched by the dump and prepare functions, the registers referencing its contents are.
• Optimisation: Emit multiple LNAMES into the same record rather than creating a single record per. After every .enter_lname call, the partial record is temporarily completed and written to the object file at the last prepared position. The .prepare_lname function updates the last prepared position variable.
• Emit mix of LEDATA and FIXUPP records. Turns out, a FIXUPP record always references the very prior LEDATA record. (Page 44 of the OMF: Relocatable Object Module Format document, "Each subrecord in a FIXUPP object record either defines a thread for subsequent use, or refers to a data location in the nearest previous LEDATA or LIDATA record.") So in case of multiple LEDATA records it isn't valid to write all LEDATA records first and then all FIXUPP records. Multiple FIXUP entries will be written to the same FIXUPP record.
• Create PUBDEF record for the list start. This needs the twlGroupIndex, twlSegmentIndex, name of the current list, and the current ?ledatabase variable before the list items are collected.
• Extract three functions for working with PUBDEF records. This is the .prepare_one_pubdef, .index_one_pubdef, and .finish_one_pubdef functions. (The "one" in their name indicates that they're used to create a single PUBDEF per record, rather than possibly compacting several ones.)
• Create a PUBDEF record for the list end. This re-uses all three PUBDEF functions. It uses the ?ledatabase variable after all list entries have been collected. The name that it writes is the list name with ".END" appended.
• Create a PUBDEF record for the list amount. This re-uses two of the PUBDEF functions, writing four zero bytes for the group, segment, and frame fields. It uses the total LEDATA size determined from the first .inner_collect call, shifted right by 1 bit. The name it writes is the list name with ".AMOUNT" appended.
• Parses the R key letter, named twlRepeat. Not used yet.
• Implement repetition. This is special in that a thrice nested loop occurs, calling .inner_collect_partial from within .emit_ledata which can be called by .collect_list (itself a callback called by .inner_collect). The most deeply nested loop is being called with the current seek and current ?seektempinner value preserved on the stack, calling .collect_repeated_ledata to initialise the data of the LEDATA record that is currently being built. This function has a special exit condition: If the current LEDATA record is fully initialised, then it discards a word off the stack (its near return address into the innermost loop) before returning. So, it returns to the loop caller rather than the loop itself.
• Implement repetition step width. The "W" (twlWidth) key letter must occur after an "R" key letter. For repetition, it sets the step width of the repeated entries, which defaults to 1.
• Allow special case of an empty list that has entries, but all of them are R0 entries (repetition with repeat count zero).
• Default to disable wllist support, add the /XL switch to enable it, and increment ecm release to r11.
• Add the /XLD switch, which can be used to display debugging output of the wllist facility. The debugging output now defaults to not being displayed. One /XLD displays the "Used wllist label:" lines, while two or more /XLD switches also display the messages in mlpass1a.nas for every wllist label entered into the temporary list file.
• Fix wllist debug output, specifically the "Used wllist label:" messages. DX was no longer valid at the point the display happened. Also, the display occurs earlier in .collect_list now.

I am planning on porting lDebug so it can build as a number of object files then linked by WarpLink. Both wlcalc and wllist will be needed for this, along with the absolute $ trick to enter a nobits section mode in NASM where the nobits section overlaps with a progbits section.