====== Late September, early October work ======

**2024-10-13**

Last Sunday (2024-10-06) I travelled back from my partner's place, so I didn't get to write a blog post. Therefore we'll cover two weeks' worth of work today.


===== kernwrap =====

All of these changes were picked from lDebug.

  * [[https://hg.pushbx.org/ecm/kernwrap/rev/353429a4bc27|Add support for INICOMP_HEATSHRINK_Z]] to optimise heatshrink depacker
  * [[https://hg.pushbx.org/ecm/kernwrap/rev/c9e1c56f69c3|Support mvcomp compression method]]
  * [[https://hg.pushbx.org/ecm/kernwrap/rev/7a7f2ab4d00e|Update config file]] to add zerocomp and mvcomp compression methods


===== Enhanced DR-DOS =====

Prompted by [[https://github.com/SvarDOS/edrdos/issues/116|a SvarDOS issue for the EDR-DOS kernel]], I [[https://hg.pushbx.org/ecm/edrdos/rev/374911957af5|optimised the bdevio.nas function div64]]. I'd noticed that its stack layout exactly matches the div32 function. The div64 function implements a very slow bitwise loop to do the division with up to 64-bit dividend and up to 32-bit divisor. My optimisation consists of checking the upper 32 bits of the dividend; if they're zero, pass control to the div32 function instead. This causes a great speedup apparently.

I later [[https://hg.pushbx.org/ecm/edrdos/rev/529340d7b94d|picked a comment]] from [[https://github.com/SvarDOS/edrdos/commit/33c091a5d3c2f7dac1648844cf43f524e67e2f62|the SvarDOS repo's pick of my changeset]].

[[https://hg.pushbx.org/ecm/edrdos/rev/1c95556ed6bc|The only unrelated changeset]] in our EDR-DOS repo is [[https://github.com/SvarDOS/edrdos/commit/9b5332b28ce62849c0b24a0ac75182206109d270|a pick from SvarDOS]] to supply a video page and attribute for a few int 10.0E calls in drbio/console.nas (as usual I picked the change in the JWasm source texts into both that and the NASM ported sources for the lDOS repo). ([[https://github.com/SvarDOS/edrdos/issues/117|Issue at SvarDOS EDR-DOS repo]].)


===== inicomp =====

  * [[https://hg.pushbx.org/ecm/inicomp/rev/dd81110642e4|Add testfile application that uses the inicomp depackers]] to depack a single file, and write to an output file. This involves passing the depacked size from the depack function to the caller.
  * [[https://hg.pushbx.org/ecm/inicomp/rev/0b9dc801dcb8|Add -v switch to testfile]] for verbose debugging output.
  * In heatshrink depacker verbose output [[https://hg.pushbx.org/ecm/inicomp/rev/e60c60b1a59e|display destination and source offsets]].
  * In heatshrink depacker [[https://hg.pushbx.org/ecm/inicomp/rev/f5f72d7078a2|add verbose message for storing zeroes]] (backreference matches that reference the initial window before the start of the uncompressed data).
  * [[https://hg.pushbx.org/ecm/inicomp/rev/13563a9b6e28|Add -e switch to testfile]] for exceptional debugging messages. (The -v switch enables a superset of the -e switch messages.)
  * [[https://hg.pushbx.org/ecm/inicomp/rev/adbcba7c1d12|Fix heatshrink depacker to work in the inicomp stage again]], it had the _TESTFILE define too late.
  * [[https://hg.pushbx.org/ecm/inicomp/rev/04de03dc0726|Optimise heatshrink depacker with _Z define]] to drop code for storing zeroes for initial window backreferences, shrinking the depacker somewhat.


The remaining changes involve the mvcomp compression method, based on SvarDOS's language file packer:

  * [[https://hg.pushbx.org/ecm/inicomp/rev/bc45b98b99c5|Add mvcomp depacker]].
  * [[https://hg.pushbx.org/ecm/inicomp/rev/71ddecbbb8dd|Support _MVCOMP define]] for mvcomp depacker.
  * Optimise mvcomp depacker:
    * [[https://hg.pushbx.org/ecm/inicomp/rev/80102d0ff030|Inline get_word]] as only a single spot ever called it.
    * In copy_data [[https://hg.pushbx.org/ecm/inicomp/rev/b38ab09d4e3d|only support lengths < 256 Bytes]].
    * [[https://hg.pushbx.org/ecm/inicomp/rev/49968d4ec942|Drop unused .end_check code]].
  * [[https://hg.pushbx.org/ecm/inicomp/rev/0b1c60bfe1df|Double the number of progress dots]] emitted by mvcomp depacker.
  * [[https://hg.pushbx.org/ecm/inicomp/rev/6e6e0530b90f|Add testfile verbose output]] for matches to mvcomp depacker.


===== lDOS boot =====

The FAT32 + FSIBOOT boot sector loader was updated a little. [[https://hg.pushbx.org/ecm/ldosboot/rev/aae8332780e8|In a new revision of the experimental protocol]], the directory search function now returns with ds set equal to es (the directory buffer segment) upon success. A week later, I [[https://hg.pushbx.org/ecm/ldosboot/rev/dec4786efc0a|updated the FSIBOOT revision to the non-experimental FSIBOOT5]] and pushed the changes to both the ldosboot.exp and the ldosboot repos.


===== instsect =====

The only changes concern logging the dates of [[https://hg.pushbx.org/ecm/instsect/rev/14dac8f0e9c7|the FSIBEX05]] and [[https://hg.pushbx.org/ecm/instsect/rev/eacc2cdc10a9|FSIBOOT5 revisions]] of the two-stage FAT32 loaders.


===== heatshrink =====

[[https://github.com/ecm-pushbx/heatshrink/commit/1f752f7ce58afb44feb20104a00db5ff0626ddb7|Patch to add the -z switch]]:

<blockquote>Add -z switch (no initial window references) to encoder

As discussed on the SvarDOS issue tracker, refer to:
[[https://github.com/SvarDOS/bugz/issues/127#issuecomment-2386861614|SvarDOS/bugz#127 (comment)]]

<blockquote>Particular the store zeros calls concern me because they
may impose complications on the depacker, if it doesn't
use a circular buffer. I may look into adding a mode to
heatshrink that forbids references that point before the
depacked data.</blockquote>

For example, the lDOS inicomp depacker for heatshrink does some
special handling to store zeros. This code can be dropped if the
encoder's new -z switch is used:
https://hg.pushbx.org/ecm/inicomp/file/f5f72d7078a2/heatshr.asm#l443</blockquote>

Also [[https://github.com/atomicobject/heatshrink/pull/90|submitted as a Pull Request]], albeit it is unlikely to receive any attention.


===== lDebug =====

  * [[https://hg.pushbx.org/ecm/ldebug/rev/b6553d152eec|Add option INICOMP_HEATSHRINK_Z]] with auto-detection whether heatshrink supports the -z switch that I added in the heatshrink patch. This allows to enable the _Z define to inicomp's heatshrink depacker to optimise it, at very little cost to the size of the compressed payload.
  * [[https://hg.pushbx.org/ecm/ldebug/rev/0eb694db7a6c|Fix assembling of externally-built help pages]], ''ext_data_max_size_equate'' wasn't defined
  * [[https://hg.pushbx.org/ecm/ldebug/rev/80f7994fe8cf|Add handling of heatshrink -z switch]] to compressed help build in mak.sh
  * hshrink.asm: [[https://hg.pushbx.org/ecm/ldebug/rev/d228233cee10|Add _Z define for optimisation of help page depacker]]
  * [[https://hg.pushbx.org/ecm/ldebug/rev/75f746f31ec7|Support mvcomp inicomp method]] in mak.sh and [[https://hg.pushbx.org/ecm/ldebug/rev/8f87353aa6a0|in cfg.sh]]
  * [[https://hg.pushbx.org/ecm/ldebug/rev/8f87353aa6a0|Update cfg.sh to list]] both mvcomp and zerocomp

Other than [[https://hg.pushbx.org/ecm/ldebug/rev/e2f30c2944e7|an update to the news manual section]], all other changes concern the TSC Extension for lDebug ([[https://github.com/SvarDOS/edrdos/issues/116|created because of a request]] from [[https://github.com/boeckmann|@boeckmann]]):

  * [[https://hg.pushbx.org/ecm/ldebug/rev/ebc6fa50986c|Initial commit of the ELD]], featuring installable ELD, a dword variable LASTTSC, and two different commands, TSC (display TSC and write variable) and TSC QUIET (only write variable).
  * [[https://hg.pushbx.org/ecm/ldebug/rev/d9eb98e766a3|Add tsc.eld]] to source/eld/mak.sh and to extlib.asm
  * [[https://hg.pushbx.org/ecm/ldebug/rev/1afc9216abdf|Make LASTTSC an array variable]] to allow access to high dword
  * [[https://hg.pushbx.org/ecm/ldebug/rev/7de44d7d0da0|Add two patch areas to the debugger entry section]], in run (any mode) and intrtn (86 Mode only)
  * [[https://hg.pushbx.org/ecm/ldebug/rev/add833c457e3|Install tsc.eld handlers]] into the patch areas


===== mvcomp =====

The mvcomp algorithm is a simple, word-based compression method created for the SvarDOS multi-language support library. I contributed mvsize, a tool that reads in the original file and the compressed file, and returns how much buffer space is needed exactly to depack the packed data within an in-place depack buffer.

  * [[https://hg.pushbx.org/ecm/mvcomp/rev/61f748e8a075|Initial commit]], only depack once with a full buffer (not in-place depacking).
  * [[https://hg.pushbx.org/ecm/mvcomp/rev/6ffb58fb1838|Do binary search to find the minimum buffer size]] in the stubborn way, like inicomp test A mode.
  * [[https://hg.pushbx.org/ecm/mvcomp/rev/cef934136d38|Fix a wrong overflow check]].
  * [[https://hg.pushbx.org/ecm/mvcomp/rev/338250570cff|Fix wrong filenames in error messages]].
  * [[https://hg.pushbx.org/ecm/mvcomp/rev/fae4318abace|Change overflow checks to do the rounding first and check whether it results in a smaller value]], rather than hardcoding the maximum values.
  * [[https://hg.pushbx.org/ecm/mvcomp/rev/5b599752225e|Allow to specify an allocation unit]] (must be a power of 2) on the command line as a third parameter.
  * [[https://hg.pushbx.org/ecm/mvcomp/rev/04a4af663882|Allow empty string as third parameter]], for the default unit of 1 paragraph.
  * [[https://hg.pushbx.org/ecm/mvcomp/rev/0cc061c18c01|Calculate maxbytesahead variable during first depack run]], which should calculate the same buffer size as determined by the stubborn inicomp inspired test run.


I also [[https://hg.pushbx.org/ecm/mvcomp/rev/d49e7a66aeac|fixed a few typos in the mvcomp documentation]], and [[https://hg.pushbx.org/ecm/mvcomp/rev/f3107a7219c4|initially documented how I translated the svn repository into an hg repo]]. All other changesets consist of picks from the original mvcomp repo. In turn, mvcomp's repo picked my changes into [[https://sourceforge.net/p/mvcomp/code/HEAD/tree/|the original svn repository]].

The development of mvcomp mostly consisted of a back and forth [[https://github.com/SvarDOS/bugz/issues/127|in a SvarDOS issue related to their choice of TREE application]]. I do not care much for TREE, but I did participate in the talks about mvcomp.

One memorable part is [[https://github.com/SvarDOS/bugz/issues/127#issuecomment-2397315701|the in-place depack buffer size formula]] by [[https://github.com/mateuszviste|@mateuszviste]] (whose initials gave mvcomp its name). This formula says that the combined in-place buffer, including the initial prefix size (as given by my mvsize tool also) and the size of the compressed payload, can always be sized as ''UncompressedSize + UncompressedSize / 31 + 1''. This is a worst-case calculation, unlike mvsize it may result in allocating a larger buffer than needed. However, it appears to be a universal upper boundary for the needed buffer size.

[[https://github.com/SvarDOS/bugz/issues/127#issuecomment-2397384669|I didn't believe this at first]] when he shared it, but [[https://github.com/SvarDOS/bugz/issues/127#issuecomment-2400500892|did come around after some more discussion]]. In practice SvarLANG uses its packer's maxbytesahead approach which is equivalent to mvsize (and also done by the depacker in the latest mvsize revision).

{{tag>kernwrap edrdos inicomp ldosboot fsiboot instsect heatshrink ldebug mvcomp svardos}}


~~DISCUSSION~~