|
|
|
|
— |
blog:pushbx:2024:1013_late_september_early_october_work [2024-10-13 17:59:13 +0200 Oct Sun] (current)
ecm created |
| | ====== Late September, early October work ====== |
| | |
| | **2024-10-13** |
| | |
| | Last Sunday (2024-10-06) I travelled back from my partner's place, so I didn't get to write a blog post. Therefore we'll cover two weeks' worth of work today. |
| | |
| | |
| | ===== kernwrap ===== |
| | |
| | All of these changes were picked from lDebug. |
| | |
| | * [[https://hg.pushbx.org/ecm/kernwrap/rev/353429a4bc27|Add support for INICOMP_HEATSHRINK_Z]] to optimise heatshrink depacker |
| | * [[https://hg.pushbx.org/ecm/kernwrap/rev/c9e1c56f69c3|Support mvcomp compression method]] |
| | * [[https://hg.pushbx.org/ecm/kernwrap/rev/7a7f2ab4d00e|Update config file]] to add zerocomp and mvcomp compression methods |
| | |
| | |
| | ===== Enhanced DR-DOS ===== |
| | |
| | Prompted by [[https://github.com/SvarDOS/edrdos/issues/116|a SvarDOS issue for the EDR-DOS kernel]], I [[https://hg.pushbx.org/ecm/edrdos/rev/374911957af5|optimised the bdevio.nas function div64]]. I'd noticed that its stack layout exactly matches the div32 function. The div64 function implements a very slow bitwise loop to do the division with up to 64-bit dividend and up to 32-bit divisor. My optimisation consists of checking the upper 32 bits of the dividend; if they're zero, pass control to the div32 function instead. This causes a great speedup apparently. |
| | |
| | I later [[https://hg.pushbx.org/ecm/edrdos/rev/529340d7b94d|picked a comment]] from [[https://github.com/SvarDOS/edrdos/commit/33c091a5d3c2f7dac1648844cf43f524e67e2f62|the SvarDOS repo's pick of my changeset]]. |
| | |
| | [[https://hg.pushbx.org/ecm/edrdos/rev/1c95556ed6bc|The only unrelated changeset]] in our EDR-DOS repo is [[https://github.com/SvarDOS/edrdos/commit/9b5332b28ce62849c0b24a0ac75182206109d270|a pick from SvarDOS]] to supply a video page and attribute for a few int 10.0E calls in drbio/console.nas (as usual I picked the change in the JWasm source texts into both that and the NASM ported sources for the lDOS repo). ([[https://github.com/SvarDOS/edrdos/issues/117|Issue at SvarDOS EDR-DOS repo]].) |
| | |
| | |
| | ===== inicomp ===== |
| | |
| | * [[https://hg.pushbx.org/ecm/inicomp/rev/dd81110642e4|Add testfile application that uses the inicomp depackers]] to depack a single file, and write to an output file. This involves passing the depacked size from the depack function to the caller. |
| | * [[https://hg.pushbx.org/ecm/inicomp/rev/0b9dc801dcb8|Add -v switch to testfile]] for verbose debugging output. |
| | * In heatshrink depacker verbose output [[https://hg.pushbx.org/ecm/inicomp/rev/e60c60b1a59e|display destination and source offsets]]. |
| | * In heatshrink depacker [[https://hg.pushbx.org/ecm/inicomp/rev/f5f72d7078a2|add verbose message for storing zeroes]] (backreference matches that reference the initial window before the start of the uncompressed data). |
| | * [[https://hg.pushbx.org/ecm/inicomp/rev/13563a9b6e28|Add -e switch to testfile]] for exceptional debugging messages. (The -v switch enables a superset of the -e switch messages.) |
| | * [[https://hg.pushbx.org/ecm/inicomp/rev/adbcba7c1d12|Fix heatshrink depacker to work in the inicomp stage again]], it had the _TESTFILE define too late. |
| | * [[https://hg.pushbx.org/ecm/inicomp/rev/04de03dc0726|Optimise heatshrink depacker with _Z define]] to drop code for storing zeroes for initial window backreferences, shrinking the depacker somewhat. |
| | |
| | |
| | The remaining changes involve the mvcomp compression method, based on SvarDOS's language file packer: |
| | |
| | * [[https://hg.pushbx.org/ecm/inicomp/rev/bc45b98b99c5|Add mvcomp depacker]]. |
| | * [[https://hg.pushbx.org/ecm/inicomp/rev/71ddecbbb8dd|Support _MVCOMP define]] for mvcomp depacker. |
| | * Optimise mvcomp depacker: |
| | * [[https://hg.pushbx.org/ecm/inicomp/rev/80102d0ff030|Inline get_word]] as only a single spot ever called it. |
| | * In copy_data [[https://hg.pushbx.org/ecm/inicomp/rev/b38ab09d4e3d|only support lengths < 256 Bytes]]. |
| | * [[https://hg.pushbx.org/ecm/inicomp/rev/49968d4ec942|Drop unused .end_check code]]. |
| | * [[https://hg.pushbx.org/ecm/inicomp/rev/0b1c60bfe1df|Double the number of progress dots]] emitted by mvcomp depacker. |
| | * [[https://hg.pushbx.org/ecm/inicomp/rev/6e6e0530b90f|Add testfile verbose output]] for matches to mvcomp depacker. |
| | |
| | |
| | ===== lDOS boot ===== |
| | |
| | The FAT32 + FSIBOOT boot sector loader was updated a little. [[https://hg.pushbx.org/ecm/ldosboot/rev/aae8332780e8|In a new revision of the experimental protocol]], the directory search function now returns with ds set equal to es (the directory buffer segment) upon success. A week later, I [[https://hg.pushbx.org/ecm/ldosboot/rev/dec4786efc0a|updated the FSIBOOT revision to the non-experimental FSIBOOT5]] and pushed the changes to both the ldosboot.exp and the ldosboot repos. |
| | |
| | |
| | ===== instsect ===== |
| | |
| | The only changes concern logging the dates of [[https://hg.pushbx.org/ecm/instsect/rev/14dac8f0e9c7|the FSIBEX05]] and [[https://hg.pushbx.org/ecm/instsect/rev/eacc2cdc10a9|FSIBOOT5 revisions]] of the two-stage FAT32 loaders. |
| | |
| | |
| | ===== heatshrink ===== |
| | |
| | [[https://github.com/ecm-pushbx/heatshrink/commit/1f752f7ce58afb44feb20104a00db5ff0626ddb7|Patch to add the -z switch]]: |
| | |
| | <blockquote>Add -z switch (no initial window references) to encoder |
| | |
| | As discussed on the SvarDOS issue tracker, refer to: |
| | [[https://github.com/SvarDOS/bugz/issues/127#issuecomment-2386861614|SvarDOS/bugz#127 (comment)]] |
| | |
| | <blockquote>Particular the store zeros calls concern me because they |
| | may impose complications on the depacker, if it doesn't |
| | use a circular buffer. I may look into adding a mode to |
| | heatshrink that forbids references that point before the |
| | depacked data.</blockquote> |
| | |
| | For example, the lDOS inicomp depacker for heatshrink does some |
| | special handling to store zeros. This code can be dropped if the |
| | encoder's new -z switch is used: |
| | https://hg.pushbx.org/ecm/inicomp/file/f5f72d7078a2/heatshr.asm#l443</blockquote> |
| | |
| | Also [[https://github.com/atomicobject/heatshrink/pull/90|submitted as a Pull Request]], albeit it is unlikely to receive any attention. |
| | |
| | |
| | ===== lDebug ===== |
| | |
| | * [[https://hg.pushbx.org/ecm/ldebug/rev/b6553d152eec|Add option INICOMP_HEATSHRINK_Z]] with auto-detection whether heatshrink supports the -z switch that I added in the heatshrink patch. This allows to enable the _Z define to inicomp's heatshrink depacker to optimise it, at very little cost to the size of the compressed payload. |
| | * [[https://hg.pushbx.org/ecm/ldebug/rev/0eb694db7a6c|Fix assembling of externally-built help pages]], ''ext_data_max_size_equate'' wasn't defined |
| | * [[https://hg.pushbx.org/ecm/ldebug/rev/80f7994fe8cf|Add handling of heatshrink -z switch]] to compressed help build in mak.sh |
| | * hshrink.asm: [[https://hg.pushbx.org/ecm/ldebug/rev/d228233cee10|Add _Z define for optimisation of help page depacker]] |
| | * [[https://hg.pushbx.org/ecm/ldebug/rev/75f746f31ec7|Support mvcomp inicomp method]] in mak.sh and [[https://hg.pushbx.org/ecm/ldebug/rev/8f87353aa6a0|in cfg.sh]] |
| | * [[https://hg.pushbx.org/ecm/ldebug/rev/8f87353aa6a0|Update cfg.sh to list]] both mvcomp and zerocomp |
| | |
| | Other than [[https://hg.pushbx.org/ecm/ldebug/rev/e2f30c2944e7|an update to the news manual section]], all other changes concern the TSC Extension for lDebug ([[https://github.com/SvarDOS/edrdos/issues/116|created because of a request]] from [[https://github.com/boeckmann|@boeckmann]]): |
| | |
| | * [[https://hg.pushbx.org/ecm/ldebug/rev/ebc6fa50986c|Initial commit of the ELD]], featuring installable ELD, a dword variable LASTTSC, and two different commands, TSC (display TSC and write variable) and TSC QUIET (only write variable). |
| | * [[https://hg.pushbx.org/ecm/ldebug/rev/d9eb98e766a3|Add tsc.eld]] to source/eld/mak.sh and to extlib.asm |
| | * [[https://hg.pushbx.org/ecm/ldebug/rev/1afc9216abdf|Make LASTTSC an array variable]] to allow access to high dword |
| | * [[https://hg.pushbx.org/ecm/ldebug/rev/7de44d7d0da0|Add two patch areas to the debugger entry section]], in run (any mode) and intrtn (86 Mode only) |
| | * [[https://hg.pushbx.org/ecm/ldebug/rev/add833c457e3|Install tsc.eld handlers]] into the patch areas |
| | |
| | |
| | ===== mvcomp ===== |
| | |
| | The mvcomp algorithm is a simple, word-based compression method created for the SvarDOS multi-language support library. I contributed mvsize, a tool that reads in the original file and the compressed file, and returns how much buffer space is needed exactly to depack the packed data within an in-place depack buffer. |
| | |
| | * [[https://hg.pushbx.org/ecm/mvcomp/rev/61f748e8a075|Initial commit]], only depack once with a full buffer (not in-place depacking). |
| | * [[https://hg.pushbx.org/ecm/mvcomp/rev/6ffb58fb1838|Do binary search to find the minimum buffer size]] in the stubborn way, like inicomp test A mode. |
| | * [[https://hg.pushbx.org/ecm/mvcomp/rev/cef934136d38|Fix a wrong overflow check]]. |
| | * [[https://hg.pushbx.org/ecm/mvcomp/rev/338250570cff|Fix wrong filenames in error messages]]. |
| | * [[https://hg.pushbx.org/ecm/mvcomp/rev/fae4318abace|Change overflow checks to do the rounding first and check whether it results in a smaller value]], rather than hardcoding the maximum values. |
| | * [[https://hg.pushbx.org/ecm/mvcomp/rev/5b599752225e|Allow to specify an allocation unit]] (must be a power of 2) on the command line as a third parameter. |
| | * [[https://hg.pushbx.org/ecm/mvcomp/rev/04a4af663882|Allow empty string as third parameter]], for the default unit of 1 paragraph. |
| | * [[https://hg.pushbx.org/ecm/mvcomp/rev/0cc061c18c01|Calculate maxbytesahead variable during first depack run]], which should calculate the same buffer size as determined by the stubborn inicomp inspired test run. |
| | |
| | |
| | I also [[https://hg.pushbx.org/ecm/mvcomp/rev/d49e7a66aeac|fixed a few typos in the mvcomp documentation]], and [[https://hg.pushbx.org/ecm/mvcomp/rev/f3107a7219c4|initially documented how I translated the svn repository into an hg repo]]. All other changesets consist of picks from the original mvcomp repo. In turn, mvcomp's repo picked my changes into [[https://sourceforge.net/p/mvcomp/code/HEAD/tree/|the original svn repository]]. |
| | |
| | The development of mvcomp mostly consisted of a back and forth [[https://github.com/SvarDOS/bugz/issues/127|in a SvarDOS issue related to their choice of TREE application]]. I do not care much for TREE, but I did participate in the talks about mvcomp. |
| | |
| | One memorable part is [[https://github.com/SvarDOS/bugz/issues/127#issuecomment-2397315701|the in-place depack buffer size formula]] by [[https://github.com/mateuszviste|@mateuszviste]] (whose initials gave mvcomp its name). This formula says that the combined in-place buffer, including the initial prefix size (as given by my mvsize tool also) and the size of the compressed payload, can always be sized as ''UncompressedSize + UncompressedSize / 31 + 1''. This is a worst-case calculation, unlike mvsize it may result in allocating a larger buffer than needed. However, it appears to be a universal upper boundary for the needed buffer size. |
| | |
| | [[https://github.com/SvarDOS/bugz/issues/127#issuecomment-2397384669|I didn't believe this at first]] when he shared it, but [[https://github.com/SvarDOS/bugz/issues/127#issuecomment-2400500892|did come around after some more discussion]]. In practice SvarLANG uses its packer's maxbytesahead approach which is equivalent to mvsize (and also done by the depacker in the latest mvsize revision). |
| | |
| | {{tag>kernwrap edrdos inicomp ldosboot fsiboot instsect heatshrink ldebug mvcomp svardos}} |
| | |
| | |
| | ~~DISCUSSION~~ |
| |
