2024-12-04

I finally get to describe the porting work done on the fork of MS-DOS v4 some.

The final workflow went something like this:

• Set and export f variable to next object basename (uncapitalised, without filename extension)
• Run scriptlet to hg cp the original file, and commit with an appropriate message
• Run fixmem.pl on the file
• Observe any errors or messages from fixmem.pl
• Check results manually using hg d
• If fixmem.pl script needs to be edited, do that, then hg up -C and rerun fixmem.pl
• When satisfied, commit with a scriptlet that writes a changeset message and includes the current revision ID hash of fixmem
• Do some more scriptlets if needed, such as replacing the asmvars mmacros or convert an Installed equate to an %iassign
• Try to run ident86 with a -b switch that builds the new object using NASM (and links using ./mak.sh notracelist)
• Fix any causes of errors
• Run ident86 with -b -e -E -r -c to have it apply edits to source files as needed. For share.exe, also pass -Z 0x2642 or the like to detect differences due to relocations
• Observe if ident86 finds a difference that it cannot edit by itself, and fix
• Run ident86 without -b -E -r -c to do a full run comparing the files. Observe if any line differences remain.
• Run scriptlet that prepares a changeset message with the ident86 report and the filename, either titled "identicalise <file>" or "<file> is already identicalised".
• Copy the prepared changeset message using screen's copy mode.
• Run scriptlet that runs hg ci with a configuration parameter to allow empty commits, and then copies the new files (binary, .tls file, .map file) to the test subdirectory for working on the next object file.
• Run scriptlet to modify the makefile to refer to the new files (.nas, .mac)
• Watch the result with hg d
• Run hg ci with the message indicating the makefile is prepared to use the ported file
• Run a hg rm command to delete the original file (needed to cause nmake to pick up the correct ported file)
• Run hg ci with a message indicating the old file was deleted
• Go back and work on the next object file

• Set and export f variable to next object basename (uncapitalised, without filename extension)

f=sharesr
export f

• Run scriptlet to hg cp the original file, and commit with an appropriate message

hg cp src/CMD/SHARE/"${f^^}".ASM src/CMD/SHARE/"${f,,}".nas &&
  hg ci src/CMD/SHARE/"${f,,}".nas -m "copy ${f^^}.ASM to ${f,,}.nas, no change"
  f="${f,,}"

(Paths adjusted to work with src/DOS/, src/BIOS/, src/INC/, or src/CMD/SHARE/.) The last variable assignment was to change the $f variable from allcaps to all uncapitalised, which isn't needed any longer. All the uses of $f in this scriptlet force it allcaps or all uncapitalised using the bash variable modifying replacements.

• Copy .nas file to edit to a .tmp file to prepare for fixmem.pl, then run fixmem.pl on the files

cp -a src/CMD/SHARE/"${f,,}.nas" src/CMD/SHARE/"${f,,}.tmp" &&
  ~/proj/fixmem/fixmem.pl --no-istruc-labels \
  --quotemacros MSG_SERVICES -I src/INC/ -I src/CMD/SHARE/ \
  src/CMD/SHARE/"${f,,}.tmp" src/CMD/SHARE/"${f,,}.nas" \
  --segdefine psdata_seg --regdefine D_P_REG_BH_CG_SW,D_P_REG_BL_DQ_SW

The various switches:

--no-istruc-labels: disable old style writing dotted labels for istruc instances at a certain label

--quotemacros: needed for MSG_SERVICES mmacro to avoid clashing smacros and parameter names passed to the mmacro

-I: specify include directories to allow script to find files referenced in include directives

--segdefine and --regdefine: handle these comma-separated symbols as segregs or registers

Comparison, the scriptlet used to run fixmem.pl on an msbio object:

cp -a src/BIOS/"${f,,}.nas" src/BIOS/"${f,,}.tmp" &&
  ~/proj/fixmem/fixmem.pl --no-istruc-labels \
  --mactonas clocksub,ms96tpi,msbdata,msioctl,msstack,msvolid,readcloc,stkinit \
  -I src/INC/ -I src/BIOS/ src/BIOS/"${f,,}.tmp" src/BIOS/"${f,,}.nas" \
  --debug-include \
  --segdefine psdata_seg --regdefine D_P_REG_BH_CG_SW,D_P_REG_BL_DQ_SW

This additionally has the --mactonas switch to handle the files renamed from .INC to .nas to better reflect their use.

• Observe any errors or messages from fixmem.pl

• Check results manually using hg d

• If fixmem.pl script needs to be edited, do that, then hg up -C and rerun fixmem.pl

• When satisfied, commit with a scriptlet that writes a changeset message and includes the current revision ID hash of fixmem

hg ci -m "run fixmem.pl $(hg -R ~/proj/fixmem id -i) on $f.nas"

• Do some more scriptlets if needed, such as replacing the asmvars mmacros or convert an Installed equate to an %iassign

perl -i -ne '
if (/^\s*asmvars?\s+/i) {
  s/asmvars/asmvar/i;
  s/[\>\<]+//g;
  while (s/^(\s*asmvar\s+)([A-Za-z0-9_]+)\s*?(?:,(.*?))?([\r\n]*)$/$1$3$4/i) {
    print STDERR ">>$_<<\n";
    print "%ifndef ".$2.$4;
    print " \%iassign $2 0".$4;
    print "%endif".$4;
  };
} else {
  print;
}' src/CMD/SHARE/*.nas
hg d
hg ci -m "$f.nas: replace asmvar macro use"
perl -i -pe 's/^\s*(Installed)\s+equ\b/%iassign $1/i' src/CMD/SHARE/"$f".nas
hg d
hg ci -m "$f.nas: replace Installed equate by %iassign"

• Try to run ident86 with a -b switch that builds the new object using NASM (and links using ./mak.sh notracelist)

DEBUG=0 time -p ~/proj/ident86/ident86.py test/share.exe src/CMD/SHARE/share.exe \
  src/CMD/SHARE/share.tls src/CMD/SHARE/sharew.map -s -a 256 -S -b '
  rm -f src/CMD/SHARE/share.exe &&
  cdexec src/CMD/SHARE/ nasm -Ox -P../../INC/nasm.mac -I../../INC/ -I../../DOS/ "$f".nas \
  -fobj -o"$f".obj -l"$f".lst -I ../lmacros/ -Lp &&
  ./mak.sh notracelist &&
  ./makshrtl.sh' -e 2 -d 16 -p '\.lst$::.nas' -p '\\::/' -p '^::src/CMD/SHARE/' \
  -E -c "$f".20241201.cki -f 0x60 -r -z -Z 0x2642; beep

The various switches:

-s: side by side view, very useful (and may be needed for editing)

-a 256: specify auto difference length so as to stop listing different bytes after a certain length in a single difference range is reached

-S: show spots to edit

-b: specify build scriptlet to run so as to rebuild with the object file that is to be inspected currently

-e 2: indicate second binary file is the one being edited

-d 16: have ident86 stop displaying after this many bytes displayed after the "earliest difference" (line that is not samesame and not fuzzy same either)

-p: specify patterns to convert .tls listing source to an assembly source text file name

-E: enable automatic edit

-c: specify a cookie file to speed up finding more differences in subsequent iterations

-r: repeat run after an edit has been applied

-z: ignore byte differences in header

-Z 0x2642: apply relocation to the specified segment in order to find differences due to relocations

The build scriptlet is as follows:

'
  rm -f src/CMD/SHARE/share.exe &&
  cdexec src/CMD/SHARE/ nasm -Ox -P../../INC/nasm.mac -I../../INC/ -I../../DOS/ "$f".nas \
  -fobj -o"$f".obj -l"$f".lst -I ../lmacros/ -Lp &&
  ./mak.sh notracelist &&
  ./makshrtl.sh'

rm: make sure the executable is deleted so it won't be found if nmake fails to build it

cdexec: run assembler in the specified subdirectory

-P../../INC/nasm.mac: pre-include the NASM conversion macros file (based on fixmem's nasm.mac)

-I: specify include paths to NASM

-fobj: specify OMF output format for NASM

-l: create a listing file, needed for convlist.pl and ident86

-Lp: create a listing file even if errors occurred, to help debugging

./mak.sh notracelist: run nmake to link the final executable with the object file generated by NASM. Specifies notracelist to avoid creating the trace listing files not needed, which take a long time to generate.

./makshrtl.sh: create the single trace listing file needed for ident86 (this one for share, ./makbiotl.sh for msbio and ./makdostl.sh for msdos)

• Fix any causes of errors

• Run ident86 with -b -e -E -r -c to have it apply edits to source files as needed. For share.exe, also pass -Z 0x2642 or the like to detect differences due to relocations

As above. The NASM build step should succeed in every iteration of this loop, having fixed the errors previously.

• Observe if ident86 finds a difference that it cannot edit by itself, and fix

This is usually short jumps that straddle the rel8 displacement limit. Sometimes mistakes during the fixmem.pl step show up here as well.

• Run ident86 without -b -E -r -c to do a full run comparing the files. Observe if any line differences remain.

DEBUG=0 time -p ~/proj/ident86/ident86.py \
  test/share.exe src/CMD/SHARE/share.exe \
  src/CMD/SHARE/share.tls src/CMD/SHARE/sharew.map \
  -s -a 256 -S -e 2 -d 16 -p '\.lst$::.nas' -p '\\::/' -p '^::src/SHARE/' |
  tee report.txt; beep

Alternative, using the -I switch added to ident86 later:

DEBUG=0 time -p ~/proj/ident86/ident86.py \
  test/share.exe src/CMD/SHARE/share.exe \
  src/CMD/SHARE/share.tls src/CMD/SHARE/sharew.map \
  -s -a 256 -S -e 2 -d 16 -p '\.lst$::.nas' -p '\\::/' \
  -c "$f".20241201.cki -f 0x60 -z -Z 0x2642 \
  -I src/CMD/SHARE/ -I src/INC/; beep

In this run the last -p pattern is omitted so that source text files are not always searched for in the src/SHARE/ subdirectory. Instead, the -I switches are given in order to specify what pathes to search for the source text files.

• Run scriptlet that prepares a changeset message with the ident86 report and the filename, either titled "identicalise <file>" or "<file> is already identicalised".

if [ -z "$(hg d)" ]
then
  echo -ne "$f.nas is already identicalised\n\nFull ident86 report:\n\n"
else
  echo -ne "identicalise $f.nas\n\nFull ident86 report:\n\n"
fi
cat report.txt |
  grep -vE '[0-9A-Fa-f]{6}  first:.*second:[0-9A-F]{2}|first:00 != second:eof|[0-9A-Fa-f]{6} up to below|no difference' |
  tail -n20

The ident86 report includes the used ident86 revision's ID hash and the lDebug version.

• Copy the prepared changeset message using screen's copy mode.

• Run scriptlet that runs hg ci with a configuration parameter to allow empty commits, and then copies the new files (binary, .tls file, .map file) to the test subdirectory for working on the next object file.

hg ci --config ui.allowemptycommit=1 && cp -at test src/CMD/SHARE/{share.exe,share.tls,sharew.map}

• Run scriptlet to modify the makefile to refer to the new files (.nas, .mac)

l="clocksub ms96tpi msbdata msioctl msstack msvolid readcloc stkinit"
for g in $l
do
  perl -i -ne '
  if (/^\s*'"$f"'\.obj:/) { $replace = 1; };
  if (defined $replace and $replace) {
    s/\.asm/.nas/g;
    s/\.inc/.mac/g;
    s/'"$g"'\.mac/'"$g"'\.nas/g;
    print;
    if (not /\\[\r\n]*$/) { $replace = 0; };
  } else {
    print;
  }' src/CMD/SHARE/MAKEFILE
done

The $l list is for the msbio files that were renamed from .INC filenames to .nas ones to better reflect their use. The $replace variable makes it so subsequent lines in the makefile, after a trailing backslash, are also processed. Only after a line that does not end in a backslash, the scriptlet will not replace extensions any longer.

• Watch the result with hg d

hg d

• Run hg ci with the message indicating the makefile is prepared to use the ported file

hg ci -m "prepare makefile to use $f.nas ported file"

• Run a hg rm command to delete the original file (needed to cause nmake to pick up the correct ported file)

hg rm src/CMD/SHARE/"${f^^}.ASM"

• Run hg ci with a message indicating the old file was deleted

hg ci -m "$f.asm: delete original file"

• Go back and work on the next object file