Problem statement

Because strings must end with '\0', CDIR_FILENAME_LEN (38) is 1 byte too small for a Git object file name.

We need to:

• Edit the TempleOS kernel
• Extend the FAT32 driver to allow for more LFNs

Problem statement: the plan

To write a Git object to disk in TempleOS:

1. Increase CDIR_FILENAME_LEN (38) by 1
2. An assertion in FileSysRedSea.HC must be commented out
3. RedSea FS must not be used on the system, no ISOs/RAM disks
4. FAT32 driver must be patched to support 1 more LFN record
5. A new kernel with these changes must be compiled (See /Adam/Opt/Boot/BootHDIns.HC)

TempleOS kernel hacking 101

1. Increase CDIR_FILENAME_LEN (38) by 1

In /Kernel/KernelA.HH:

- #define CDIR_FILENAME_LEN      38
+ #define CDIR_FILENAME_LEN      39

TempleOS kernel hacking 101

2. An assertion in FileSysRedSea.HC must be commented out

In /Kernel/BlkDev/FileSysRedSea.HC:

- #assert CDIR_SIZE==64
+ //#assert CDIR_SIZE==64

TempleOS kernel hacking 101

4. FAT32 driver must be patched to support 1 more LFN record

In /Kernel/BlkDev/FileSystemFAT.HC, edit:

• DirLongNameFill()
• FAT32DirFill()

Kernel hacking: FileSysFAT.HC

Look for mentions of "13" and "26"

switch (de->ord&0x3F) {
  case 1:
    for (i=0;i<13;i++)
      if (!(tmpde->name[i]=ptr[fat_long_name_map[i]]))
        return TRUE;
  break;
  case 2:
    for (i=0;i<12;i++)
      if (!(tmpde->name[i+13]=ptr[fat_long_name_map[i]]))
        return TRUE;
  break;
}

Kernel hacking: FileSysFAT.HC

Add the extra record:

  case 2:
-   for (i=0;i<12;i++)
+   for (i=0;i<13;i++)
      if (!(tmpde->name[i+13]=ptr[fat_long_name_map[i]]))
        return TRUE;
    break;
+   case 3:
+   for (i=0;i<13;i++)
+     if (!(tmpde->name[i+2*13]=ptr[fat_long_name_map[i]]))
+       return TRUE;
+   break;

Kernel hacking: FileSysFAT.HC

Look for mentions of "13" and "26"

if ((l=StrLen(tmpde->name))>13) {
  ...
  ld[de_cnt].attr=RS_ATTR_LONG_NAME;
  for (i=13;i<l;i++)
    ptr[fat_long_name_map[i-13]]=tmpde->name[i];
  i++;
  for (;i<26;i++)
    ptr[fat_long_name_map[i-13]](U16)=0xFFFF;
  ...

Kernel hacking: FileSysFAT.HC

Insert new logic before old name length check

+ l=StrLen(tmpde->name);
+  if (l>2*13) {
+  ...
+   for (i=2*13;i<l;i++)
+     ptr[fat_long_name_map[i-2*13]]=tmpde->name[i];
+   i++;
+   for (;i<3*13;i++)
+     ptr[fat_long_name_map[i-2*13]](U16)=0xFFFF;
+   l=2*13;
+ }
-  if ((l=StrLen(tmpde->name))>13) {
+  if (l>13) {

Kernel hacking: FileSysFAT.HC

A patched FileSysFAT.HC:

• Ugly but functional
• #define FAT32_VFAT_LEN 13
• Does not refactor existing driver code

DEFLATE

DEFLATE is designed for LZ77 compression, but supports uncompressed blocks as well.

• RFC-1950: zlib file format
• RFC-1951: DEFLATE algorithm

"Blocks" add headers and a checksum, "uncompressed" is null-compression

DEFLATE: zpipe

Use zpipe to read git objects without "git cat-file":

$ git cat-file -t e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
$ git cat-file -p e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 \
  | wc -c

Output:

blob
0

DEFLATE: zpipe

Use zpipe to read git objects without "git cat-file":

$ cd .git/objects/e6
$ zpipe -d < 9de29bb2d1d6434b8b29ae775ad8c2e48c5391 \
  | od -vAx -t x1z

Output:

000000 62 6c 6f 62 20 30 00 >blob 0.<
000007

DEFLATE: zpipe

Use zpipe to read git objects without "git cat-file":

$ cd .git/objects/cf
$ zpipe -d < 91034ac3780fd8b2392442a69430b6cef94cef \
  | od -vAx -t x1z

Output:

000000 62 6c 6f 62 20 37 00 48 69 20 6d 6f 6d 21 >blob 7.Hi mom!<
00000e

DEFLATE: Hacking zpipe

Hack zpipe.c to write uncompressed blocks:

- ret = def(stdin, stdout, Z_DEFAULT_COMPRESSION);
+ ret = def(stdin, stdout, 0);

• "non-compressible blocks are limited to 65,535 bytes"
• Cannot be chained like compressed blocks
• Equivalent to "git config core.loosecompression 0"

DEFLATE: Is a 65,535-byte limit a problem?

Find all files and lengths:

find . -type f -size +65535c -exec ls -l {} \; \
  | awk '{print $5 " " $NF}' | sort -rn

• The kernel and compiler binaries/maps are larger
• The only core files >65,535 in TempleOS are:
  • /Misc/Bible.TXT (4.2 MB)
  • /Adam/AutoComplete/ACWords.DATA (1.2 MB)
  • /Misc/PCIDevices.DD (349 KB)
  • /Kernel/KernelA.HH (91 KB)
• Unzip(,FUF_RECURSE);

Unzip() all the things

DEFLATE: Unzip() caveats

Run Unzip("C:/",FUF_RECURSE); so you can grep but remember:

• Edit /Adam/AutoComplete/ACTask.HC:

    } else if (FileFind("::/Doc/StandBy.DD.Z"))
      Type("::/Doc/StandBy.DD.Z",0);

• This keeps AutoComplete from being empty
• Or disable ACInit in HomeSys.HC

Searching HolyC code

Learn HolyC by reading HolyC. Search codebase with ripgrep and fzf:

set RG_PREFIX "rg --column --line-number \
  --no-heading --color=always --smart-case "

fzf --bind "change:reload:$RG_PREFIX {q} || true" \
        --ansi  | awk -F: '{print $1}' | xargs -r less

More on fzf: Bread on Penguins - "fzf, everyone's favorite fuzzy finder" - YouTube

Git.HC: Git("add","EmptyFile.TXT");

1. MAlloc() buffer == DEFLATE headers & checksum + 'blob 0\0' + file size
2. MemCpy() 'blob 0\0' & file data to buffer at DEFLATE header offset
3. SHA-1 Git object (Git header & data) with Sha1.HC
4. Write DEFLATE headers & checksum into buffer
5. Write buffer to disk with FileWrite()
6. Write file info to index (it's still complicated)

Git.HC: Read file with FileRead

U8 *buf, *header;
I64 size;

buf=FileRead("EmptyFile.TXT",&size);
header=MStrPrint("blob %d",size);

Git.HC: MAlloc object buffer

#define DEFLATE_CHKSUM_LEN     4
#define DEFLATE_HEADER         2
#define DEFLATE_BLOCK          5

I64 obj_len=StrLen(header)+1 + size + DEFLATE_CHKSUM_LEN + \
  DEFLATE_HEADER + DEFLATE_BLOCK;
U8 *obj=MAlloc(obj_len);

obj[0](U16)=0x0178; // RFC-1951 Deflate header
obj[DEFLATE_HEADER_LEN]=1;
obj[DEFLATE_HEADER_LEN+1]=size&0xFF;
obj[DEFLATE_HEADER_LEN+2]=(size>>8)&0xFF;
obj[DEFLATE_HEADER_LEN+3]=~obj[DEFLATE_HEADER_LEN+1];
obj[DEFLATE_HEADER_LEN+4]=~obj[DEFLATE_HEADER_LEN+2];

Git.HC: Write data to buffer

U8 *ptr;

ptr=obj + DEFLATE_HEADER_LEN + DEFLATE_BLOCK;
MemCpy(ptr,header,StrLen(header));
ptr+=StrLen(header);
*ptr++=0; // the '\0' in 'blob X\0'
MemCpy(ptr,buf,size);

Git.HC: Sha1.HC sold separately

#include "Sha1.HC";
Sha1Context ctx;
I64 i,j; U32 tmp;
U8 *digest_str,*digest_bytes;

Sha1Init(&ctx); Sha1Update(&ctx,obj + \
  DEFLATE_HEADER_LEN + DEFLATE_BLOCK,size+StrLen(blob));
digest_str=Sha1Final(&ctx);

for(i=0;i<5;i++) {
  tmp=EndianU32(ctx.state[i]);
  for(j=0;j<4;j++) {
    digest_bytes[(4*i)+j]=tmp&255; tmp>>=8;
  }
}

Git.HC: Write file to disk

U8 *dir=MStrPrint(".git/objects/%c%c/", digest_str[0], digest_str[1]);
U8 *file=MStrPrint("%s", digest_str+2);

DirMk(dir);
Cd(dir);
FileWrite(file,obj,obj_len);

But what does all of this look like?

Git on TempleOS: Limitations

• DEFLATE file size limit, no merging, no renaming
• No packing/unpacking, pushing/pulling
• Poor scaling: files only, no trees of trees
• FAT32 hack unstable, breaks RedSea, makes fsck dangerous
• "designed for _one_ thing, and one thing only: tracking a series of directory states in a way that can be replicated¹.... _really_ nasty"

¹ Except for mtimes, even though Git tracks them to the nanosecond

Git on TempleOS: Lamentations

What has Git done for TempleOS? Three "D"s:

• Desiring: I originally thought a source control tool on TempleOS would be useful
• Depressing: reading about Git internals was eye-opening, and the opposite of fun
• Demoralizing: building a Git client made me not want to use a Git client

This led me to ask myself a question

Git on TempleOS: Revelation

Why are we so dependent on Git? Common arguments:

• But Git is so [popular/fast/sophisticated]!
• But Git is familiar!
• But everyone uses the Git workflow!

Git is 20 years old. Did good software development practices not exist before 2005? What else is there?

We came in peace for all mankind

We came in peace for all mankind

34C3 - The Ultimate Apollo Guidance Computer Talk - YouTube

We came in peace for all mankind

"Three minutes before the lunar lander reached the Moon's surface... alarms were triggered. The [computer] was designed to support seven simultaneous programs running, but Aldrin’s request was the eighth. The result was a series of unexpected error codes during the live descent. The on-board flight software captured these alarms with the 'never supposed to happen displays'.... Hamilton had prepared for just this situation years before[.]"