Nice idea :-)

https://www.ext2fsd.com/?tag=git
https://github.com/gkostka/lwext4

make mingw
cd build_mingw
make

builds lwext4 for Windows.

Code that can make the partition table, filesystem & write a file in git:dcpdist. lwext4 pretty much works on Mac except that the call to ftello(dev_file) in file_dev_open returns 0 rather than the correct partition size - hardcoding that value seems to let it make a valid ext2 filesystem.

The on disk filesystem should probably be ext3 for maximum compatibility. Please see:
https://www.isdcf.com/ISDCF_Public/HDD_Recommendations.html

Maybe it is already?

Unknowns:

• enumerating available drives (some work in hacks/drives.cc)
• getting privileges

DoM branch is git:dist

Things to do:

• Some UI for unmounting filesystems so they can be written to.
• macOS / Windows support in the UI (getting available drives, and probably some other stuff).

Improving the windows DE; exploring

cd ~/src
cdist -p dcpomatic -c dist --environment-version 2.15.x -t windows-64 -w dcpomatic-windows shell
cd dcpomatic-windows/src/dcpomatic
bash platform/windows/copy_deps.sh

Then on a VM with /home mounted as Z:

z:
cd \src\dcpomatic-windows\src\dcpomatic
platform\windows\set_paths.bat
build\src\tools\dcpomatic_dist.exe

Seems to work but performance is terrible. Some notes...

The first slow thing is write_bgroups. It uses the block cache to write (I think) the inode and block bitmaps and the group descriptions into each block group. I don't think the block cache helps much here (BICBW). You can make the block cache larger (e.g. 65536) to make write_bgroups instantaneous but that just means you are delaying the write until ext4_block_cache_flush.

The second slow thing is init_bgs(), specifically the code which runs if ext4_bg_has_flag(bg, EXT4_BLOCK_GROUP_BLOCK_UNINIT) and ext4_bg_has_flag(bg, EXT4_BLOCK_GROUP_INODE_UNINIT). I'm not clear exactly what this is about or whether it's necessary.

I think the underlying problem is that writes are randomly spread over the disk and are done in small blocks. For example, the write_bgroups writes block group #0's bitmaps and then the group descriptor for block group #0 in block group 0000001, 0000002, 0000003, .... Then it does the same for group 0000001.

This might be mitigated by the block cache but that never seems to coalesce writes. So when you ext4_block_cache_flush it just dumps the blocks in any old order and does 512-byte fwrites. Sorting and coalescing a large cache seems like it might be slow. Doing that in ext4_block_cache_flush might be the easiest way to test this theory though.

It seems like the cleanest solution might be to remove the block cache altogether and just make sure that the whole of the block group "header" (i.e. group descriptions, block bitmap, inode bitmap, inode table) is written at the same time.

Or maybe ext4_block_cache_flush could walk lba_root, coalesecing and writing?

Exploring the walk of lba_root but something is causing validation to fail; now running the same code with the old _flush to see if that fixes it. /dev/disk looks like it might be quicker than /dev/rdisk with the ordered writes.

Strangely, simply selecting a 4096 byte block size speeds up formatting a lot (hours to minutes). I tried most of the things suggested above:

1. Walk lba_root rather than the other list, so that writes are ordered.
2. Coalescing writes while doing this
3. Using /dev/rdisk

and all of them either made no difference or made things worse.

Testing again on Mac with a 1TB external (spinning) USB drive, formatting is again very slow. Disk read/write use of about 150KB reported by activity monitor. Stuck in write_bgroups.
The call to write_bgroup_block (every 128 times round the loop) takes a long time.
Each write_bgroup_block call's loop runs 4623 times, each loop is quite quick.
So it looks like O(n^2) here, 4623/128 calls to write_bgroup_block, each does 4623 internal loops.
Every internal loop iteration calls set_direct
carl2-opt branch improves speed of write_bgroups; using /dev/rdisk along with this makes the write_bgroups much faster.
Now init_bgs is slow. Looks like it's walking the disk again initialising block and inode bitmaps - can we not do this in write_bgroups?

Tried to add this to write_bgroups in 6313cdffcc56683effb4d568b274e4ebbbab7353 but it sems to reduce the speed back to where it was; need to check the disk access patterns of the new write_bgroups.

Initialising the inode tables requires a lot of writes, and mke2fs does not do it unless lazy_itable_init is switched off (on by default); perhaps this is safe for us not to do also (though it does seem that kernel versions before 2.6.37 don't support lazy_itable_init [the kernel does the initialisation on first mount].

Another option might be to write the empty blocks using ext4_block_writebytes in ext4_fs_init_inode_table so that they're written in one lump. Not sure if this will improve things much.

Another possible optimisation: ext4_block_get_noread's call to ext4_block_cache_shake seems to cause a seek to 4096 and write for each block group when doing a mkfs.

The carl2-opt and disk-opt branches do:

• progress reporting during format
• don't initialise inode tables
• use writebytes in a couple of places
• write block group headers in more sequential/disk-friendly ways

Test case of copying 1.1Gb DCP from internal (spinning) disk to external (spinnnig) 2TB disk via USB on my old mac mini running Catalina.
Total time: 12:04
Format time: 6:09
Write time: 5:16
Verify time: 0:38

Activity monitor reports about 4MB/s data write during write part.
About 33MB/s data read during verify.

Extrapolating to 175GB DCP: 14h
Extrapolating to 75GB DCP: 6h

Should test writing speed of this drive when mac formatted / throughput described by activity monitor; odd that we're seeing 10 times worse write speed. Also need to look again at what blocks are actually getting read/written during the write part.

Did a bit of this looking and made ed2692cd03672bc11e4b6644378f07ce7fc02fc0 which needs testing. On each block we update things like the inode and superblock free blocks count because we turn off caching around sections where we write direct to the disk. Unless I am missing something, this seems unnecessary; it's right that the cache be used just for updating inode descriptors/superblocks etc. without being cleaned out by a large data block write. But I think this is achieved by the metadata-manipulation methods using via-cache write calls and the others using direct calls. BICBW.

That patch seems to make no real difference.

Needless to say macOS can copy the DCP to the save drive (macOS formatted) in about 10s with >100MB write speed reported.

Doing a simple stdio fwrite() in 4k blocks, 1GB, takes 10m57s with /dev/disk and 4m55s with /dev/rdisk suggesting that the ext2 code isn't doing so badly.

Using posix write() to /dev/rdisk, otherwise the same test takes 1m8s ~4x quicker.

POSIX: block_size=4096 blocks=262144 gap=0 rdisk=0 nocache=0 time=64.286583
POSIX: block_size=8192 blocks=131072 gap=0 rdisk=0 nocache=0 time=64.255241
POSIX: block_size=16384 blocks=65536 gap=0 rdisk=0 nocache=0 time=68.197258
POSIX: block_size=32768 blocks=32768 gap=0 rdisk=0 nocache=0 time=68.389603
POSIX: block_size=4096 blocks=262144 gap=0 rdisk=0 nocache=1 time=66.222176
POSIX: block_size=8192 blocks=131072 gap=0 rdisk=0 nocache=1 time=64.539406
POSIX: block_size=16384 blocks=65536 gap=0 rdisk=0 nocache=1 time=64.662483
POSIX: block_size=32768 blocks=32768 gap=0 rdisk=0 nocache=1 time=66.905128
POSIX: block_size=4096 blocks=262144 gap=0 rdisk=1 nocache=0 time=69.026535
POSIX: block_size=8192 blocks=131072 gap=0 rdisk=1 nocache=0 time=35.261715
POSIX: block_size=16384 blocks=65536 gap=0 rdisk=1 nocache=0 time=23.672134
POSIX: block_size=32768 blocks=32768 gap=0 rdisk=1 nocache=0 time=11.774684
POSIX: block_size=4096 blocks=262144 gap=0 rdisk=1 nocache=1 time=69.525856
POSIX: block_size=8192 blocks=131072 gap=0 rdisk=1 nocache=1 time=35.356491
POSIX: block_size=16384 blocks=65536 gap=0 rdisk=1 nocache=1 time=23.697132
POSIX: block_size=32768 blocks=32768 gap=0 rdisk=1 nocache=1 time=11.814719
POSIX: block_size=65536 blocks=16384 gap=0 rdisk=1 nocache=1 time=8.268696
POSIX: block_size=131072 blocks=8192 gap=0 rdisk=1 nocache=1 time=6.767652
POSIX: block_size=262144 blocks=4096 gap=0 rdisk=1 nocache=1 time=7.698212
POSIX: block_size=524288 blocks=2048 gap=0 rdisk=1 nocache=1 time=6.834740

• nocache seems to make no difference
• rdisk is much faster, larger blocks are much faster until you get to around 128k then it bottoms out

Format time: 03:07
Copy time: 00:21
Verify time: 00:55

Total 04:21