View Issue Details

IDProjectCategoryView StatusLast Update
0000400DCP-o-matic[All Projects] Featurespublic2020-07-05 22:32
Reportercarl Assigned Tocarl  
PriorityimmediateSeverityfeatureReproducibilityhave not tried
Status acknowledgedResolutionopen 
Product Version 
Target Version2.16.0Fixed in Version 
Summary0000400: Copy to external distribution drive
Description

[]

Tagsdisk, major, next
Branch
Estimated weeks required8
Estimated work requiredMajor

Relationships

parent of 0001730 resolvedcarl Need to build lwext4 static on Linux or rename the .so 
parent of 0001731 resolvedcarl Improve layout of conversational dialogues in _disk 
parent of 0001738 resolvedcarl Installing the macOS package does not kill an existing _writer process 
parent of 0001739 resolvedcarl macOS _writer needs to start on demand 
parent of 0001740 new Some uninstaller needed for macOS package 
parent of 0001741 new Test macOS with a mounted USB stick, trying to write to a second stick 
parent of 0001742 new Does disk fail with Paragon installed? 
parent of 0001743 resolvedcarl Improve progress estimation in disk 
parent of 0001744 resolvedcarl Unmount fails on Linux when a drive's name has spaces 
parent of 0001745 new disk testing 
parent of 0001746 new Progress reporting for disk writes on Linux is bad 
parent of 0001747 new Consider security of disk / disk_writer / nanomsg setup with malicious actors 
child of 0000624 acknowledged Wizard mode 
Not all the children of this issue are yet resolved or closed.

Activities

andrew.levine

2015-02-22 09:16

reporter   ~0000511

Nice idea :-)

carl

2019-03-12 22:19

administrator   ~0003139

https://www.ext2fsd.com/?tag=git
https://github.com/gkostka/lwext4

carl

2019-06-10 21:50

administrator   ~0003373

make mingw
cd build_mingw
make

builds lwext4 for Windows.

carl

2019-06-11 01:01

administrator   ~0003374

Last edited: 2019-12-02 15:36

View 4 revisions

Code that can make the partition table, filesystem & write a file in git:dcpdist. lwext4 pretty much works on Mac except that the call to ftello(dev_file) in file_dev_open returns 0 rather than the correct partition size - hardcoding that value seems to let it make a valid ext2 filesystem.

mhm

2019-10-24 23:13

reporter   ~0003502

The on disk filesystem should probably be ext3 for maximum compatibility. Please see:
https://www.isdcf.com/ISDCF_Public/HDD_Recommendations.html

Maybe it is already?

carl

2019-11-30 23:32

administrator   ~0003617

Last edited: 2019-12-01 14:30

View 2 revisions

Unknowns:

  • enumerating available drives (some work in hacks/drives.cc)
  • getting privileges

carl

2019-12-02 15:36

administrator   ~0003622

DoM branch is git:dist

carl

2020-01-20 00:14

administrator   ~0003716

Things to do:

  • Some UI for unmounting filesystems so they can be written to.
  • macOS / Windows support in the UI (getting available drives, and probably some other stuff).

carl

2020-03-16 23:37

administrator   ~0003752

Last edited: 2020-03-17 00:12

View 2 revisions

Improving the windows DE; exploring

cd ~/src
cdist -p dcpomatic -c dist --environment-version 2.15.x -t windows-64 -w dcpomatic-windows shell
cd dcpomatic-windows/src/dcpomatic
bash platform/windows/copy_deps.sh

Then on a VM with /home mounted as Z:

z:
cd \src\dcpomatic-windows\src\dcpomatic
platform\windows\set_paths.bat
build\src\tools\dcpomatic_dist.exe

carl

2020-04-28 21:05

administrator   ~0003786

Last edited: 2020-04-28 21:09

View 2 revisions

Seems to work but performance is terrible. Some notes...

The first slow thing is write_bgroups. It uses the block cache to write (I think) the inode and block bitmaps and the group descriptions into each block group. I don't think the block cache helps much here (BICBW). You can make the block cache larger (e.g. 65536) to make write_bgroups instantaneous but that just means you are delaying the write until ext4_block_cache_flush.

The second slow thing is init_bgs(), specifically the code which runs if ext4_bg_has_flag(bg, EXT4_BLOCK_GROUP_BLOCK_UNINIT) and ext4_bg_has_flag(bg, EXT4_BLOCK_GROUP_INODE_UNINIT). I'm not clear exactly what this is about or whether it's necessary.

I think the underlying problem is that writes are randomly spread over the disk and are done in small blocks. For example, the write_bgroups writes block group #0's bitmaps and then the group descriptor for block group #0 in block group 0000001, 0000002, 0000003, .... Then it does the same for group 0000001.

This might be mitigated by the block cache but that never seems to coalesce writes. So when you ext4_block_cache_flush it just dumps the blocks in any old order and does 512-byte fwrites. Sorting and coalescing a large cache seems like it might be slow. Doing that in ext4_block_cache_flush might be the easiest way to test this theory though.

It seems like the cleanest solution might be to remove the block cache altogether and just make sure that the whole of the block group "header" (i.e. group descriptions, block bitmap, inode bitmap, inode table) is written at the same time.

Or maybe ext4_block_cache_flush could walk lba_root, coalesecing and writing?

carl

2020-04-28 23:21

administrator   ~0003787

Last edited: 2020-04-28 23:23

View 2 revisions

Exploring the walk of lba_root but something is causing validation to fail; now running the same code with the old _flush to see if that fixes it. /dev/disk looks like it might be quicker than /dev/rdisk with the ordered writes.

carl

2020-05-02 17:33

administrator   ~0003793

Strangely, simply selecting a 4096 byte block size speeds up formatting a lot (hours to minutes). I tried most of the things suggested above:

  1. Walk lba_root rather than the other list, so that writes are ordered.
  2. Coalescing writes while doing this
  3. Using /dev/rdisk

and all of them either made no difference or made things worse.

Issue History

Date Modified Username Field Change
2014-08-11 17:24 carl New Issue
2014-09-12 23:04 carl Assigned To => carl
2014-09-12 23:04 carl Status new => acknowledged
2015-02-22 09:16 andrew.levine Note Added: 0000511
2015-06-12 11:20 carl Severity minor => feature
2015-06-12 15:32 carl Estimated work required => Major
2015-08-27 15:56 carl Target Version => 2.x
2017-05-12 22:11 carl Relationship added parent of 0000624
2019-01-30 00:56 carl Target Version => 2.16.0
2019-03-12 22:19 carl Note Added: 0003139
2019-06-10 21:50 carl Note Added: 0003373
2019-06-11 01:01 carl Note Added: 0003374
2019-10-24 21:57 carl Priority normal => urgent
2019-10-24 23:13 mhm Note Added: 0003502
2019-11-02 22:57 carl Tag Attached: next-2
2019-11-27 16:21 carl Tag Attached: major
2019-11-27 22:01 carl Estimated weeks required => 8
2019-11-30 23:25 carl Note Edited: 0003374 View Revisions
2019-11-30 23:32 carl Note Added: 0003617
2019-12-01 14:30 carl Note Edited: 0003617 View Revisions
2019-12-02 15:29 carl Note Edited: 0003374 View Revisions
2019-12-02 15:36 carl Note Edited: 0003374 View Revisions
2019-12-02 15:36 carl Note Added: 0003622
2019-12-17 20:57 carl Tag Detached: next-2
2019-12-17 20:57 carl Tag Attached: next
2020-01-20 00:14 carl Note Added: 0003716
2020-03-16 23:37 carl Note Added: 0003752
2020-03-17 00:12 carl Note Edited: 0003752 View Revisions
2020-04-12 22:11 carl Relationship added parent of 0001730
2020-04-12 22:12 carl Relationship added child of 0001731
2020-04-12 22:15 carl Relationship replaced parent of 0001731
2020-04-28 21:05 carl Note Added: 0003786
2020-04-28 21:09 carl Note Edited: 0003786 View Revisions
2020-04-28 23:21 carl Note Added: 0003787
2020-04-28 23:23 carl Note Edited: 0003787 View Revisions
2020-05-02 17:33 carl Note Added: 0003793
2020-05-02 17:39 carl Relationship deleted parent of 0000624
2020-05-02 17:39 carl Relationship added child of 0000624
2020-05-02 17:39 carl Relationship added parent of 0001738
2020-05-02 17:40 carl Relationship added parent of 0001739
2020-05-02 17:41 carl Relationship added parent of 0001740
2020-05-02 17:42 carl Relationship added parent of 0001741
2020-05-02 17:43 carl Relationship added parent of 0001742
2020-05-02 19:38 carl Relationship added parent of 0001743
2020-05-02 22:18 carl Relationship added parent of 0001744
2020-05-02 22:23 carl Relationship added parent of 0001745
2020-05-02 22:38 carl Relationship added parent of 0001746
2020-05-13 12:36 carl Relationship added parent of 0001747
2020-05-17 19:47 carl Tag Attached: required
2020-05-17 19:59 carl Tag Detached: required
2020-05-17 19:59 carl Priority urgent => immediate
2020-07-05 22:32 carl Tag Attached: disk