View Bug Details

IDProjectCategoryView StatusLast Update
0001757DCP-o-maticBugspublic2020-12-16 00:38
Reportercarl Assigned Tocarl  
PriorityimmediateSeverityminorReproducibilityhave not tried
Status closedResolutionfixed 
Target Version2.16.0 
Summary0001757: Writing to disk very slow on Windows
Description

e.g. estimates of 30 hours for a feature (115Gb)

Tagsdisk
Branch
Estimated weeks required
Estimated work requiredUndecided

Activities

carl

2020-08-04 23:50

administrator   ~0003887

Estimates of 2 hours for a trailer...

carl

2020-08-05 00:33

administrator   ~0003888

https://docs.microsoft.com/en-us/windows/win32/fileio/file-buffering

?

carl

2020-08-06 00:03

administrator   ~0003890

Last edited: 2020-08-06 00:04

  • Every call to ext4_fwrite seems to write some directory entry / superblock (or something) so if you write 4k blocks it would appear to write your data block and then 3 more housekeeping blocks; either need to pass as much as possible into ext4_fwrite or somehow arrange for the ext4_block_cache_write_back(..., 0) call to happen less often

  • It looks like the memory blocks passed to file_bwrite are not aligned to 4k boundaries, which the MS docs seem to suggest is bad for performance.

carl

2020-08-06 09:38

administrator   ~0003891

Last edited: 2020-08-06 22:55

  1. Made the fwrite() buffer aligned to 4096-byte boundary, ran a test of "BTaER" trailer onto DataTraveler SE9 -> 13:53 (although a Windows update was running at the time)
  2. Another run -> 14:01
  3. Same as 1/2 but with unaligned buffer -> 13:12; same
  4. Same as 1/2 but with a buffer 4 times bigger -> 12:55; a little faster.

NTFS copy DCP to stick and umount: 5:03
NTFS read back from stick to local drive: 1:10

so DoM is about half the speed.

carl

2020-08-09 23:47

administrator   ~0003892

Should be a lot better after 6c685e0cd6143dd7a8b1a9b5631b1bc9c0c4d687 but there's probably still room for improvement.

Bug History

Date Modified Username Field Change
2020-06-05 14:56 carl New Bug
2020-06-05 15:08 carl Description Updated
2020-06-05 15:08 carl Estimated work required => Undecided
2020-07-05 23:32 carl Tag Attached: disk
2020-08-04 23:50 carl Assigned To => carl
2020-08-04 23:50 carl Status new => confirmed
2020-08-04 23:50 carl Note Added: 0003887
2020-08-05 00:33 carl Note Added: 0003888
2020-08-06 00:03 carl Note Added: 0003890
2020-08-06 00:04 carl Note Edited: 0003890
2020-08-06 09:38 carl Note Added: 0003891
2020-08-06 10:36 carl Note Edited: 0003891
2020-08-06 11:07 carl Note Edited: 0003891
2020-08-06 21:34 carl Note Edited: 0003891
2020-08-06 21:59 carl Note Edited: 0003891
2020-08-06 21:59 carl Note Edited: 0003891
2020-08-06 22:55 carl Note Edited: 0003891
2020-08-09 23:47 carl Status confirmed => resolved
2020-08-09 23:47 carl Resolution open => fixed
2020-08-09 23:47 carl Note Added: 0003892
2020-12-16 00:38 carl Status resolved => closed