View Bug Details

IDProjectCategoryView StatusLast Update
0002555DCP-o-maticBugspublic2023-07-05 13:58
Reporteroverlookmotel Assigned Tocarl  
PrioritynormalSeverityminorReproducibilityhave not tried
Status resolvedResolutionfixed 
PlatformMacOSOS XOS Version10.14
Product Version2.16.57 
Summary0002555: dcpomatic2_map takes a long time

I just tried out dcpomatic2_map in 2.16.59 on a similar DCP to the one described in 0002542. It's a 96-minute long feature, split into 3 reels.

I'd expected it to be almost instantaneous, as was using -l option to make hard links rather than copying files. But it was taking much longer than this (not sure how long, I gave up after 10 mins or so).

The CPL and MXF files appeared in the output folder immediately, but the ASSETMAP, PKL and VOLINDEX were not present.

It was using about 30% of one CPU core, so it was doing something, didn't seem to have hung.

I'm wondering: Does it calculate hashes of the MXF assets to put in the new PKL?

If so, would it be possible to just take the already-calculated hashes from the ASSETMAPs/PKLs of the input DCPs, rather than calculating them again from scratch? This would, I assume, make the process almost instantaneous.

TagsNo tags attached.
Branchdont-hash (in dom) dont-rehash (libdcp)
Estimated weeks required
Estimated work requiredUndecided



2023-06-08 15:10

reporter   ~0005740

One further thought: I also think it'd be more bulletproof from a content integrity point of view to use existing hashes rather than recalculating them.

For example: Let's say a drive one of the input DCPs is on is faulty and some of the original DCP's data has been corrupted. If the new PKL includes the original hashes from the source PKL, you'll end up with a DCP which has hashes in PKL that don't match the actual content of the files. Which is what you want, because the content has been corrupted, and best outcome is that the server refuses to ingest the DCP, so you're made aware and can get it fixed.

If DOM recalculates the hash from the data on drive, it'll produce a DCP which will pass hash check (because newly calculated hashes match the actual hash of the corrupt data), will happily ingest, but may fail in playback when the server hits the corrupt data.

This scenario isn't entirely academic. We've seen this happen in the wild when someone sought to "fix" a DCP which was failing hash check by just bunging the MXFs into DCP-o-matic and asking it to "rewrap" the content. That produced a new DCP with hashes that matched the content, and so would ingest successfully. But of course this just disguised the corruption, rather than actually fixing it, and the projector conked out halfway through the show.


2023-06-08 15:13

reporter   ~0005741

By the way, sorry for the recent flurry of bug reports. I'm looking to upgrade from 2.14.x to 2.16.x in the next few months, so have been testing things out prior to that.


2023-06-20 12:56

reporter   ~0005766

I agree that dcpomatic2_map should use existing hashes rather then recalculate them.

Also please prioritize using the hashes in the CPL if present (they are optional) as this information are more persistent. ASSETMAP, PKL and VOLINDEX are mainly used for transport and are removed/regenerated on ingest on some servers.

Whereas CPL:s are always kept unchanged, so if hashes are used in the CPL they should have priority.


2023-06-21 15:47

reporter   ~0005771

Please also take a look at the additional suggestions in 0002445


2023-06-29 00:51

administrator   ~0005797

@carl 53b3783a9d493dae07c51d6b99cd238bfcfca5b5


2023-06-29 12:15

reporter   ~0005798

Thanks Carl. Is there a test release I can use to experiment with this change and confirm resolution?


2023-06-30 23:31

administrator   ~0005805

@overlookmotel I can make one - is AppImage best for you?


2023-07-01 17:14

reporter   ~0005806

Thanks Carl. A Mac OS build (10.14) would be preferable if possible.


2023-07-01 21:55

administrator   ~0005807

Let me know if there's any funny business with this build and macOS' "checking" of apps as I had to change how this is done recently and it's hard to know whether it's working or not.


2023-07-02 11:24

reporter   ~0005815

On Mac OS 10.14.6, DOM GUI in the test build opens fine. "Verifying" takes a while (0000001:0000001 min) but that's I don't think unusual vs other DOM versions. I haven't tested any of the other applications - let me know if you'd like me to.

I'll test out dcpomatic2_map later this week.


2023-07-02 11:25

reporter   ~0005816

Mantis! Why you mangle my text? That meant to say verifying takes approx 1 min.


2023-07-05 13:57

reporter   ~0005842

Confirmed fixed in test build d21efc8.


2023-07-05 13:58

administrator   ~0005843

Thanks for checking it out.

Bug History

Date Modified Username Field Change
2023-06-08 03:01 overlookmotel New Bug
2023-06-08 15:10 overlookmotel Note Added: 0005740
2023-06-08 15:13 overlookmotel Note Added: 0005741
2023-06-20 12:56 mhm Note Added: 0005766
2023-06-20 21:08 carl Branch => dont-hash (in dom and libdcp)
2023-06-20 21:08 carl Estimated work required => Undecided
2023-06-20 21:08 carl Assigned To => carl
2023-06-20 21:08 carl Status new => confirmed
2023-06-21 15:47 mhm Note Added: 0005771
2023-06-25 23:05 carl Branch dont-hash (in dom and libdcp) => dont-rehash (in dom and libdcp)
2023-06-26 23:16 carl Branch dont-rehash (in dom and libdcp) => dont-hash (in dom) dont-rehash (libdcp)
2023-06-29 00:51 carl Note Added: 0005797
2023-06-29 00:51 carl Status confirmed => resolved
2023-06-29 00:51 carl Resolution open => fixed
2023-06-29 12:15 overlookmotel Note Added: 0005798
2023-06-30 23:31 carl Note Added: 0005805
2023-07-01 17:14 overlookmotel Note Added: 0005806
2023-07-01 21:55 carl Note Added: 0005807
2023-07-02 11:24 overlookmotel Note Added: 0005815
2023-07-02 11:25 overlookmotel Note Added: 0005816
2023-07-05 13:57 overlookmotel Note Added: 0005842
2023-07-05 13:58 carl Note Added: 0005843