View Bug Details

IDProjectCategoryView StatusLast Update
0001380DCP-o-maticFeaturespublic2021-05-23 00:02
Reportercarl Assigned To 
PriorityhighSeverityminorReproducibilityhave not tried
Status acknowledgedResolutionopen 
Summary0001380: Could check MXF hashes from info file as the CPL/PKL hashes are being calculated
Description

Which would detect corruption between writing the MXF to disk and reading it back.

Tagscorrectness
Branch
Estimated weeks required
Estimated work requiredMedium

Relationships

related to 0001381 closedcarl Hash/checksum data transferred over the network for encoding servers 

Activities

overlookmotel

2018-10-16 11:07

reporter   ~0002716

Thanks for creating this issue.

Personally, I think this would be a good idea. Much as you can't trust the network to transmit data 100% correctly, I don't think the filesystem should be trusted either. It's mostly trustworthy, but to be bullet-proof, it'd be ideal if the hashes were checked when the files are read back from the drive.

Would it be difficult to implement?

For anyone who comes across this issue, the context is a conversation on the DCP-o-matic mailing list. I made a large batch of DCPs but discovered that the drive I made them on was faulty and was corrupting data. Because DOM writes the MXF files to disc and then reads them back again to creates the hashes which are stored in the DCP XML, if the data has been corrupted on the disc in the meantime, that corruption would be "baked in" to the DCP's MXF files and the hashes would reflect the corrupted data. So hash-checking the DCP would not reveal whether this has happened or not.

carl

2018-10-17 10:33

administrator   ~0002719

Probably not too difficult. Maybe could be optional with 0001381 if performance is a problem.

carl

2020-04-17 00:00

administrator   ~0003764

@carl tricky part is that the CPL/PKL hashes are being calculated in libdcp (dcp::make_digest) so to save doing the disk IO twice we'd need some way for libdcp to check the data as well as calculating its checksum.

Perhaps a functor (passed in to Asset::hash and hence into make_digest) which gets given the block and can throw exceptions (?!) if things aren't right.

carl

2020-04-17 00:07

administrator   ~0003765

@carl or get DoM to do the disk IO, passing data to libdcp for digesting.

carl

2020-04-17 00:16

administrator   ~0003766

@carl another thing is this wouldn't check the sound MXF, but I suppose we can assume that if the picture MXF is OK we're probably on a working system.

Bug History

Date Modified Username Field Change
2018-10-16 10:55 carl New Bug
2018-10-16 11:07 overlookmotel Note Added: 0002716
2018-10-17 10:32 carl Relationship added related to 0001381
2018-10-17 10:32 carl Priority normal => high
2018-10-17 10:32 carl Status new => acknowledged
2018-10-17 10:33 carl Note Added: 0002719
2019-01-08 01:11 carl Target Version 2.14.0 => 2.16.0
2019-05-01 10:55 carl Estimated work required Unknown => Medium
2019-11-27 23:34 carl Tag Attached: correctness
2020-04-17 00:00 carl Note Added: 0003764
2020-04-17 00:07 carl Note Added: 0003765
2020-04-17 00:16 carl Note Added: 0003766
2021-05-23 00:02 carl Target Version 2.16.0 =>