URGENT: Hash mismatch

Anything and everything to do with DCP-o-matic.
Bonusfeatures75
Posts: 5
Joined: Thu Oct 31, 2019 9:31 pm

URGENT: Hash mismatch

Post by Bonusfeatures75 » Thu Oct 31, 2019 9:36 pm

Just delivered a DCP to a festival and I got back an error log from them. I have no idea how this would have failed or what I did wrong.

The dcp was created in dcp o matic from a 4k scope 5.1 file. After creation I used DCP transfer to validate the dcp and move it to a linux drive created by the software. Everything checked out.

I have attached the error log they sent, can someone please tell me what went wrong? Need to re deliver ASAP. Thanks!
Attachments
JDDCPvalidation.txt
(5.98 KiB) Downloaded 19 times

carl
Site Admin
Posts: 1082
Joined: Thu Nov 14, 2013 2:53 pm

Re: URGENT: Hash mismatch

Post by carl » Thu Oct 31, 2019 9:49 pm

Hi there,

The video file in your DCP has been corrupted at some point. The two most likely reasons that come to mind are:
  • Some problem with the drive you used. Does DCP Transfer validate the DCP before or after the copy, or both?
  • It looks like maybe you copied the whole DCP-o-matic project onto the drive, rather than just the DCP. Is that right? If so, that may have caused problems (I'm guessing here...). Perhaps you could try again with just the DCP itself (see here.

Bonusfeatures75
Posts: 5
Joined: Thu Oct 31, 2019 9:31 pm

Re: URGENT: Hash mismatch

Post by Bonusfeatures75 » Thu Oct 31, 2019 10:26 pm

I created the DCP via dcp o matic right onto an external drive. Then via that external drive I validated and copied the dcp folder with dcp transfer (not the dcp o matic folders) onto the drive.

Is there a way to make dcp o matic validate the dcp after its been created before I do any transfers? I'm at a loss here.

carl
Site Admin
Posts: 1082
Joined: Thu Nov 14, 2013 2:53 pm

Re: URGENT: Hash mismatch

Post by carl » Thu Oct 31, 2019 11:08 pm

If you validated the DCP with DCP Transfer I would hope it would spot any problems at that stage.

You can load the DCP you made into the DCP-o-matic player, then select "Verify", but I would expect DCP Transfer to have done the same checks. If they passed it would suggest that the DCP was OK when it left you. It's quite hard to be sure, though.

Where are you located, roughly?

Bonusfeatures75
Posts: 5
Joined: Thu Oct 31, 2019 9:31 pm

Re: URGENT: Hash mismatch

Post by Bonusfeatures75 » Fri Nov 01, 2019 12:59 am

I'm located in NY.

I ran another validation on the same file via DCP player and it validated properly.

I just got off the phone with DCP transfer and they told me their latest version has a few issues with the version of paragon included. They advised me to roll back to a previous version, which I did.

I am now going to go through this process again... we will see.

Carsten
Posts: 1462
Joined: Tue Apr 15, 2014 9:11 pm
Location: Germany

Re: URGENT: Hash mismatch

Post by Carsten » Fri Nov 01, 2019 1:05 am

As Carl pointed out - it is usually best to verify the DCP during/after it has been copied to an external/transport drive. Typical errors that a verification looks for are data corruption/copy errors (which can have various technical reasons), and that is what the hash check is able to detect.

Unfortunately, there is a systematic flaw in the way DCP-o-matic creates DCP files and hash values - DCP-o-matic writes the files first, then, finally after creation performs the hash computation. If for some reason, the writing of the files creates corrupt files, the hash computation afterwards will still create a valid hash for an invalid file. As a hash is targeted at file integrity, but not structural integrity. An invalid/broken MXF file can still have a valid hash, if the hash has been computed on the broken file.

That said - in the log you supplied, it is evident that the hash doesn't match the file - so, as Carl wrote, something must have happened to the video file/MXF between YOUR and THEIR verification.

- Carsten

Bonusfeatures75
Posts: 5
Joined: Thu Oct 31, 2019 9:31 pm

Re: URGENT: Hash mismatch

Post by Bonusfeatures75 » Fri Nov 01, 2019 1:17 am

I have done this also. DCP transfer validates both before and after the transfer and it checked out.

The people at the festival are saying something about how according to the validation report, at least the audio file doesnt match, and they are saying it could be something with dcp o matic?

Is there anything in that validation report pointing to an audio file issue?

Carsten
Posts: 1462
Joined: Tue Apr 15, 2014 9:11 pm
Location: Germany

Re: URGENT: Hash mismatch

Post by Carsten » Fri Nov 01, 2019 1:03 pm

No, the log is pointing at a hash mismatch with the video file.

CPL:
960e2119-ae73-4f63-beff-0704be8c42de: text/xml;asdcpKind=CPL: cpl_960e2119-ae73-4f63-beff-0704be8c42de.xml
960e2119-ae73-4f63-beff-0704be8c42de: Checking hash value: OK (00:00:00)

Video:
7c5292dc-0ca6-4271-b991-9c4f2218760b: application/x-smpte-mxf;asdcpKind=Picture: j2c_7c5292dc-0ca6-4271-b991-9c4f2218760b.mxf
7c5292dc-0ca6-4271-b991-9c4f2218760b: Checking hash value: Mismatch (00:10:40)

Audio:
0ddb9a1a-3f90-4f14-ab7c-29d3424ecd82: application/x-smpte-mxf;asdcpKind=Sound: pcm_0ddb9a1a-3f90-4f14-ab7c-29d3424ecd82.mxf
0ddb9a1a-3f90-4f14-ab7c-29d3424ecd82: Checking hash value: OK (00:00:22)


So, since they are pointing at an audio file, it seems they are using a verification software, but can't judge the results.

What if you make them try to ingest that DCP onto one of their servers? Servers perform checks themselves.

Maybe their verification process has issues. Maybe their ext2/3 reader software has issues (if the verification software runs in windows or OS X). Can you ask them for the name of their verification software? The log file doesn't give me a clue.

Hash computing and checking is not specifically difficult or variable in the DCI world - if you verified successfully with DCP Transfer, there are only some very few issues that could cause this. One possible reason could be an issue with the storage device/disc, or the interface/reading of the files during checking.

Where are you located, and where is that festival? As you mention DCP Transfer, you are obviously on a Mac. Which OS X version do you use? Did you contact the DCP Transfer people already?

Don't know what these are:
---
Error: CPL 960e2119-ae73-4f63-beff-0704be8c42de: Composition type: Undetermined
Error: CPL 960e2119-ae73-4f63-beff-0704be8c42de: Expected to scrounge 1 aspect ratio (1 reel) but got 0
Error: CPL 960e2119-ae73-4f63-beff-0704be8c42de: Expected to scrounge 1 picture resolution (1 reel) but got 0
Error: CPL 960e2119-ae73-4f63-beff-0704be8c42de: Composition incomplete
---

But as they are picture related, it may be that they follow the hash failure.



- Carsten

carl
Site Admin
Posts: 1082
Joined: Thu Nov 14, 2013 2:53 pm

Re: URGENT: Hash mismatch

Post by carl » Fri Nov 01, 2019 1:28 pm

It's from dcp_inspect, so I'd be confident that it's right.
Error: CPL 960e2119-ae73-4f63-beff-0704be8c42de: Composition type: Undetermined
Error: CPL 960e2119-ae73-4f63-beff-0704be8c42de: Expected to scrounge 1 aspect ratio (1 reel) but got 0
Error: CPL 960e2119-ae73-4f63-beff-0704be8c42de: Expected to scrounge 1 picture resolution (1 reel) but got 0
Error: CPL 960e2119-ae73-4f63-beff-0704be8c42de: Composition incomplete
I think this is just some stuff it wanted to get from the picture MXF but it failed. It sounds like the corruption was fairly serious.

Carsten
Posts: 1462
Joined: Tue Apr 15, 2014 9:11 pm
Location: Germany

Re: URGENT: Hash mismatch

Post by Carsten » Fri Nov 01, 2019 1:48 pm

That probably means the verification and ext2/3 reading is native Linux.

Hmm, weird. I could only think of a disc related issue, maybe some issue with a USB/SATA bridge as well. We can not even be sure wether the DCP there is in fact corrupt at all - maybe it is 'just' a read error during the verification. That's why I suggest trying an ingest on other hardware (DCI server).

Do we know wether they verified the transport disc, or did they copy first to their storage and then verified their copy? The path given in the log doesn't give me a clue.

- Carsten

Post Reply