Black screen on Doremi servers

[Dark]

Hi,

I'am using DOM to export DCP for movie distribution. Already made more then dozens of DCPs. But with last one we found an issue...

It was 96mins movie in local "Art" distribution to around 100 cinemas. DCP play good in our postproduction cinema projector (Barco sp4k-12c with Barco Alchemi) and no problem in cimeas with different servers. But in cinemas with Doremi server there was a problem. At the same time (about 41 minutes) the image was lost, and artifacts appeared and the sound cracked. It took about 10 seconds. Then everything returns to normal and the film ends without mistakes. A lot of people even thought it belonged to that passage, so we had problems with the discovery, and we practically only found out during the screenings, where there was also a director and producer who knew the film.

The distribution was online and several cinemas also received it on disc. In the cinema that DCP had from the disk, the error did not appear in another cinema with the same disk, and after searching for the error cinema, they had a common Doremi DCP2000 server. Which is very common in our country (about 70%). There was no error when injesting DCP, the server loaded everything correctly.

I produce DCP using the version: 2.14.48
I use DPX 12b input from davinci resolve.
Audio as separate stems.
DCP bitrate 200MB/s
24FPS
Interop
Input gamma 2.4
I was using workstation (Threadripper 3975) with ervers 2x(2x intel xeon 16core)

I would like to ask, if this error is common?
What could have caused it?
Is there any way to found this kind of error without trying every server in cinemas? DCP play ok in davinci resolve, easydcp player, DOM player....



Few days back, i created new DCP with 240MB/s, SMTPE, only on workstation (servers disabled) and from first reports, its ok and no problems even on Doremi servers.

Any ideas?
Thank you

[carl]

Hi there,

As far as I know the Doremi servers do check hashes during ingest, so I think it's unlikely to be due to file corruption.

It might be interesting to run the DCP through the verifier that is included in the test version of DCP-o-matic, as that particular verifier does a reasonably thorough check of the J2K bitstream.

Is it possible for me to get a copy of the DCP? Perhaps I can look and see if there's something odd about the image file around 41 minutes.

Others with more current projection experience than me may be along with some ideas too, I hope!

Best,
Carl

[carl]

One other thing that I just thought of is whether you can talk to a technician responsible for one of the sites where the error occurred, to see if it's possible to extract logs from the Doremi.

[Carsten]

Although you mentioned 200MBit/s, it could be a data rate problem. Doremis tend to show such issues. Usually they are repeatable. I have experienced something similar, but, admittedly, with a data rate of 230MBit/s. I lowered it to 200MBit/s, and the problem went away.

Did I understand correctly, that this issue occurred on multiple servers?

I would assume either a data rate issue, or a broken J2K stream. Unfortunately, data corruption during encoding in certain parts of the J2K stream can not be detected easily. Hashing will only take place on the final encoded stream, and if there are logical issues within the stream, the hash may still be correct.

The only way to check this is to create the DCP from the same metadata project data again on a different system (e.g. local encoding only), then compare the first and second J2K files (in that case, hash comparison should suffice). Of course, all settings need to be exactly the same between both encodings. Are you sure your Master and remote servers run the same software version?


I have one professionally encoded unencrypted DCP stored, which shows a green flash frame half way into the feature. Hashes are okay, it verifies okay. Data corruption during encoding or pre-hashing may still make it into a formally correct DCP. Well it is not 100% formally correct, because there could be logical errors in the J2K data structure. But even the J2K data structure could be correct if the data corruption occured during compression e.g. if the source image was corrupt or became corrupted during encoding. However, in that specific case, the error would show on every server. J2K data stream corruptions, though, may show on some server types, but not on others.

- Carsten

[Dark]

To Carl:
- test is running now, i started it from cmd. I will paste results when finished EDIT: report at the end
- sorry, i cant send it, its in distribution it can't get out
- i can try to ask for logs, but it was not our "friendly" cinema but i will try to ask


To Carsten:
- yea at first I also thought it would be a high bitrate, but when the producer told me today that he was given info about the new DCP, that it was fine and it was 240MB / s so I was unsure. A friendly technician from AV MEDIA, a.s. from which we have a projector (and they will come to us for calibration in 2 days, so I will ask more of them) says that Doremi should handle 250MB / s without any problems and that it was probably a conversion error .. but the strange thing is that only on doremi and all other systems did not have the slightest problem with that
- yes it was about 8 cinemas (which we know) and all of them have Doremi server. Wierd thing is, that when it first happend, first movie after our had same error... so we and the distributor thought that there was a server fault only in that movie theater
- broken J2K stream.. wouldn't it appear on other servers?
- I still have the original data and the project, so I will try it to render it only on workstation and then try to calculate checksum of both


I think i could find log from DCP creation, but i did not found any abnormal in it.



Verification report:

Code: Select all

C:\Program Files\DCP-o-matic 2 15_178\bin>dcpomatic2_verify.exe \\10.24.1.1\Kino_DCP\SOTK\SnyOToulavychK_FTR-5_F_CZ-CZ_51_2K_20211013_IOP_OV\
Checking DCP: \\10.24.1.1/Kino_DCP\SOTK\SnyOToulavychK_FTR-5_F_CZ-CZ_51_2K_20211013_IOP_OV
Checking CPL: \\10.24.1.1/Kino_DCP\SOTK\SnyOToulavychK_FTR-5_F_CZ-CZ_51_2K_20211013_IOP_OV\cpl_bbf14728-5f2f-427c-8be2-1e784900f57f.xml
Checking reel
Checking picture asset hash: \\10.24.1.1/Kino_DCP\SOTK\SnyOToulavychK_FTR-5_F_CZ-CZ_51_2K_20211013_IOP_OV\j2c_bd18478f-8146-4282-91ad-334c0aca151b.mxf
Checking picture frame sizes: \\10.24.1.1/Kino_DCP\SOTK\SnyOToulavychK_FTR-5_F_CZ-CZ_51_2K_20211013_IOP_OV\j2c_bd18478f-8146-4282-91ad-334c0aca151b.mxf
Checking sound asset hash: \\10.24.1.1/Kino_DCP\SOTK\SnyOToulavychK_FTR-5_F_CZ-CZ_51_2K_20211013_IOP_OV\pcm_97f5f357-22e7-4d6d-83d2-9aff8b003dae.mxf
Checking sound asset metadata: \\10.24.1.1/Kino_DCP\SOTK\SnyOToulavychK_FTR-5_F_CZ-CZ_51_2K_20211013_IOP_OV\pcm_97f5f357-22e7-4d6d-83d2-9aff8b003dae.mxf
Checking PKL: \\10.24.1.1/Kino_DCP\SOTK\SnyOToulavychK_FTR-5_F_CZ-CZ_51_2K_20211013_IOP_OV\pkl_1643e654-1050-4e74-a16f-f69827a0e48a.xml
Checking ASSETMAP: \\10.24.1.1/Kino_DCP\SOTK\SnyOToulavychK_FTR-5_F_CZ-CZ_51_2K_20211013_IOP_OV\ASSETMAP
Bv2.1 error: This DCP does not use the SMPTE standard.
Bv2.1 error: The JPEG2000 codestream uses 2 guard bits in a 2K image instead of 1.
D

[carl]

To Carl:
- test is running now, i started it from cmd. I will paste results when finished EDIT: report at the end
- sorry, i cant send it, its in distribution it can't get out
- i can try to ask for logs, but it was not our "friendly" cinema but i will try to ask

Thank you - there seems nothing bad in that verification. The guard bits thing is mentioned in SMPTE Bv2.1 but DCP-o-matic was only recently modified to use the recommended 2 guard bits. It seems unlikely to be the problem.

Remaking and comparing video MXFs is a good idea. I don't think the hashes will ever match because of datestamps and so on I can send you a tool to compare the new DCP with the old one (which will compare the bitstreams of each individual frame, which I think should be the same).

[Carsten]

Yup, comparing the MXFs directly would probably not give useful results. I remember when I found that J2K optimisation bug a few years ago, I unpacked the J2K MXF into individual J2K images and compared those. They were identical between two runs with the same compression rate (and build).

As this turned out to be a show stopper issue, it would probably be worth investigating into it.

@Dark: It is not unlikely that specific J2K data is able to break certain decoders, but not others. I remember we had issues with J2K generated by Davinci Resolve that showed on some servers, but not others. The major servers all use different decoding hard- and software, and if there is corrupt data, I would expect differences between them. The common Doremi Dolphin boards are known to have issues when data rate peaks, so I would never create standard (24fps) DCPs with more than 220-230Mbit/s. HFR capable systems (usually IMBs/IMS) can be expected to decode 400-500MBit/s. The actual visual quality benefit between 200 and 250 MBit/s from my experience is zero.
I have also seen Doremi IMBs show artefacts with (synthetic) high detail content in 4k well below their spec'd data rate limit.

[Dark]

To Carl:
- test is running now, i started it from cmd. I will paste results when finished EDIT: report at the end
- sorry, i cant send it, its in distribution it can't get out
- i can try to ask for logs, but it was not our "friendly" cinema but i will try to ask

Thank you - there seems nothing bad in that verification. The guard bits thing is mentioned in SMPTE Bv2.1 but DCP-o-matic was only recently modified to use the recommended 2 guard bits. It seems unlikely to be the problem.

Remaking and comparing video MXFs is a good idea. I don't think the hashes will ever match because of datestamps and so on I can send you a tool to compare the new DCP with the old one (which will compare the bitstreams of each individual frame, which I think should be the same).

No Thank you for helping me.

Sure if you have some tips, or SW you can send... I am all in. Also I have big technical background, IT university, programing and so on.. so i will try everything you can send me.

I just had a chat with producer and if it is possible to cut out "problem section" out without reencoding, then i could be able to send it to you. Maybe load it to DOM, trim from start and end, and export? Or try ffmpeg?

I just learned that the new version of DCP (240MB / s) is playing properly in one cinema (the old version had a problem) but the other cinema has the same problem as the old version.

Movie is full of complicated graphic images and animations and on lower bitrate (200MB/s) we had some visual compresion artefats in images (not in bad section, which is made from camera files, no images)

Can I some how unpack j2c from mxf? And check the questionable frames?


Thank you

[carl]

I just had a chat with producer and if it is possible to cut out "problem section" out without reencoding, then i could be able to send it to you. Maybe load it to DOM, trim from start and end, and export? Or try ffmpeg?

That's kind, thanks - it should be possible to do. Let's try unpacking the MXF into J2K your side first though...

Can I some how unpack j2c from mxf? And check the questionable frames?

There's a tool called asdcp-unwrap which can do that. There's a windows build here that perhaps you could try. If you run asdcp-unwrap on the video MXF you should get a .j2c file per frame. It would be interesting to look at the frames around the fault. Initially maybe just see if there are any crazily small or big frames (in terms of bytes).

[Dark]

Ok, i will try it out. thx

Meantime I tried to get the average bitrate calculated, so I'm attaching a screan, as this SW can't be exported.