Possibly fixed by acfbd1c01020ea95d9e7eb3d63ddb14b9407732a

Wow! Really?

Apparently so, according to their tech support. A user is making black DCPs and they're crashing a DSS200. Just trying to get a test version out to the user, but the Apple notorization stuff is playing up, as per usual.

I assume they also use a very low bitrate in addition? Don't know wether that makes a difference, though.

Weird...unfortunately, currently I can't test on a DSS200.

I use long 'real' blacks very often in combination with timed text. Maybe, instead of adding noise, one should recommend to them to use their own 'black noise' source?

How is 'black' actually created without a source content, e.g. when I place audio in an otherwise empty project? Do you actually create a J2K from a 0data-image of container dimension? Would be interesting to find if there is a difference between that method an having a 'real' 0-padded source file. Maybe something happens there.

I assume they also use a very low bitrate in addition? Don't know wether that makes a difference, though.

Not sure, though pure black frames come out very small, no matter what your bitrate setting is.

Maybe, instead of adding noise, one should recommend to them to use their own 'black noise' source?

We could do, though I think lots of people would miss/ignore that advice and then potentially have crashes... The patch only adds noise when the size of an encoded frame is below some threshold, so it shouldn't affect things 99% of the time. And it's really a very small amount of noise.

How is 'black' actually created without a source content, e.g. when I place audio in an otherwise empty project?

A black image is created and encoded into JPEG2000. Then on the next frame it spots that it's the same as the last and re-uses the same JPEG2000 data. Apparently the manufacturer in question told them that low bitrates are a problem for the DSS-200.

In general, I wouldn't care to add something to assure compatibility. I feel a little bit concerned being urged to 'add something that isn't there' to every DCP. It would be nice if we could do more testing around this. e.g. would black be the same as 'near' black, does it have to be noise, etc. You remember that J2k which was caused by essentially two more or less irrelevant formal bytes. Wondering wether something like that could fix this issue as well. Do we know wether it could already be one single frame that causes issues, or does it have to be a sequence of specific length?

I think we did talk about keeping j2k streams within our bandwidth constraints (when we talked about e.g. 4k or High frame rate). If the noise is really only added when the resulting j2k is too small that would probably be acceptable. However, couldn't that lead to increased encoding time, repeatedly checking and re-encoding? I do know that compressing perfect black is very fast.

Funfact: The old Dolby DCP encoding station (SCC2000), now legacy equipment, was known for it's code-stream effectiveness, it was able to create very low bandwidth J2K while maintaining high visual quality. Wondering wether it's encoder added noise to black...

Are we sure that these low bandwidth J2K are actually compliant? Do they pass dcp_inspect, clairmeta? Maybe there is a bug in OpenJPEG? Should we contact them about it?

In general, I wouldn't care to add something to assure compatibility

I'm a little surprised by that! If somebody makes a DCP with a little black in it, plays it back on a DSS-200 and it crashes, DCP-o-matic will be blamed 100% of the time; and that's where all the FUD starts. It's unfortunate, but true, and it feels to me like we should do things to stop these crashes if they don't negatively affect anything else.

I feel a little bit concerned being urgend to 'add something that isn't there' to every DCP

Sure, but this is not every DCP - only ones with very low-bitrate frames.

It would be nice if we could do more testing around this

Indeed, I have asked the reporter to check this build, which will be a useful first data point.

would black be the same as 'near' black

If Dolby's explanation for the crash is correct, then yes.

It's true that their suggestion was to use an encoder where you could specify a minimum bitrate, but at first glance this isn't trivial with libopenjpeg so adding noise could perhaps be the next best way to test things. I'll ask the openjpeg folks if there's an easy way to ensure a minimum bitrate.

If the noise is really only added when the resulting j2k is too small that would probably be acceptable

That's exactly what is happening, yes.

However, couldn't that lead to increased encoding time, repeatedly checking and re-encoding?

Yes, slightly; but if we have 100 frames of the same black we'll only need to do this for the first frame.

Wondering wether it's encoder added noise to black...

Difficult to tell unless we have a black DCP encoded with it. Even then it might not be obvious.

Are we sure that these low bandwidth J2K are actually compliant?

No, good idea, we could check. Though I don't think dcp_inspect/clairmeta do much codestream-level checking of JPEG2000 data.

In general, I wouldn't care to add something to assure compatibility

I'm a little surprised by that!

I guess I put this the exact opposite way I meant it. Of course we need to do everything possible to assure DCP-o-matic DCPs do not crash any servers. Even if it is clearly the manufacturers fault. or, at least in this case where it is highly unlikely that the manufacturer will ever fix it.

It's a bit strange that this Dolby weakness comes up so late in time. Shouldn't we have seen such issues earlier? Can you supply some of the information you mentioned to me by mail?

I guess I put this the exact opposite way I meant it.

Ah, I see now :)

It's a bit strange that this Dolby weakness comes up so late in time. Shouldn't we have seen such issues earlier? Can you supply some of the information you mentioned to me by mail?

It is indeed very strange... I'll follow up by mail.

Weird. I remember there is also that Doremi weakness that crashes them well below the 250MBit spec limit. We should probably implement some general bitrate watchguard. But how do we deal with this in the GUI? Keeping bitrates high enough for Dolbys is comparably easy, check for it, and apply fix. Wondering wether we should issue a warning about data rates of more than 220MBit/s or so as well?

There's a hint about 245MBit/s, and the verifier warns about individual frames above some limit as well. Maybe the 245MBit/s warning should go down, or maybe just limit the GUI to 220 unless the "let me use whatever bitrate I like" checkbox is ticked.

Yes. I think I suggested that 245MBit/s myself a long time ago once I heard about that Doremi weakness first time. But it was just a rough guess to keep the data rate from peaking over 250MBit/s. You may remember that I created an actual documentary DCP in late 2019 that crashed already at 230MBit/s on Doremis. I needed to solve it quickly and simply recreated it at 200MBit/s.
I also checked the actual data rate and they were precisely within specs. I guess we need a warning somewhere around 210-220MBit/s. I may do some testing around that, but it is probably wise to lower that warning threshold already now. 220 is a nice enough number... Limiting in the GUI is a good idea, but still leave that warning in, and set it to 220, I'd say. Some people are unnecessarily concerned about data rates always being to low for their precious content.

Speaking of blacks, blacks, dark blacks and real blacks, this is one of my favourites...

https://www.youtube.com/watch?v=wx8-mysJG2s

Maybe it's enough to add some blue for Dolbys...

You let Dougal make a DCP?!
https://www.youtube.com/watch?v=xJhbP4q32vs

Of course!

NOOOOOOOOOOOOOOT11!!!

User reports that the noise hack with 65536 bytes minimum frame size fixes the DSS 200 crashes.

Now, what a pretty number...

Reporter also says 16KB setting in DoM works; I'll probably hard-code this for 2.16.0 and perhaps we can investigate more later.

a2dbe09d787ad10881ac52761772d6d50d2e29e9