[Feature Request] Decrypt and keep structure intact

[IoannisSyrogiannis]

I am looking for my credentials for mantis. Until then, I thought of using the forum for the request.

In the past, I came to need to decrypt DCPs, in order to archive them. In most cases, the structure was the simplest, like making a one reel DCP out of DCP-o-matic. On some others, there might have been multi-reel CPLs with VFs or packaged multi-CPL DCPs.
On the latter, I was sometimes using the Dolby Cineasset. A program to which I have no access any more. The reason was that I wanted to keep the structure intact, with all the reels and headers and feet, and that was the only way.

What I would like, would be to be able to do exactly that. Essentially, to be able to get (mostly make, since those are DCPs made and versioned for the organization I work for) a DKDM, and keep the CPL/PKL structure the same, with the important difference, of having the essences (video, audio, subtitles, in case of SMPTE) non-encrypted.
Decrypting is a one-way street, if someone wants to keep an archive of a DCP that will outlast the theatrical releases and keeping the structure intact goes a long way on archiving, as it's not only what is shown on screen that has value.

I am sure that people who have experience on the matter will want to explain to me that:
1) Apart from encryption, long term storage will eventually affect signatures.
2) DCP-o-matic is not built on a way that favors non-destructive editing or headers, feet, china girls etc.

I am aware. What I am asking essentially, is for DCP-o-matic giving me the ability to (a) decrypt the full essences, without discarding non-shown parts of the files, and then, (b) retaining the characteristics of the CPLs.
With the first part (a) to be the most important, as, for the second, I could maybe edit the original .xmls manually(?).
But, then again, if I could automate the second part as well, would be a huge time-saver.
I believe that both parts may be achieved without huge changes on how DCP-o-matic works, and I am certain that some of us will be gratefully benefited by such a feature.
@Carl, give it a thought

[carl]

Hi Ioannis,

As you say the main DCP-o-matic is sadly not well-arranged for operations like this, but I think it could make sense in one of the other tools (perhaps the editor or map). Would a CLI or GUI tool be best for you?

[IoannisSyrogiannis]

Thank you Carl, for the consideration.
Either CLI or GUI, as long as CLI is well documented, with example lines etc.
I expect to do a number of DCPs, but I don't expect to create scripts, to automate.

[Carsten]

Wouldn't this be possible with some ASDCPLIB+scripting?

[carl]

Perhaps. I think extracting the key from the KDM would be the fiddly part.

[IoannisSyrogiannis]

These last two days, I experimented on "manually" doing the job.
I use quote-marks because I didn't use a pen and paper, and I used quite a few software applications.
For anyone interested, here is what I did. Please, feel free to suggest better steps:

The encrypted DCP was a nine-reel 4K feature, pretty large in size/high in bitrate for my taste. What you're gonna do?
I created a KDM and I exported the CEK/CipherValue with a command like:

Code: Select all

xmlstarlet sel -t -v "//enc:CipherValue" -n KDM.xml > KDMCEK.txt

I had, then to remove line breaks and I made text files out of each.

I then decoded base64

Code: Select all

base64 -d ev1.txt > ev1.bin

to get a 256 bytes long bin file, and consequently, I decrypted the bin file.

Code: Select all

openssl rsautl -decrypt -oaep -inkey PK.pem -in ev1.bin -out dv1.bin

Now, the decrypted key is 138 bytes long.

Yet, not knowing which part is the decryption key, I had to -more or less- map the outcome. I miss the SMPTE document, so I figured that much:
1-16 Common on different KDMs
17-36 Common on different KeyIds of the same KDM
37-52 CPL UUID
53-56 Key Type (MDIK, MDAK, MDSK)
57-72 KeyId
73-122 Validity time
123-138 Decryption key

I gave it two steps to separate the necessary and make a file where I would include all keys:

Code: Select all

tail -c 16 dv1.bin > kv1.bin
xxd -ps kv1.bin >> keys.txt

I now used those to unwrap the essences. I would love an asdcp command that would save me unwrapping and wrapping again, but I didn't find any:

Code: Select all

asdcp-unwrap -k <decrypted_cek_hex> encrypted_video.mxf

On audio, I would get:
Will write out a regular wave file.
On SMPTE subtitles, I would get an error and the subtitles:
Error opening file /<true-type-font-uuid>: Permission denied
Program stopped on error.
File open failure.
There was a post in the past here, about parsing SMPTE subtitle .mxf files for the fonts. Nothing there either. But I didn't bother with the subtitles in the final DCP. A VF can be made with the files.
Note: both the audio and text files were with no extention. (.wav/.xml or similar.)

Then, with the j2c files and the audio, I came upon a dilemma. I can wrap them back with asdcp-wrap, and I could use the original CPL to make another. But what about signing?
I decided to go with DCP-o-matic (of course).
I made a project and added the folders and files. The most time demanding thing was aligning the essences. I checked and double checked to get it right, and the headers and tails were helpful on that. Of course, I didn't cut anything. Chose "split-by-content" reels and corrected the annotation text.

After wrapping the 4K DCP in the unseen speed of 100fps (that could be a nice joke to pull on someone, with a screenshot not showing the original content), I backed up the CPL, PKL and asset map files and started with CPL:
Introduced again the entry points and durations on each reel and saved.
I then checked size and hash.

Code: Select all

openssl sha1 -binary CPL.xml | openssl base64

I corrected the values on PKL and -then- the size of CPL on the asset map file. (PKL was the same size.)

Ran the verifier, and the edited DCP checked O.K.

I played some of it, to check for any errors on durations and entry points on "splices".
(That reminds me: maybe a mode on player for 1/8th of the size, for the sake of 4Ks?)

The whole endeavour took the best part of Sunday (almost all was figuring out what and how to do) and a fair part of Monday (that is still today, on this part of the world). It was enlightening.

[Carsten]

Wondering what happens when you just decrypt essence files and keep the original XMLs. Would that create a DCP that is formally broken?

[IoannisSyrogiannis]

You mean, decrypt, without unwrapping?
My guess (and waiting for others to add theirs), is that the UUIDs and hash checksums, if not file names would be different. So, they wouldn't play well with the CPL and PKL.
UUIDs might have been kept, but I don't think that would be good, if they met with their mysterious brothers in a cinema server.
I wonder, though, if decrypting without unwrapping, if that is a possibility, would save the wrapped (and then lost if unwrapped) fonts.

[Carsten]

Admittedly, these would not be able to be played on a plain vanilla server. But that's given - a DCP contains file/data integrity mechanisms that would mark even the smallest manipulation. So, if you have a DKDM, which is a necessary prerequisite for your task, why not archive the encrypted DCP together with the KDM or decryption keys. Of course, at some time, all certs involved would expire, but as long as you use software on these DCPs, that will never be an issue. You could even archive the code to decrypt the MXFs with the DCP.

When you wrap a new DCP, then of course, new UUIDs have to be created.

[IoannisSyrogiannis]

Carsten, for the sake of exchanging opinions, we agree upon the fact that when one decrypts essence files and keeps the original XMLs, the result would not constitute a DCP. As you write, that is a given.
I don't understand what you have in mind when you write "plain vanilla server". As opposed to what kind of server? A hypothetical one that would screen the essence files of a DCP by using a CPL from another? I see no chance that such a thing is going to exist, ever. And rightfully so, if you think its main caveat: Its mode of operation is exactly the problem that its creator/user would have to troubleshoot. I can't imagine the complexity of the tasks such a server would have to perform, in order to do that.

Now, besides that, on your "why not" proposition, the answer lies greatly on the purposes of archiving.
If, for instance, the person that archives does that for a definite time, and considers themselves to be the recipient/proprietor/manager of the archive, then the encrypted content would do!
One could argue, what about the extra human hours necessary in an event similar to the one that dear Jim Cassedy called "the Y2K24 bug"? Yet, looking reasonably "short-term" and in an individual or small business scope, that is a "legitimate" programming and risk taken.
If, on the other hand, the person that archives targets probable use for a longer time period. When the person that archives (or, its role in the institution or organization) will be outlived by the archiving material, archiving the material inside a puzzle box (to use a metaphor), with a sticker on it, that describes how one would be able to open it and reach the material with means available at the time of archiving is counter-productive. Especially if we are considering a number of DCPs that are submitted encrypted and may show demand for use from time to time.

Starting from the fact that DCPs are not the go-to format for archiving, encrypted DCPs are more precarious and much further down on that chart. I imagine that you do have quite a lot experience with old DCPs and that you came upon situations where a (D)KDM was unobtainable. When we talk about productions where the best or only available copy is a DCP, the chances are rising.

To conclude, I don't know if anyone would find the idea of archiving encrypted with the (D)KDM on the side, or/and the code, software and specifications of the system necessary to decrypt appealing, but -to me- that is not recommended practice. So, one way or another, I try to find the way and devote the time to decrypt and keep ready-to-use DCPs.

I would have seen under another prism the question:
Why do you feel the need to re-wrap the essences, instead of keeping them like some sort of distribution master?
But that question-never-put would have to be weighted against complexity introduced.