Not sure if this is standard or a bug. I have created a DCP for this same show multiple times, although DoM 2.10.5 (Win10) is doing a very slow "computing digests" (approx. 5 hours) after transcoding. I recall the entire process taking about 3-4 hours, including transcoding on a single 12-core Xeon machine. However, my previously sent encrypted DCP somehow did not ingest/validate well at the distributor, so I am making another from scratch.
DCP info:
-Feature, in reels
-5.1 audio (separate tracks)
-TIFF seq (in XYZ)
Transcoded using two servers, server 1=single socket 14-core Xeon, server 2=dual socket 14-core Xeon. Server 2 was running from the CLI, Server 1 from the GUI. Server 2 only used half of the available threads even though it was running with the "-t 56" command.
I'll send the metadata logs, once the process is finished but was wondering if I am just using a foolish set-up.
Any help is appreciated, thanks!
Slow "Computing digest"?
-
Carsten
- Posts: 2804
- Joined: Tue Apr 15, 2014 9:11 pm
- Location: Germany
Re: Slow "Computing digest"?
Calculating a digest for a long feature can be comparably slow, even on a machine with many cores, as this is a single-threaded task. In that case, the machine(s) rush through the j2k like mad, but then the digest could take longer than the J2k transcode...
You can speed it up by splitting the DCP into reels (as you did). Then each reel can have it's own digest compute thread (since 2.10.1). It will only compute the digest on the local machine though. Yet with a 14 core xeon and a reasonable reel number, like 8, it should go faster than what you experience. Maybe there is something wrong. From what I know, the digest will only run through the created DCP, not touching any of the source footage any more. So the fact you are using a TIFF sequence shouldn't change anything. However, access time to the DCP created certainly plays a role, e.g. if the DCP is on an external USB2 disc. If the DCP disc is slow, having multiple reels/digest threads will make it even worse.
Please send the log to Carl, he may find out what is going on during digest calculation. Maybe there is a bug in the multi-reel digest calculation.
How long is your feature, what type (framerate, resolution, datarate), and how long does it take to render with these two machines?
From my experience, you should set the local machine to 28 threads in the GUI, and the remote server to 56. I don't think that the remote server can be saturated with 56threads though, a gigabit network will not be able to carry all the data. Which is what you experience.
At the same time, the local machine could be loaded with unnecessary frame preps for these remote threads. So maybe you should set the remote server to 28 threads only.
If this is 2k, you should see something like >30fps during J2k compression, I would assume.
I remember that digest calculation singlethreaded/single reel for a full length feature took around 30min or so - in that ball park. With eight reels, you should expect this to come down to something lik 5min or so...? If everything works as expected...
- Carsten
You can speed it up by splitting the DCP into reels (as you did). Then each reel can have it's own digest compute thread (since 2.10.1). It will only compute the digest on the local machine though. Yet with a 14 core xeon and a reasonable reel number, like 8, it should go faster than what you experience. Maybe there is something wrong. From what I know, the digest will only run through the created DCP, not touching any of the source footage any more. So the fact you are using a TIFF sequence shouldn't change anything. However, access time to the DCP created certainly plays a role, e.g. if the DCP is on an external USB2 disc. If the DCP disc is slow, having multiple reels/digest threads will make it even worse.
Please send the log to Carl, he may find out what is going on during digest calculation. Maybe there is a bug in the multi-reel digest calculation.
How long is your feature, what type (framerate, resolution, datarate), and how long does it take to render with these two machines?
From my experience, you should set the local machine to 28 threads in the GUI, and the remote server to 56. I don't think that the remote server can be saturated with 56threads though, a gigabit network will not be able to carry all the data. Which is what you experience.
At the same time, the local machine could be loaded with unnecessary frame preps for these remote threads. So maybe you should set the remote server to 28 threads only.
If this is 2k, you should see something like >30fps during J2k compression, I would assume.
I remember that digest calculation singlethreaded/single reel for a full length feature took around 30min or so - in that ball park. With eight reels, you should expect this to come down to something lik 5min or so...? If everything works as expected...
- Carsten
-
Carsten
- Posts: 2804
- Joined: Tue Apr 15, 2014 9:11 pm
- Location: Germany
Re: Slow "Computing digest"?
Hmm, currently I am not doing much more than some testing with DOM, and usually only in stills, so I didn't notice this as they render quickly overall. But when I look close, I notice that even short pieces, like 30s of stills, take very long computing the digest. I think there is something wrong in the test version...
- Carsten
- Carsten