Batching encode server requests

[rtX]

If it was technically possible, I think that batching frames for encoding servers to process might be more efficient than what I observe as being one frame at a time being passed, processed and returned to and from each server. It looks like the servers are not working to anything like their full processing capability, as there are often significant periods of inactivity between frames and the encoding rate of the main dcpomatic program does not increase by anything like the potential it might when adding a number of servers. As an example, my Linux machine with 4 cores processes at about 0.4 frames per second. If I add two four-core windows machines and a 2 core machine to the network and run the encode server on each of them, the frame rate only increases to about 0.6 frames per second even if the servers are doing little/nothing else other than process frames for the main program. This despite 'theoretically' increasing the computing power about 3.5 times. I know that the processors run at different specifications etc. and that network transmission may slow rates marginally, but this still is not what one might expect.

Is it possible for the main program to provide the server/s with say, between 20 and 100 frames to process at a time so that the servers can return them batched or individually? If individual returns were implemented, it might enable the main program to predict when the server will need more work and send it ahead of receiving back the last frame from its previous batch. If returned in batches, the server could indicate its anticipated readiness for more work when it had finished processing say 90% of the previous batch.

I'm thinking of installing a dual boot Linux system on one of the four core Windows 7 machines to see if running the server under Linux increases productivity. I'll report back as and when I do that.

Thanks again for a great program.

[carl]

The client/server code is supposed to keep the servers busy when it can. Batching things might speed things up, but matters are slightly complicated by the fact that the master machine has to write its frames to the output DCP in order; if 20 frames were going to come back from one server but then they had to wait for 20 from another server things might get tricky.

The first thing to check would be: are you using (at least) gigabit ethernet? Secondly, it's possible that DCP-o-matic has obtained the wrong values for the core counts on your master and/or servers (so it's not giving them enough work to do). The first few hundred lines of the "log" file that the master writes might be interesting.

[rtX]

I'll look at the log file. I added the encode servers about 12 hours after starting the encode. I checked on the master machine (there's a menu option to check encode servers). It 'seemed' to have the correct number of threads... Except one of the machines was a core i3 - it showed that as a 4 thread - not sure if that is correct...

[carl]

Can you send the log file to me? (carl@dcpomatic.com)

[rtX]

Will do. It'll have to be a link to download it as I recall it being large.