Hi
I am new to this software and taking over from Lars Kelto at the Traverse City Film Festival who passed away two weeks ago.
I am tasked with setting up the server farm for the dcp rendering. I have got a couple of servers running as a test bed and all seems to work when run from the main program. when i try and run it through the batch converter it will not do any conversion. I have setup the two ips of the servers under preferences. The main program sees the encoding servers under tools/encoding servers but the batch processor does not.
Our festival is next month so getting down to the wire. Any help would be appreciated.
Thanks
Alan
DCO-O-Matic Batch processor
-
Carsten
- Posts: 2668
- Joined: Tue Apr 15, 2014 9:11 pm
- Location: Germany
Re: DCO-O-Matic Batch processor
What OS is running on these encode servers and main DOM machine, which DOM version do you use?
Maybe the Firewall is preventing communication for the batch processor.
If you feel under pressure, tell us as much as possible - server/CPU type, OS, etc., and there'll be people to help you through.
This job has been done for the festival before, but I guess the system you are talking about has been set up new?
I suggest to use DOM 2.10.5 for the job, unless you have some very special requirements.
- Carsten
Maybe the Firewall is preventing communication for the batch processor.
If you feel under pressure, tell us as much as possible - server/CPU type, OS, etc., and there'll be people to help you through.
This job has been done for the festival before, but I guess the system you are talking about has been set up new?
I suggest to use DOM 2.10.5 for the job, unless you have some very special requirements.
- Carsten
-
scozz76
- Posts: 70
- Joined: Sun May 15, 2016 10:13 am
Re: DCO-O-Matic Batch processor
I agree with Carsten, sounds like the firewall.
I found with windows 10 if you reset your network adapter (settings/network & Internet/status/network reset) you'll be asked to allow DCP-O-MATIC for access permission via the firewall when running the programs again.
I also found the latest stable release would not talk to the latest test release under win10.
I found with windows 10 if you reset your network adapter (settings/network & Internet/status/network reset) you'll be asked to allow DCP-O-MATIC for access permission via the firewall when running the programs again.
I also found the latest stable release would not talk to the latest test release under win10.
-
Tudortech
- Posts: 10
- Joined: Fri Jun 16, 2017 3:32 pm
Re: DCO-O-Matic Batch processor
Hi, thanks for the responses. I have found that if I select a file to be processed by clicking process by batch , it will take me to the batch processing program but will sit there and will not start until I close the original dcp window. Then the processing begins and I can open up the original dcp program again and the processing continues. Also when I open the batch processing program GUI and select tools and encoder servers none appear until I again close the original dcp program. Also under the batch processor I am unable to add additional servers and have it stick . Only the first one will stay in the list under preferences servers.
Currently servers are windows 2012/2008 with no firewall and workstation is windows 10. Using latest version of dom .
Currently servers are windows 2012/2008 with no firewall and workstation is windows 10. Using latest version of dom .
-
Carsten
- Posts: 2668
- Joined: Tue Apr 15, 2014 9:11 pm
- Location: Germany
-
Carsten
- Posts: 2668
- Joined: Tue Apr 15, 2014 9:11 pm
- Location: Germany
Re: DCO-O-Matic Batch processor
Hmm, yes, main GUI and batch processor are fighting for the remote encode servers. As you start DOM main GUI first, it reserves the remote encoding servers. Batch processor starts second, no encode servers left. When you exit Main GUI, remote encode servers are released, and batch processor will use them.
Is it possible you also ticked 'Only servers encode' in advanced prefs? Because, even without remote encode servers, batch processor should start working. Only if you disallow local J2C encoding, nothing will happen.
I was able to reproduce that lock-up on my personal Mac machine as well - started local main GUI, it locked the remote encoder, batch processor will not see remote encoding servers. I have 'only servers encode' ticked. Now when I send a job to batch converter, it stays at 'Encoding 0%' forever - because it is not allowed to encode locally, and also doesn't see remote encoders.
I remember Carl put some thinking into this in the latest test versions, but with your festival ahead, I would not advise you to use the recent test versions, as they have many new features and possible instabilities. Stick with 2.10.5 release version for now, and simply always start batch converter BEFORE you start DCP-o-matic main GUI, and always leave it running. That way, batch converter will always have priority to the encoding servers.
You may even put batch converter into auto start.
On my machine, I need to untick 'only servers encode' both in DOM and batch converter and quit both apps before the setting is changed (because they share the same prefs).
Tell us what types/numbers of CPUs all these machines use.
- Carsten
Is it possible you also ticked 'Only servers encode' in advanced prefs? Because, even without remote encode servers, batch processor should start working. Only if you disallow local J2C encoding, nothing will happen.
I was able to reproduce that lock-up on my personal Mac machine as well - started local main GUI, it locked the remote encoder, batch processor will not see remote encoding servers. I have 'only servers encode' ticked. Now when I send a job to batch converter, it stays at 'Encoding 0%' forever - because it is not allowed to encode locally, and also doesn't see remote encoders.
I remember Carl put some thinking into this in the latest test versions, but with your festival ahead, I would not advise you to use the recent test versions, as they have many new features and possible instabilities. Stick with 2.10.5 release version for now, and simply always start batch converter BEFORE you start DCP-o-matic main GUI, and always leave it running. That way, batch converter will always have priority to the encoding servers.
You may even put batch converter into auto start.
On my machine, I need to untick 'only servers encode' both in DOM and batch converter and quit both apps before the setting is changed (because they share the same prefs).
Tell us what types/numbers of CPUs all these machines use.
- Carsten
Last edited by Carsten on Sun Jun 18, 2017 1:12 am, edited 1 time in total.
-
carl
- Site Admin
- Posts: 2375
- Joined: Thu Nov 14, 2013 2:53 pm
Re: DCO-O-Matic Batch processor
Thanks Carsten, I think this is a good way to work around the bug in DCP-o-matic. I would agree with the suggestion to stick with 2.10.5 for production use.Stick with 2.10.5 release version for now, and simply always start batch converter BEFORE you start DCP-o-matic main GUI, and always leave it running. That way, batch converter will always have priority to the encoding servers.
-
Tudortech
- Posts: 10
- Joined: Fri Jun 16, 2017 3:32 pm
Re: DCO-O-Matic Batch processor
I am using 2.10.5 standard release version.
So far I have one physical server with one Xeon 8 core processor . The virtual servers are allocated 4 cores and 4gb memory each. Thise two servers are running server 2012 r2. The second server is a similar machine but with 12gb of memory and is split into 3 virtual servers with 4 virtual cores and 4gb each and running server 2008r2.
Thanks for the mention of the settings. I am glad you were able to create the issue as I thought I was going crazy. What is a good rule of thumb as to the number of threads a machine can reliably run ?
Thanks for all your help
Alan
So far I have one physical server with one Xeon 8 core processor . The virtual servers are allocated 4 cores and 4gb memory each. Thise two servers are running server 2012 r2. The second server is a similar machine but with 12gb of memory and is split into 3 virtual servers with 4 virtual cores and 4gb each and running server 2008r2.
Thanks for the mention of the settings. I am glad you were able to create the issue as I thought I was going crazy. What is a good rule of thumb as to the number of threads a machine can reliably run ?
Thanks for all your help
Alan
-
Carsten
- Posts: 2668
- Joined: Tue Apr 15, 2014 9:11 pm
- Location: Germany
Re: DCO-O-Matic Batch processor
Hmm. With the virtual servers, difficult...
One thing is sure - if the encode server runs out of RAM and starts swapping, you can forget about faster rendering. Swapping really brings them down.
Do you have choice to limit these machines to running only one virtual server, instead of two or three, or are these servers doing other important things at the same time?
4GB per virtual server is not much. Though I guess that with an 8core Xeon (it certainly has HT enabled), setting 8 threads for each server/encode server should still work with 4GB, at least as long as you work in 2k/HD. If you have a chance to monitor the swapping on these machines, during some tests, do so.
What type of machine/CPU is the one running the batch processor?
Maybe during one encoding run of a feature you may look at the fps achieved during encoding in the batch converter window with all servers active, and report back. We may be able to give an estimation wether that is acceptable or not.
Worst thing that could happen is that you won't allow a decent machine running batch processor to encode locally as well, and have the remote servers all swapping...
- Carsten
One thing is sure - if the encode server runs out of RAM and starts swapping, you can forget about faster rendering. Swapping really brings them down.
Do you have choice to limit these machines to running only one virtual server, instead of two or three, or are these servers doing other important things at the same time?
4GB per virtual server is not much. Though I guess that with an 8core Xeon (it certainly has HT enabled), setting 8 threads for each server/encode server should still work with 4GB, at least as long as you work in 2k/HD. If you have a chance to monitor the swapping on these machines, during some tests, do so.
What type of machine/CPU is the one running the batch processor?
Maybe during one encoding run of a feature you may look at the fps achieved during encoding in the batch converter window with all servers active, and report back. We may be able to give an estimation wether that is acceptable or not.
Worst thing that could happen is that you won't allow a decent machine running batch processor to encode locally as well, and have the remote servers all swapping...
- Carsten
-
Tudortech
- Posts: 10
- Joined: Fri Jun 16, 2017 3:32 pm
Re: DCO-O-Matic Batch processor
Hi
The box running the batch is a relatively old Core 2 quad 2.5ghz running win10. I did notice that it was only running at 100mb network. I changed it to a gigabit port and it sped up.This is only a test setup as I work out the kinks and tweek the speed. I will try a faster machine for the batch processor later today. I changed one of the physical servers to only run two virtual servers and that helped. when I am running something through it I am getting approx 10fps from four servers. The servers are maxing on cpu usage but barely using 1.5GB of 4GB memory. There is no swapping going on and the disk queue is very short. One server runs approx 4fps and the other 3 run approx 2fps. Trying to understand why the difference. I have all four servers running with 8 threads each. Any help you can give in tweeking the setup is greatly appreciated. We plan to add more servers, just waiting to see what local businesses will loan us. Have you seen better results with any particular operating system?
Thanks
Alan
The box running the batch is a relatively old Core 2 quad 2.5ghz running win10. I did notice that it was only running at 100mb network. I changed it to a gigabit port and it sped up.This is only a test setup as I work out the kinks and tweek the speed. I will try a faster machine for the batch processor later today. I changed one of the physical servers to only run two virtual servers and that helped. when I am running something through it I am getting approx 10fps from four servers. The servers are maxing on cpu usage but barely using 1.5GB of 4GB memory. There is no swapping going on and the disk queue is very short. One server runs approx 4fps and the other 3 run approx 2fps. Trying to understand why the difference. I have all four servers running with 8 threads each. Any help you can give in tweeking the setup is greatly appreciated. We plan to add more servers, just waiting to see what local businesses will loan us. Have you seen better results with any particular operating system?
Thanks
Alan