Page 1 of 1

Small encoding farm problems

Posted: Thu Jan 20, 2022 9:31 pm
by Kewl
So I set up this small encoding farm with eight computers, all macOS, some with Big Sur, some with Monterey. 96 threads total. DOM 2.15.186.

But I get this error that points to the master computer:
Could not open file /Users/daniel/Desktop/Fellini Premières Fois/info/185_2K_dc44c9da55df01170d55afe11b29b539_24_150000000_P_S_0_180132000 for read/write (24) with /Users/daniel/Desktop/Fellini Premières Fois/info/185_2K_dc44c9da55df01170d55afe11b29b539_24_150000000_P_S_0_180132000
Could it be because of the name of the project "Fellini Premières Fois"?

Re: Small encoding farm problems

Posted: Thu Jan 20, 2022 9:54 pm
by carl
It looks like error 24 is basically "too many open files". I think on the master with 96 threads there might indeed be a lot of open files. It looks like there's some suggestions here about how to raise the limit. I'd be interested to know how you get on if you tried any of those...

Re: Small encoding farm problems

Posted: Thu Jan 20, 2022 10:48 pm
by Kewl
I'll check that out.

Re: Small encoding farm problems

Posted: Fri Jan 21, 2022 2:41 pm
by Kewl
In the meantime, what seems to be the number of threads before hitting the "error 24" wall? If all threads are used, the 12 iMacs farm is at least 48 threads, and could be at big as 96 threads.

Is the host's RAM also a factor? And the video and audio codecs of the source file?

Re: Small encoding farm problems

Posted: Fri Jan 21, 2022 5:33 pm
by Kewl
I reduced the size of the farm to 56 threads (iMac M1 8 threads + 2 * Mac Pro Intel 24 threads). All went well. DOM 2.15.187

Re: Small encoding farm problems

Posted: Fri Jan 21, 2022 7:29 pm
by carl
I have no idea about thread numbers; really I should look to see what these open files are in case there's something silly that DoM is doing which could be fixed.

I don't think RAM or codecs are a factor - the max file limit is just an arbitrary limit set by the operating system.

Re: Small encoding farm problems

Posted: Fri Jan 21, 2022 10:13 pm
by Carsten
I remember that I tried insanely high thread numbers occasionally. Up to 96 according to my logs, and never had problems. However, that was clearly all with 2.14.x So it's definitely not just the thread number setting.

AND - that was only with low numbers of remote machines. And all in WIN7-64Pro. May be different with a high number of clients.

Re: Small encoding farm problems

Posted: Fri Jan 21, 2022 10:27 pm
by Kewl
Hum... When it didn't work yesterday, the farm had one host and seven servers (4, 8 or 24 threads per server). So maybe the number of servers is a factor...

Re: Small encoding farm problems

Posted: Sun Jan 23, 2022 8:46 pm
by Kewl
Would it be possible to have a certain number of hosts with the same encode servers? Or can an encode server have more than one host, possibly at the same time?

Re: Small encoding farm problems

Posted: Mon Jan 24, 2022 4:39 pm
by Kewl
Kewl wrote: Sun Jan 23, 2022 8:46 pm can an encode server have more than one host, possibly at the same time?
To answer my question: yes, it works!
Capture d’écran, le 2022-01-24 à 11.30.55.png
Capture d’écran, le 2022-01-24 à 11.30.55.png (146.12 KiB) Viewed 2120 times