Small encoding farm problems

[Kewl]

So I set up this small encoding farm with eight computers, all macOS, some with Big Sur, some with Monterey. 96 threads total. DOM 2.15.186.

But I get this error that points to the master computer:

Could not open file /Users/daniel/Desktop/Fellini Premières Fois/info/185_2K_dc44c9da55df01170d55afe11b29b539_24_150000000_P_S_0_180132000 for read/write (24) with /Users/daniel/Desktop/Fellini Premières Fois/info/185_2K_dc44c9da55df01170d55afe11b29b539_24_150000000_P_S_0_180132000

Could it be because of the name of the project "Fellini Premières Fois"?

[carl]

It looks like error 24 is basically "too many open files". I think on the master with 96 threads there might indeed be a lot of open files. It looks like there's some suggestions here about how to raise the limit. I'd be interested to know how you get on if you tried any of those...

[Kewl]

I'll check that out.

[Kewl]

In the meantime, what seems to be the number of threads before hitting the "error 24" wall? If all threads are used, the 12 iMacs farm is at least 48 threads, and could be at big as 96 threads.

Is the host's RAM also a factor? And the video and audio codecs of the source file?

[Kewl]

I reduced the size of the farm to 56 threads (iMac M1 8 threads + 2 * Mac Pro Intel 24 threads). All went well. DOM 2.15.187

[carl]

I have no idea about thread numbers; really I should look to see what these open files are in case there's something silly that DoM is doing which could be fixed.

I don't think RAM or codecs are a factor - the max file limit is just an arbitrary limit set by the operating system.

[Carsten]

I remember that I tried insanely high thread numbers occasionally. Up to 96 according to my logs, and never had problems. However, that was clearly all with 2.14.x So it's definitely not just the thread number setting.

AND - that was only with low numbers of remote machines. And all in WIN7-64Pro. May be different with a high number of clients.

[Kewl]

Hum... When it didn't work yesterday, the farm had one host and seven servers (4, 8 or 24 threads per server). So maybe the number of servers is a factor...

[Kewl]

Would it be possible to have a certain number of hosts with the same encode servers? Or can an encode server have more than one host, possibly at the same time?

[Kewl]

can an encode server have more than one host, possibly at the same time?

To answer my question: yes, it works!

Capture d’écran, le 2022-01-24 à 11.30.55.png