DOM crashed over night (out of memory)

[markusm]

Hi,

i am just new to dcp-o-matic and already loving it. for now i created a few dcps from various sources, including an after effect project and more. never had a single problem till now. currently, i am making a dcp from a UHD Bluray (3840x1600 2.39:1, 4:2:0 YUV 10bit LE HDR with 5.1 DTS audio). Never did that before, but it looks like its working to some degree. It's painfully slow (~ 2.9 fps/s average), the master is a i7-4790k with 32 GB RAM, one encoding server (i7-4700mq, 16 GB RAM, 1000MBits/s cable network). somehow, tonight DOM crashed or was crashed by windows with an out of memory exception, which is strange because for the most time only 4-5 of 32 GB were used.

windows event log:

Code: Select all

Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: dcpomatic2.exe (6532) consumed 37867864064 Bytes, Dropbox.exe (10572) consumed 220708864 Bytes and dwm.exe (1212) consumed 193380352 Bytes. 

encoding log:

Code: Select all

14.03.2017 04:25:53: FFmpeg: [truehd @ 00000000073F90B0] Lossless check failed - expected 47, calculated ca.
14.03.2017 04:27:15: FFmpeg: [truehd @ 00000000073F90B0] Lossless check failed - expected 47, calculated ca.
14.03.2017 04:30:02: FFmpeg: [truehd @ 00000000073F90B0] Lossless check failed - expected 47, calculated ca.
14.03.2017 04:30:49: Checked X:\UHD-Test\info\239_4K_e4a1fe04deaeb067c209a2048e598078_24_200000000_P_S_0_586916000 (which exists) length is 6995376 perms are 438
14.03.2017 04:31:31: FFmpeg: [truehd @ 00000000073F90B0] Lossless check failed - expected 47, calculated ca.
14.03.2017 04:31:44: WARNING: avcodec_decode_audio4 failed (-1094995529)
14.03.2017 04:31:44: FFmpeg: [truehd @ 00000000073F90B0] too many audio samples in frame
14.03.2017 04:31:44: WARNING: avcodec_decode_audio4 failed (-1094995529)
14.03.2017 04:31:44: FFmpeg: [truehd @ 00000000073F90B0] substream 1 length mismatch
14.03.2017 04:31:44: WARNING: avcodec_decode_audio4 failed (-1094995529)
14.03.2017 04:31:44: FFmpeg: [truehd @ 00000000073F90B0] Lossless check failed - expected 22, calculated f8.
14.03.2017 04:31:44: Terminating thread 1 of 12
14.03.2017 04:31:44: Thread terminated
14.03.2017 04:31:44: Terminating thread 2 of 12
14.03.2017 04:31:44: Thread terminated
14.03.2017 04:31:44: Terminating thread 3 of 12
14.03.2017 04:31:44: Thread terminated
14.03.2017 04:31:44: Terminating thread 4 of 12
14.03.2017 04:31:44: Thread terminated
14.03.2017 04:31:44: Terminating thread 5 of 12
14.03.2017 04:31:44: Thread terminated
14.03.2017 04:31:44: Terminating thread 6 of 12
14.03.2017 04:31:44: Thread terminated
14.03.2017 04:31:44: Terminating thread 7 of 12
14.03.2017 04:31:44: Thread terminated
14.03.2017 04:31:44: Terminating thread 8 of 12
14.03.2017 04:31:44: Thread terminated
14.03.2017 04:31:44: Terminating thread 9 of 12
14.03.2017 04:31:44: Thread terminated
14.03.2017 04:31:44: Terminating thread 10 of 12
14.03.2017 04:31:44: Thread terminated
14.03.2017 04:31:44: Terminating thread 11 of 12
14.03.2017 04:31:44: Thread terminated
14.03.2017 04:31:44: Terminating thread 12 of 12
14.03.2017 04:31:44: Thread terminated
14.03.2017 04:31:44: ERROR: Zu wenig Speicher
14.03.2017 04:31:44: ERROR: Das System hat zuwenig freien Speicher. Wenn Sie das Programm unter einem 32-Bit Betriebssystem ausführen, versuchen Sie, die Anzahl der Encoding Threads in den Voreinstellungen n zu reduzieren.
14.03.2017 04:31:44: ERROR: Error in job: Zu wenig Speicher (Das System hat zuwenig freien Speicher. Wenn Sie das Programm unter einem 32-Bit Betriebssystem ausführen, versuchen Sie, die Anzahl der Encoding Threads in den Voreinstellungen n zu reduzieren.)

(roughly translated: to less memory. if the program is running under a 32 bit operating system, try reducing the number of encoding threads)

system information:
master:
Intel Core i7-4790k
32 GB RAM
4x 1TB SSD
1x Nvidia GTX 1070
Gigabit Network
Windows 10 Pro

slave:
Intel Core i7-4700MQ
16 GB RAM
2x 1 TB SSD
Gigabit Network
Windows 10 Pro

DCP Information:
Source: as above
Target: 4k DCP 200 MBIt/s SMTPE

[Carsten]

Normally this indicates too many encoding threads, but it seems you only used 12. So I can only assume there's a bug and/or problem with the footage you are using. I have seen strange things with problematic audio codecs (like Dolby TrueHD).

I also guess there hasn't been much testing on UHD content yet.

- Carsten

[markusm]

Yep, i used only 4/8 Threads on the master, and 8/8 on the client. I'll try another run tonight, after assigning a 256 GB (SSD) swap file. maybe brute force will help

That's the info from ffprobe.log regarding the codecs:

Code: Select all

Stream #0:0(eng): Video: hevc (Main 10), yuv420p10le(tv, bt2020nc/bt2020/smpte2084), 3840x1600, SAR 1:1 DAR 12:5, 23.98 fps, 23.98 tbr, 1k tbn, 23.98 tbc (default)
...
Stream #0:1(en): Audio: dts (DTS-HD MA), 48000 Hz, 5.1(side), s16p (default)

[carl]

I think this is most likely to be a bug in the decoder. I have recently changed a lot of the internals of how the decoder works, which will hopefully make it more robust against this sort of error. The changes haven't been released yet, though. The best thing may be for me to make a test version for you. It would be difficult to debug without the source video file, which I guess is large.

[Milling]

That's why I never use the maximum of 12 threads.

[Carsten]

12 is not a maximum setting, and it is no problem to set even higher thread numbers on machines with a 64Bit OS.

It doesn't make sense, however, to set many more threads on machines that have fewer HW threads available. At some point, it will slow down the machine, as every encoding thread needs memory as well, and may force the machine into swapping. And then performance goes belly up.

With the recent releases of DOM, the thread setting that DOM chooses automatically after initial installation doesn't need to be touched. If you messed around with it, you may need to check your actual machine specs to find out the best setting.

e.g.
i7-47xx/4core HT - 8 threads
i5/2core HT - 4 threads
i7-5820/6core HT - 12 threads

If you choose a thread # setting below your actual available HT HW threads, it will cost you encoding speed. If you choose a setting slightly above your available HT HW threads, it's no problem. If you go well beyond, e.g. three times, four times, etc., it will slow down the encoding. And it will certainly crash if you run a 32Bit version/OS.

- Carsten

[markusm]

Sorry for the late answer. I think it's safe to assume that my problem was caused by a corrupt input file and not by number of threads, which was lower than maximum possible