Encoding Server Crashing

[mpheyse]

I'm an OpenDCP user I've been trying out DCP-o-matic for its remote server option. This would be a big advantage as currently I have to manually move frame groups to each "server" to do the Jpeg2k encoding (the only real time sink) with OpenDCP.

Here is our setup

Our new primary machine, built with DCP encoding in mind.
Intel i-Gen4 i5 4670K 3.6 GHz 4-core.
8GB ram, 64bit Win7

Our Sub Renders are (4 identical)
Intel i-Gen2 Pent G630T 3.2Ghz 2-core
4GB ram, 32bit WinXP

Our 'SuperComputer"
Intel i-Gen4 i7 4770K 4GHz 4-core 8-thread HT.
4GB ram, 32bit WinXP
The Problem is that the Server keeps crashing on every machine. If we run the command line dcpomatic_server_cli.exe, or the dcpomatic_server.exe

We've tried both 64 and 32 bit version of DCP-o-matic on the Primary, all servers are running the 32 bit.

We are getting frames done and sent back to the primary, but never very many (audio too).

The network usage is low (max is about 12% on Gigabit, with only the Primary and SuperComputer plugged in)

Server processor usage is only in the 60%-80% (across all cores) until it crashes.
Windows Firewall is off

All XP machines are on a private (no internet) lan, so they don't have Virus Checker installed. They are very clean installs, with just MSaccess as the only installed software. (SuperComputer has K-lite video codec pack installed)

The Primary only has Windows Defender, which is turned off

The Primary isn't running the server program, and DCP-o-matic is not crashing on it

Using less than max cores "dcpomatic_server_cli -t 7" still crashes.

I've been debating on where to make a bug report, since we've never had a server last through a small test (3 min video -> 16minute encode time)

When a server crashes its .exe, we can manually restart it and it will join in and do some work until it crashes again. But as the Sub Renders don't have full local consoles this is annoying to do.

DCP-o-matic Version 1,73.0 git 814382b0a3

What am I missing, is this a new bug? Have others gotten the server to work?

[carl]

Hi,

I would suspect lack of RAM on your encode servers for the crashes (or rather, lack of RAM that can be addressed by a single process under 32-bit operating systems). Maybe you could try dropping them right down to something like:

dcpomatic_server_cli -t 2

and seeing if that makes any difference. Do you need to run a 32-bit OS on the server machines?

Also, what resolution is your input video?

Best, Carl

[Carsten]

I'm still a fan of XP myself, but for a task like this, especially when you want to go 'big' with render nodes, etc, you have to use WIN7-64 or WIN8/8.1

As Carl says, 32Bit is prone to crashing with even rather low numbers of encode threads, like 4. At the same time it is a fact that the best encoding speeds can be achieved by some overthreading on most machines, like 4-12 on those machines you mention. As WIN32 is hard memory limited, I don't think there is a way to solve this. So if you keep the number of threads small to prevent crashing, the encoding speed will probably stay way below what is possible with that hardware and the whole network encoding will not make much sense.

If for some reason you have to stay with XP on the encode nodes, a bootable Linux from a USB stick with a dcp-o-matic CLI may be a solution at least for these machines.


Carl, is there some documentation on the server live images?

- Carsten

[carl]

Carl, is there some documentation on the server live images?

Not really They are just versions of Debian which boot up and start a DCP-o-matic encoding server. I will put some documentation on the website. On Linux it should be as easy as

Code: Select all

dd if=dcpomatic_server_image.iso of=/dev/sd_MY_MEMORY_STICK_BE_CAREFUL_TO_GET_THE_RIGHT_ONE bs=1M

then boot off the memory stick...

[carl]

Actually, there is brief mention at the bottom of this manual page:
http://dcpomatic.com/manual/html/ch10.html

Is there anything else anyone needs to know?

[mpheyse]

I didn't mention this but the "Severs" can run the regular (non-server app) just fine. So I wouldn't just jump to the memory issue.

yes the OS's need to be 32bit for there other uses

I only breefly tried the linux ISO, with a tool that "burns" it to a USB thumb drive. The live linux install doesn't prompt at a boot menu (like unix) so it tries to find a the "CD" and fails. I did't have a CD on me to burn this weekend

[mpheyse]

Hi,

I would suspect lack of RAM on your encode servers for the crashes (or rather, lack of RAM that can be addressed by a single process under 32-bit operating systems). Maybe you could try dropping them right down to something like:

dcpomatic_server_cli -t 2

I did try dcpomatic_server_cli -t 7 (on the 8thread machine) to leave a open thread hole for the OS/networking still crashed

but I will try that, when we have the system setup again. (only on weekends, and I will not be in town this weekend)

Hi,
Also, what resolution is your input video?

Not that we care about this perticular vidoe, it was just a test clip

Its just a test trailer we grabbed off youtube, for a short content test.

Its a Scope print on a 1080p frame

Here is the details.

Code: Select all

General
Complete name                  : E:\Lucy\Lucy_-_Trailer_(Official_-_HD).mp4
Format                         : MPEG-4
Format profile                 : Base Media
Codec ID                       : isom
File size                      : 50.5 MiB
Duration                       : 2mn 37s
Overall bit rate               : 2 696 Kbps
Writing application            : Lavf54.29.104

Video
ID                             : 1
Format                         : AVC
Format/Info                    : Advanced Video Codec
Format profile                 : High@L4.0
Format settings, CABAC         : Yes
Format settings, ReFrames      : 3 frames
Codec ID                       : avc1
Codec ID/Info                  : Advanced Video Coding
Duration                       : 2mn 37s
Bit rate                       : 2 435 Kbps
Width                          : 1 920 pixels
Height                         : 1 080 pixels
Display aspect ratio           : 16:9
Frame rate mode                : Variable
Frame rate                     : 23.976 fps
Minimum frame rate             : 23.974 fps
Maximum frame rate             : 23.981 fps
Color space                    : YUV
Chroma subsampling             : 4:2:0
Bit depth                      : 8 bits
Scan type                      : Progressive
Bits/(Pixel*Frame)             : 0.049
Stream size                    : 45.6 MiB (90%)

Audio
ID                             : 2
Format                         : AAC
Format/Info                    : Advanced Audio Codec
Format profile                 : LC
Codec ID                       : 40
Duration                       : 2mn 37s
Bit rate mode                  : Constant
Bit rate                       : 256 Kbps
Channel count                  : 2 channels
Channel positions              : Front: L R
Sampling rate                  : 44.1 KHz
Compression mode               : Lossy
Stream size                    : 4.80 MiB (9%)

here is the XML used in all of our tests

Code: Select all

<?xml version="1.0"?>
<Metadata>
  <Version>9</Version>
  <Name>Lucy</Name>
  <UseISDCFName>1</UseISDCFName>
  <DCPContentType>TLR</DCPContentType>
  <Container>239</Container>
  <Resolution>2K</Resolution>
  <Scaler>bicubic</Scaler>
  <WithSubtitles>0</WithSubtitles>
  <J2KBandwidth>100000000</J2KBandwidth>
  <ISDCFMetadata>
    <ContentVersion>1</ContentVersion>
    <AudioLanguage></AudioLanguage>
    <SubtitleLanguage></SubtitleLanguage>
    <Territory></Territory>
    <Rating></Rating>
    <Studio></Studio>
    <Facility></Facility>
    <PackageType></PackageType>
    <TempVersion>0</TempVersion>
    <PreRelease>0</PreRelease>
    <RedBand>0</RedBand>
    <Chain></Chain>
    <TwoDVersionOfThreeD>0</TwoDVersionOfThreeD>
    <MasteredLuminance></MasteredLuminance>
  </ISDCFMetadata>
  <VideoFrameRate>24</VideoFrameRate>
  <ISDCFDate>20141018</ISDCFDate>
  <AudioChannels>6</AudioChannels>
  <ThreeD>0</ThreeD>
  <SequenceVideo>1</SequenceVideo>
  <Interop>0</Interop>
  <Signed>1</Signed>
  <Encrypted>0</Encrypted>
  <Key>a1046d3455f7e78e5f305062e2926025</Key>
  <Playlist>
    <Content>
      <Type>FFmpeg</Type>
      <Path>e:\Lucy\Lucy_-_Trailer_(Official_-_HD).mp4</Path>
      <Digest>a069c10366642ec3f2fc66fe92f79ed1</Digest>
      <Position>0</Position>
      <TrimStart>0</TrimStart>
      <TrimEnd>0</TrimEnd>
      <VideoLength>3769</VideoLength>
      <VideoWidth>1920</VideoWidth>
      <VideoHeight>1080</VideoHeight>
      <VideoFrameRate>23.97602462768555</VideoFrameRate>
      <OriginalVideoFrameRate>23.97602462768555</OriginalVideoFrameRate>
      <VideoFrameType>0</VideoFrameType>
      <LeftCrop>0</LeftCrop>
      <RightCrop>0</RightCrop>
      <TopCrop>137</TopCrop>
      <BottomCrop>137</BottomCrop>
      <Scale>
        <Ratio>239</Ratio>
      </Scale>
      <ColourConversion>
        <InputGamma>2.2</InputGamma>
        <InputGammaLinearised>0</InputGammaLinearised>
        <Matrix i="0" j="0">0.4123908</Matrix>
        <Matrix i="0" j="1">0.3575843</Matrix>
        <Matrix i="0" j="2">0.1804808</Matrix>
        <Matrix i="1" j="0">0.212639</Matrix>
        <Matrix i="1" j="1">0.7151687</Matrix>
        <Matrix i="1" j="2">0.0721923</Matrix>
        <Matrix i="2" j="0">0.0193308</Matrix>
        <Matrix i="2" j="1">0.1191948</Matrix>
        <Matrix i="2" j="2">0.9505322000000001</Matrix>
        <OutputGamma>2.6</OutputGamma>
      </ColourConversion>
      <AudioGain>0</AudioGain>
      <AudioDelay>0</AudioDelay>
      <SubtitleXOffset>0</SubtitleXOffset>
      <SubtitleYOffset>0</SubtitleYOffset>
      <SubtitleScale>1</SubtitleScale>
      <AudioStream>
        <Selected>1</Selected>
        <Name>und; 2 channels</Name>
        <Id>2</Id>
        <FrameRate>44100</FrameRate>
        <Channels>2</Channels>
        <FirstAudio>0</FirstAudio>
        <Mapping>
          <ContentChannels>2</ContentChannels>
          <Gain Content="0" DCP="0">1</Gain>
          <Gain Content="0" DCP="1">0</Gain>
          <Gain Content="0" DCP="2">0</Gain>
          <Gain Content="0" DCP="3">0</Gain>
          <Gain Content="0" DCP="4">0</Gain>
          <Gain Content="0" DCP="5">0</Gain>
          <Gain Content="0" DCP="6">0</Gain>
          <Gain Content="0" DCP="7">0</Gain>
          <Gain Content="0" DCP="8">0</Gain>
          <Gain Content="0" DCP="9">0</Gain>
          <Gain Content="0" DCP="10">0</Gain>
          <Gain Content="0" DCP="11">0</Gain>
          <Gain Content="1" DCP="0">0</Gain>
          <Gain Content="1" DCP="1">1</Gain>
          <Gain Content="1" DCP="2">0</Gain>
          <Gain Content="1" DCP="3">0</Gain>
          <Gain Content="1" DCP="4">0</Gain>
          <Gain Content="1" DCP="5">0</Gain>
          <Gain Content="1" DCP="6">0</Gain>
          <Gain Content="1" DCP="7">0</Gain>
          <Gain Content="1" DCP="8">0</Gain>
          <Gain Content="1" DCP="9">0</Gain>
          <Gain Content="1" DCP="10">0</Gain>
          <Gain Content="1" DCP="11">0</Gain>
        </Mapping>
      </AudioStream>
      <FirstVideo>0</FirstVideo>
    </Content>
  </Playlist>
</Metadata>

I hope that helps

[mpheyse]

Carl, is there some documentation on the server live images?

Not really They are just versions of Debian which boot up and start a DCP-o-matic encoding server. I will put some documentation on the website. On Linux it should be as easy as

Code: Select all

dd if=dcpomatic_server_image.iso of=/dev/sd_MY_MEMORY_STICK_BE_CAREFUL_TO_GET_THE_RIGHT_ONE bs=1M

then boot off the memory stick...

We tried (mostly that), where we used Rufus http://rufus.akeo.ie/ to 'burn' the .iso to the usb key. Debian booted but dies when it tries to mount the cdrom directly, with no options to try any where else (like the usb-drive). So we figured it was a cd-only thing and we didn't have a cdr handy.

[carl]

Ah, I just noticed this:

1,73.0 git 814382b0a3

What language are the encoding machines using? I think there's a crashing bug with the server on non-English locales. I haven't managed to reproduce it yet, unfortunately.

Also, the server ISO was broken on 1.73.0. If you try 1.76.0 you might have more luck.