View Bug Details

IDProjectCategoryView StatusLast Update
0001427DCP-o-matic[All Projects] Bugspublic2018-12-11 23:44
ReporterrobnAssigned Tocarl 
PrioritynormalSeveritymajorReproducibilityalways
Status resolvedResolutionfixed 
Platformx86_64OSFedoraOS Version27
Product Version 
Target VersionFixed in Version 
Summary0001427: severe memory leak when using folder with image files
Description

I noticed that DOM 2.13.82 uses an ever increasing amount of memory while creating a DCP from an image sequence.
The memory use grows to the point that the machine starts to be unusable (and DOM has to be killed).
Trying older versions I found out that this problem is not new: it starts to appear between 2.13.57 and 2.13.61:

2.13.48 ok
2.13.55 ok
2.13.57 ok
2.13.61 bad
2.13.62 bad
2.13.64 bad
2.13.70 bad
2.13.81 bad
2.13.82 bad

Steps To Reproduce

I try to create a 4k DCP from a folder with 3840x1080 TIFF images.

The exploding memory usage is easy to observe: within the first 500 frames processed
the memory usage grows way beyond the approx 3.3 GB level it should stay.

My TIFF image sequence was created by putting 2 HD video streams side-by-side:

    ffmpeg -i file1.mov -i file2.mov -filter_complex hstack -pix_fmt rgb48le -f image2 frame-%06d.tiff

(but I suspect the problem happens with any (TIFF?) image sequence)

TagsNo tags attached.
Estimated work requiredUnknown

Activities

robn

2018-12-09 13:42

reporter   ~0002798

Oh, forgot to mention: stable DOM 2.12.13 is OK too!

Carsten

2018-12-09 14:28

manager   ~0002799

Last edited: 2018-12-09 14:29

View 2 revisions

Probably the move from ImageMagick to ffmpeg image file handling, so, probably not completely unexpected for Carl.

https://dcpomatic.com/forum/viewtopic.php?f=2&t=1180

  • Carsten

carl

2018-12-10 01:15

administrator   ~0002800

==12791== at 0x4C31E76: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==12791== by 0x4C31F91: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==12791== by 0x847B6E2: av_malloc (mem.c:87)
==12791== by 0x847B8E8: av_mallocz (mem.c:238)
==12791== by 0x742F76F: ff_fast_malloc (mem_internal.h:38)
==12791== by 0x742F76F: av_fast_padded_malloc (utils.c:78)
==12791== by 0x74160A3: tiff_unpack_strip (tiff.c:506)
==12791== by 0x74194AD: decode_frame (tiff.c:1325)
==12791== by 0x6FBD8B6: decode_simple_internal (decode.c:398)
==12791== by 0x6FBD8B6: decode_simple_receive_frame (decode.c:594)
==12791== by 0x6FBD8B6: decode_receive_frame_internal (decode.c:612)
==12791== by 0x6FC05D7: avcodec_send_packet (decode.c:674)
==12791== by 0x885BAFA: try_decode_frame (utils.c:3072)
==12791== by 0x8863E00: avformat_find_stream_info (utils.c:3895)
==12791== by 0x57F95D3: FFmpegImageProxy::image(boost::optional<dcp::Size>) const (ffmpeg_image_proxy.cc:141)
==12791== by 0x5886F73: PlayerVideo::make_image(boost::function<AVPixelFormat (AVPixelFormat)>, bool, bool) const (player_video.cc:131)
==12791== by 0x5886E15: PlayerVideo::image(boost::function<AVPixelFormat (AVPixelFormat)>, bool, bool) const (player_video.cc:112)
==12791== by 0x5743A6B: DCPVideo::convert_to_xyz(boost::shared_ptr<PlayerVideo const>, boost::function<void (dcp::NoteType, std::cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)>) (dcp_video.cc:96)
==12791== by 0x5743E74: DCPVideo::encode_locally() (dcp_video.cc:119)
==12791== by 0x5838133: J2KEncoder::encoder_thread(boost::optional<EncodeServerDescription>) (j2k_encoder.cc:335)
==12791== by 0x583E7E5: boost::_mfi::mf1<void, J2KEncoder, boost::optional<EncodeServerDescription> >::operator()(J2KEncoder, boost::optional<EncodeServerDescription>) const (mem_fn_template.hpp:165)
==12791== by 0x583E696: void boost::_bi::list2<boost::_bi::value<J2KEncoder
>, boost::_bi::value<boost::optional<EncodeServerDescription> > >::operator()<boost::_mfi::mf1<void, J2KEncoder, boost::optional<EncodeServerDescription> >, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf1<void, J2KEncoder, boost::optional<EncodeServerDescription> >&, boost::_bi::list0&, int) (bind.hpp:319)
==12791== by 0x583E521: boost::_bi::bind_t<void, boost::_mfi::mf1<void, J2KEncoder, boost::optional<EncodeServerDescription> >, boost::_bi::list2<boost::_bi::value<J2KEncoder>, boost::_bi::value<boost::optional<EncodeServerDescription> > > >::operator()() (bind.hpp:1294)
==12791== by 0x583E413: boost::detail::thread_data<boost::_bi::bind_t<void, boost::_mfi::mf1<void, J2KEncoder, boost::optional<EncodeServerDescription> >, boost::_bi::list2<boost::_bi::value<J2KEncoder
>, boost::_bi::value<boost::optional<EncodeServerDescription> > > > >::run() (thread.hpp:116)
==12791== by 0x9263BCC: ??? (in /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.65.1)
==12791== by 0x67BB6DA: start_thread (pthread_create.c:463)
==12791== by 0x6AF488E: clone (clone.S:95)
==12791==
==12791== 1,536,520 bytes in 718 blocks are definitely lost in loss record 743 of 743
==12791== at 0x4C31E76: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==12791== by 0x4C31F91: posix_memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==12791== by 0x847B6E2: av_malloc (mem.c:87)
==12791== by 0x847B8E8: av_mallocz (mem.c:238)
==12791== by 0x742F76F: ff_fast_malloc (mem_internal.h:38)
==12791== by 0x742F76F: av_fast_padded_malloc (utils.c:78)
==12791== by 0x74160A3: tiff_unpack_strip (tiff.c:506)
==12791== by 0x74194AD: decode_frame (tiff.c:1325)
==12791== by 0x6FBD8B6: decode_simple_internal (decode.c:398)
==12791== by 0x6FBD8B6: decode_simple_receive_frame (decode.c:594)
==12791== by 0x6FBD8B6: decode_receive_frame_internal (decode.c:612)
==12791== by 0x6FC05D7: avcodec_send_packet (decode.c:674)
==12791== by 0x6FC0936: compat_decode (decode.c:769)
==12791== by 0x57F9A53: FFmpegImageProxy::image(boost::optional<dcp::Size>) const (ffmpeg_image_proxy.cc:167)
==12791== by 0x5886F73: PlayerVideo::make_image(boost::function<AVPixelFormat (AVPixelFormat)>, bool, bool) const (player_video.cc:131)
==12791== by 0x5886E15: PlayerVideo::image(boost::function<AVPixelFormat (AVPixelFormat)>, bool, bool) const (player_video.cc:112)
==12791== by 0x5743A6B: DCPVideo::convert_to_xyz(boost::shared_ptr<PlayerVideo const>, boost::function<void (dcp::NoteType, std::
cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >)>) (dcp_video.cc:96)
==12791== by 0x5743E74: DCPVideo::encode_locally() (dcp_video.cc:119)
==12791== by 0x5838133: J2KEncoder::encoder_thread(boost::optional<EncodeServerDescription>) (j2k_encoder.cc:335)
==12791== by 0x583E7E5: boost::_mfi::mf1<void, J2KEncoder, boost::optional<EncodeServerDescription> >::operator()(J2KEncoder, boost::optional<EncodeServerDescription>) const (mem_fn_template.hpp:165)
==12791== by 0x583E696: void boost::_bi::list2<boost::_bi::value<J2KEncoder
>, boost::_bi::value<boost::optional<EncodeServerDescription> > >::operator()<boost::_mfi::mf1<void, J2KEncoder, boost::optional<EncodeServerDescription> >, boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf1<void, J2KEncoder, boost::optional<EncodeServerDescription> >&, boost::_bi::list0&, int) (bind.hpp:319)
==12791== by 0x583E521: boost::_bi::bind_t<void, boost::_mfi::mf1<void, J2KEncoder, boost::optional<EncodeServerDescription> >, boost::_bi::list2<boost::_bi::value<J2KEncoder>, boost::_bi::value<boost::optional<EncodeServerDescription> > > >::operator()() (bind.hpp:1294)
==12791== by 0x583E413: boost::detail::thread_data<boost::_bi::bind_t<void, boost::_mfi::mf1<void, J2KEncoder, boost::optional<EncodeServerDescription> >, boost::_bi::list2<boost::_bi::value<J2KEncoder
>, boost::_bi::value<boost::optional<EncodeServerDescription> > > > >::run() (thread.hpp:116)
==12791== by 0x9263BCC: ??? (in /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.65.1)
==12791== by 0x67BB6DA: start_thread (pthread_create.c:463)
==12791== by 0x6AF488E: clone (clone.S:95)

carl

2018-12-10 01:17

administrator   ~0002801

@carl above is output from dceecebd37d5e1b5c098ad1358310d3f276c703c

carl

2018-12-11 00:14

administrator   ~0002803

Above fixed by 6190f873eb6d1f2000d64f04e685575e687b61f1 in ffmpeg's git; needs merging into dcpomatic's ffmpeg branch.

carl

2018-12-11 23:44

administrator   ~0002806

Newer ffmpeg put into f851a375fc5bb9095def34c05a61f69e33139426 of DoM.

carl

2018-12-11 23:44

administrator   ~0002807

Should be fixed in 2.13.83.

Bug History

Date Modified Username Field Change
2018-12-09 13:11 robn New Bug
2018-12-09 13:42 robn Note Added: 0002798
2018-12-09 14:28 Carsten Note Added: 0002799
2018-12-09 14:29 Carsten Note Edited: 0002799 View Revisions
2018-12-10 01:15 carl Note Added: 0002800
2018-12-10 01:17 carl Note Added: 0002801
2018-12-11 00:14 carl Note Added: 0002803
2018-12-11 23:44 carl Note Added: 0002806
2018-12-11 23:44 carl Assigned To => carl
2018-12-11 23:44 carl Status new => resolved
2018-12-11 23:44 carl Resolution open => fixed
2018-12-11 23:44 carl Note Added: 0002807