Problems importing a large J2C sequence

Anything and everything to do with DCP-o-matic.
Post Reply
gi-jones
Posts: 8
Joined: Fri May 26, 2023 12:57 pm

Problems importing a large J2C sequence

Post by gi-jones »

Hi all
I'm trying to make a DCP of a full feature film, but DOM becomes unresponsive when I try to import the file sequence.

The sequence consiste of roughly 130000 J2C files. They are DCI compliant in 4k and made from Resolve. I have tested the full workflow with the first minute, and that works fine. But when I point DOM to the folder containing the full file set, it spends roughly 2 hours indexing (or whatnot). When that is done, the program simply crashes. One time I managed to save the project, but every time I opened it afterwards it crashed again.

Is this not a reasonable use case? Or am I doing something stupid?
carl
Site Admin
Posts: 2387
Joined: Thu Nov 14, 2013 2:53 pm

Re: Problems importing a large J2C sequence

Post by carl »

This should work - it's a shame it took 2 hours. Were the files stored on a local drive or somewhere else?
gi-jones
Posts: 8
Joined: Fri May 26, 2023 12:57 pm

Re: Problems importing a large J2C sequence

Post by gi-jones »

Should it happen significantly faster than 2 hours?

The files are on an external USB-C SSD disk. ExFAT

The thing is - this morning I tried using ASDCP (command line) to wrap an MXF, which worked fine. So I'm pretty sure the disk is ok.
gi-jones
Posts: 8
Joined: Fri May 26, 2023 12:57 pm

Re: Problems importing a large J2C sequence

Post by gi-jones »

So I tried moving the files from the external drive to my internal SSD, and suddenly the ingestion only took a couple of minutes.

I guess the problem must be with using the external drive somehow. It's a very fast one though - Samsung T7. The ExFat formatting might be a factor I guess...
Last edited by gi-jones on Fri May 26, 2023 7:52 pm, edited 1 time in total.
carl
Site Admin
Posts: 2387
Joined: Thu Nov 14, 2013 2:53 pm

Re: Problems importing a large J2C sequence

Post by carl »

Did it crash when you used the internal SSD?

It's odd that it took so long with the external. There shouldn't be too much going on with the files; just finding them all, basically. I'll see if I can see any inefficiency in that code.
Carsten
Posts: 2674
Joined: Tue Apr 15, 2014 9:11 pm
Location: Germany

Re: Problems importing a large J2C sequence

Post by Carsten »

Maybe - just for the fun of it, you could try to format the same external disk to NTFS (or whatever the native OS filesystem is), and just try the sequence import again. Maybe something weird happens because of ExFAT.
IoannisSyrogiannis
Posts: 130
Joined: Mon Nov 13, 2017 8:40 pm

Re: Problems importing a large J2C sequence

Post by IoannisSyrogiannis »

Working with Windows at work, I get a lot of ExFAT drives.
Someone had the great idea to use a couple for a film festival.
Copying a couple of terabytes took almost a day each time.
Bottom line: my bet is on ExFAT, not external vs internal drive.
gi-jones
Posts: 8
Joined: Fri May 26, 2023 12:57 pm

Re: Problems importing a large J2C sequence

Post by gi-jones »

carl wrote: Sun May 28, 2023 7:13 pm Did it crash when you used the internal SSD?
Nope, everything worked fine from there.

I agree that ExFat is probably the issue. It's not possible for me to reformat the drive right now unfortunately.
worldofcrap
Posts: 13
Joined: Tue Sep 20, 2022 11:29 pm

Re: Problems importing a large J2C sequence

Post by worldofcrap »

You don’t mention if you’re on a Mac, windows or Linux machine. If it’s MacOS, the support for ExFAT file system is DOG slow, particularly with many small files. Windows is slightly better, but it’s still nowhere near as fast as NTFS.
Post Reply