The most likely cause is that you are using a 32-bit Windows version of DCP-o-matic. These versions can only address limited amounts of RAM.
If you are running 64-bit Windows, the best solution is to install the 64-bit version of DCP-o-matic.
If you are running 32-bit Windows, go to ‘Preferences’ and decrease the ‘number of threads DCP-o-matic should use’ setting (perhaps to 2). Then try your DCP conversion again.