Heroku worker dyno giving R14 errors - problems with memory management - rmagick memory limit options

Question

I've tried to use a 2X worker dyno, waiting to not get R14 errors, but this is the result:

2013-06-04T13:03:30.723260+00:00 heroku[worker.1]: Process running mem=1047M(102.3%)
2013-06-04T13:03:30.723260+00:00 heroku[worker.1]: Error R14 (Memory quota exceeded)

However, the task for the worker is finished successfully.

Questions:

• How can I decrease the needed memory for my job? - I'm using rmagick gem to convert a list of images to a single multipage pdf.
• How dangerous is to keep the process like this (with R14 errors) as the final job is finishing successfully?

Thanks in advance

Answer 1

I finally found a solution. The problem was with rmagick. It gets all the memory it sees available. It doesn't matter if you use a 2X worker dyno instead of a 1X one. It grows to the maximum available resources. So, we have to set a limit. But in my case not all the limits worked.

You can make a system call directly this way:

convert -limit memory 0 -limit map 0 list_of_input_files output_file

This way, you avoid to use the memory cache, going directly to the disk cache. This is the only way I've found to avoid R14 errors in heroku. With other combinations like -limit memory 32 -limit map 64 or similar, it always gave me errors. I took the idea from here.

Of course, you could always use the rmagick library with this lines, however it didn't work for me and I did use the syscall approach explained before:

Magick.limit_resource(:memory, 0)
Magick.limit_resource(:map, 0)

UPDATE: I've used the nice command to ensure more priority. It seems it improves, but eventually I'm getting R14 errors, but it's true they are not so often (even with the same file!).