Heroku Application Error - Memory quota exceeded on deploy - Scout shows 4,804,423 allocations

Question

I'm trying to find a memory bloat on my Rails application on Heroku.

I've read a bunch of articles about memory bloats and tried Scout to diagnose the problem.

I'm following Christoffers question about Heroku rising memory, but i have a different problem. Mine is almost always after a deploy, then the memory usage rises and my app breaks.

Here's an example from the Heroku Metrics:

As you can see, the deploy is completed an then after the first visit the app is breaking.

I've benchmarked what i could(checked my queries) but nothing slow is showing in development. Also it doesn't seem that it's one controller, it's just the first controller to be used after the deploy that breaks. I've tried several, and they all break when they are first to load.

Scout Dashboard - Memory Allocation Breakdown is showing this as an example of a time out request for CategoriesController:

The partial and layout are rendering as it should under the image

---

System

I'm running Rails 5.1.4 with Webpacker installed and configured for Angular 2. This means that i've added a Procfile in my root with:

web: bundle exec puma -p $PORT this is recommended by Heroku documentation

The app is deploying as it should with minor errors about ... has incorrect peer dependency ...

---

Healthy and sick log

I've tried to locate a healthy Heroku log and a sick Heroku log.

Please request if this has any interest.

---

Number of workers

I've read the article Ruby Memory Use and tried to decrease the number of workers to 1 in my config/puma.rb, but with no result.

---

---

Updated!

1 . Weird behaviour with the memory usage on Heroku:

It's starting after the daily restart around 100 MB but then after the timeout (first visit) the memory usage JUMPS to just beneath 500 MB.

2. @grizzthedj mentioned the public folder. It is 27.1 MB because of the heavy PDF.js plugin.

---

I'm looking for help to locate this problem. What could this be? Do you have any idea - let my try it!

I've tried what i found possible and nothing has helped yet.

Any help will be appreciated.

You can find the app on my Github page

Answer 1

There could be a few possibilities here.

First off, I suspect that there may be an issue with your webpacker build pipeline.

In your post, you mention that your pdf.js is 27.1 MB when deployed. When I download that file from the website, it is only ~514 KB.

I don't know what your webpacker build task is doing, but I suspect it may modifying this file unintentionally(via some wild card rule perhaps).

Secondly, there are also 3 other .js files in your /public/packs/ directory that are 7+ MB each. Thats almost 50 MB of .js alone! There are also duplicate pdf.js files in your public folder:

/public/pdfjs/build/pdf.js
/public/pdf_categories/build/pdf.js 

Hard to tell if these are the only issues, but for starters, I would do the following(if you are not doing already):

1. Double check that your webpacker is not adding any bloat to your pdf.js
2. Use webpacker to minify your .js and .css files
3. Ensure that you are using gzip compression for all static assets.

Answer 2

So... I want to share something:

I had another app - an old one - that had the same problem when deploying. After each deploy the app would break - Memory quota exceeded - and after a while it would be up and running again.

In the Heroku Logs is said, before the error:

[Webpacker] Compiling…

---

I figured out from a Github Issue that my config/webpacker.yml had a small difference from theirs.

In the bottom of the webpacker.yml there is the setting for the production and mine where:

compiling: true

This means

that on the first load on a page using a webpacker file, for example a javascript_pack_tack in rails, the compiling would start, even when Webpacker already compiled the files on deployment

You can see it's done on a deploy to Heroku - after the assets:precompile:

remote:        Running: rake assets:precompile
...
Done in 24.02s.
remote:        Webpacker is installed 🎉 🍰
remote:        Using /tmp/build_../config/webpacker.yml file for setting up webpack paths
remote:        Compiling…

My app wasn't using the compiled files from deployment - so the error was that i compiled twice!!. And the memory on a hobby plan on Heroku wasn't enough for the second one.

When i changed the settings to:

compile: false

It all worked as it should!

I hope it will help somebody.