Puppeteer slow down after a few hours

Question

I'm having some issues with my node app and it slowing down after a few hours on my EC2 server.

My app uses puppeteer, bull and mongodb, I have a bull queue which my app reads to process, once the job is complete it then writes the results to mongodb, so this process runs until the queue is complete (7000 jobs). When I start the app puppeteer takes around 5-10 seconds to return the results, the longer the process runs the slower it gets, after about 2-3 hours it starts taking 30-50 seconds to run a job. If I stop and start the process the jobs return back to the initial 5-10 seconds speed for 10 minutes until returning back to 30-50 seconds.

I've checked memory, it doesn't seem to be using excessive memory it stays around the same level the whole time. I've also checked the CPU and I don't think its running into any issues with that either.

Any ideas on what else could be causing the slot downs? I have noticed when the app starts the nodejs loop delay is 0.5ms after a few hours its around the 1-2ms range. Whats the best way to debug this?

Your help is much appreciated.

Thanks

Answer 1

You'll want to close and re-open the browser for every session if you don't already. This is a tricky endeavor and can be hard to manage as you'll need to know some inner-workings of puppeteer to make sure everything is "wired right". I maintain a docker image here, and the code to generate a clean instance of Chrome per-session is here. (dead link)