Menu
We have this architecture with 2 node processes.
One polls a private API and pushes the changes to the second node if any. The second node process the data and calls a bunch other API's and eventually emits a change event to the client, a HTML5 website, with socket.io
This second node will always process the data and will always emit changes even if no clients are connected. So in my opinion the CPU or mem usage is not that greatly affected by the number of connected clients. Also note that this architecture is still running on a private staging environment.
Everything runs fine and we're ready to go live until we noticed after couple of days, maybe a week, the second node suddenly gets extremely slow while the first node is still fine.
It gets so bad that even the connection between the two nodes gets timed out and they are on the same network over localhost. It also takes more then 10 seconds to browse to the socket.io/socket.io.js file.
I know its very hard to understand the problem without seeing any code but I'm kinda pulling my hair out because we have to go live in couple of days and my logs are not revealing anything and google isn't helping either.
Whats a good practice towards building Have you ever experienced anything like this? What was the problem and how did you fix it?
Whats a good monitor and profiler for node.js? (preferably free)
What are good practices towards building a node.js app with makes a lot of outgoing API calls?
Anything or anyone that could help me in the right direction of solving or even discovering the actual problem will be greatly appreciated!
Thank you!
946
Node. js programs can be slow due to a CPU/IO-bound operation, such as a database query or slow API call. For most Node. js applications, data fetching is done via an API request and a response is returned.
At least 2GB of RAM. At least 4 vCPUs. At least 25GB of storage.
require() — the nodejs module loader is really slow and may be to blame. require() is synchronous, which means blocking when loading each module. The quick and dirty technique is to use as fewer require() statements as possible.
Since Node. js is single-threaded and non-blocking, you can achieve higher concurrency with the same resources". And when you read about what it's bad at it usually goes like this: "Since Node. js is single-threaded, CPU-intensive tasks will block all requests from completing, until the task is completed.
Never experienced anything like this but may be the second node is blocking the event loop by doing CPU intensive work or waiting for some resource synchronously.
Add some logging in your code to see how much time second node is taking for processing each change pushed by first node. May be some type of change consumes CPU for 10 seconds or so to complete.
You should also start monitoring memory, CPU and network connections. When things slow down your monitoring will provide some clue as to where is the bottle neck.
For monitoring you can try following 3 tools
Also read http://nodetime.com/blog/monitoring-nodejs-application-performance
110
It sounds like you have a memory leak somewhere in the second node, maybe from calling too many anonymous functions etc... do you notice your RAM usage slightly creeping up as it runs?
26
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
