Best practice to deploy cron services on a production Node server?

Question

Currently, I have a complex production NodeJS server that sends an e-mail to users through Sendgrid with the press of a button. This server is then deployed using Docker and Convox into a different production environment called 'production', and tested at a "staging" level.

I want to automate the functionality of sending e-mails to users by using the cron NPM package. This cron job sends an e-mail to customers everyday automatically at a specified time. A problem I'm facing is the fact that the staging environment Node environment is the exact same as the production environment (both called 'production'). One possible solution is that I change the Docker/Convox configurations to have a separate Node environment for "staging" as well to prevent the cron job from running there.

However that got me thinking - what is the best practice to deploy cron-job services with functionality that is closely coupled to processes in my Node server?

Should the cron service be running on the same Node server? A concerned that was raised about this was that it may create a more "monolithic" instance with less fragmentation.

Should the cron service be running on a separate EC2 NodeJS or AWS Lambda?

I am trying to figure out the best way to structure the deployment of cron services so that it is extensible and scalable. In the future I wish to add more cron services with similar levels of functionality.

Answer 1

Should the cron service be running on the same Node server?

If the cron service does not need access to any live state within your existing node.js server, then you are free to either include it in the existing server or break it out into its own process more akin to a microservices architecture where you break apart things that don't have to be coupled. This is simply a design choice that depends upon a bunch of different things.

It's really a tradeoff between keeping things simple for deployment and management (one server process to deploy and manage) versus separating out things that don't need to be coupled and can be run/managed/coded/tested independently.

Typically, you'd make this tradeoff decision based on complexity. If the email code is fairly trivial and doesn't particularly impact the memory or CPU usage much, then there's really no particular advantage to creating another server to manage just like you wouldn't think about breaking out other small things from your server. But, if there's a real reason to manage it separately or you can scale easier by getting it's memory and CPU usage out of your main server process, then by all means break it out into its own server.

I'll offer you a couple examples that show the extremes.

Imagine you had 20 lines of code in your server that did some maintenance operation for you once a day (imagine it was log file management). They take about 10 seconds to run and really aren't complicated at all. Would you break that out into another node.js server. Probably not because there's no main benefit to breaking it out, yet there is added complexity to now have yet another process to run and manage.

Now imagine you had a process that was kicked off at a certain time at night that crawled the entire sites of 100 related companies gathering information from their sites. Because of the size of these sites, this could run for hours and could use a fair amount of CPU, memory, bandwidth and database cycles storing what it finds. Here there are real scale and management reasons to put this in its own separate process where it can be managed and scaled separately and where running it doesn't need to affect the ability of your other server to do its job effectively.

Should the cron service be running on a separate EC2 NodeJS or AWS Lambda?

This really depends upon your scale and management needs. See the above examples.

---

On your other topic...

One possible solution is that I change the Docker/Convox configurations to have a separate Node environment for "staging" as well to prevent the cron job from running there.

This seems to be a completely separate issue from what the rest of your question is about. It seems like it would be useful to have a separate environment variable that indicates STAGING. That way code that only cares about the difference between production and development can just look at the one existing environment variable (and you don't have to change any of that code that already does this). But code that wants to know the difference between staging and production can then examine the new variable.