Correct way to implement standalone Grails batch processes?

Question

I want to implement the following:

1. Web application in Grails going to MongoDB database
2. Long-running batch processes populating and updating that database in the background

I would like for both of them to reuse the same Grails services and same GORM domain classes (using mongodb plugin for Grails).

For the Web application everything should work fine, including the dynamic GORM finder methods.

But I cannot figure out how to implement the batch processes.

a. If I implement them as Grails service methods, their long-running nature will be a problem. Even wrapping them in some async executors will unnecessarily complicate everything, as I'd like them each to be a separate Java process so they can be monitored and stopped easily and separately.

b. If I implement them as src/groovy scripts and try to launch from command line, I cannot inject the Grails services properly (ApplicationHolder method throws NPE) or get the GORM finder methods to work. The standalone GORM guides all have Hibernate in mind and overall it seems not the right route to pursue.

c. I considered the 'batch-launcher' Grails plugin but it failed to install and seems a bit abandoned.

d. I considered the 'run-script' Grails command to run the scripts from src/groovy and it seems it might actually work in development, but seems not the right thing to do in production.

I cannot be the only person with such a problem - so how is it generally solved?

How do people run standalone scripts sharing the code base and DB with their Grails applications?

Answer 1

Since you want the jobs processing to be in a separate JVM from your front-end application, the easiest way to do that is to have two instances of Grails running, one for the front-end that serves web requests, and the other to deal with job processing.

Thankfully, the rich ecosystem of plugins for Grails makes this sort of thing quite easy, though perhaps not the most efficient, since running an entire Grails application just for processing is a bit overkill.

The way I tend to go about it is to write my application as one app, with services that take care of the job processing. These services are tied to the RabbitMQ plugin, so the general flow is that the web requests (or quartz scheduled jobs) put jobs into a work queue, and then the worker services take care of processing them.

The advantage with this is that, since it's one application, I have full access to all of the domain objects, etc., and I can leverage the dissconnected nature of a message queue to scale out my front- and back-ends seperately without needing more than one application. Instead, I can just install the same application multiple times and configure the number of threads dedicated to processing jobs and/or the queues that the job processors are looking at.

So, with this setup, for development, I will usually just set the number of job processing threads to whatever makes sense for the development work I'm doing, and then just a simple grails run-app, and I have a fully functional system (assuming I have a RabbitMQ server running to use as well).

Then, when I go to deploy into production, I deploy 2 instances of the application, one for the front-end work and the other for the back-end work. I just configure the front-end instance to have 1 or 0 threads for processing jobs, and the back-end instance I give many more threads. This lets me update either portion as needed or spin up more instances if I need to scale one part or the other.

I'm sure there are other ways to do this, but I've found this to be both really easy to develop (since it's all one application), and also really easy to deploy, scale, and maintain.