Golang background processing

Question

How can one do background processing/queueing in Go?

For instance, a user signs up, and you send them a confirmation email - you want to send the confirmation email in the background as it may be slow, and the mail server may be down etc etc.

In Ruby a very nice solution is DelayedJob, which queues your job to a relational database (i.e. simple and reliable), and then uses background workers to run the tasks, and retries if the job fails.

I am looking for a simple and reliable solution, not something low level if possible.

Answer 1

While you could just open a goroutine and do every async task you want, this is not a great solution if you want reliability, i.e. the promise that if you trigger a task it will get done.

If you really need this to be production grade, opt for a distributed work queue. I don't know of any such queues that are specific to golang, but you can work with rabbitmq, beanstalk, redis or similar queuing engines to offload such tasks from your process and add fault tolerance and queue persistence.

Answer 2

A simple Goroutine can make the job: http://golang.org/doc/effective_go.html#goroutines

Open a gorutine with the email delivery and then answer to the HTTP request or whatever

If you wish use a workqueue you can use Rabbitmq or Beanstalk client like: https://github.com/streadway/amqp https://github.com/kr/beanstalk

Or maybe you can create a queue in you process with a FIFO queue running in a goroutine https://github.com/iNamik/go_container

But maybe the best solution is this job queue library, with this library you can set the concurrency limit, etc: https://github.com/otium/queue

import "github.com/otium/queue"

q := queue.NewQueue(func(email string) {
     //Your mail delivery code
}, 20)

q.Push("[email protected]")

Answer 3

I have created a library for running asynchronous tasks using a message queue (currently RabbitMQ and Memcache are supported brokers but other brokers like Redis or Cassandra could easily be added).

You can take a look. It might be good enough for your use case (and it also supports chaining and workflows).

https://github.com/RichardKnop/machinery

It is an early stage project though.

Answer 4

You can also use goworker library to schedule jobs.

http://www.goworker.org/