Multiple build queues in Jenkins

Question

I have a number of different jobs in Jenkins, partitioned out into projects with their own "pipelines" of build/test/analysis. Most of these jobs are actually remote commands instead of on-box-builds.

However, Jenkins out of the box only supports one queue for all builds. I want to define one queue per project (or view).

How would I accomplish this?

Answer 1

As far as I know this is not possible without changing the Jenkins code, but I think you could achieve the same goal with minimal maintenance using build slaves. Different builds can run concurrently on the slaves, or even on the same slave if you define multiple executors (if the slave machine has >1 CPU). You can label the slaves to control which jobs get executed on each one, so you could have a separate set of slaves for each of your pipelines.

Apart from the overhead of making sure the basic slave machines stay running, the Jenkins specific overhead for running a slave is minimal. You can use a process on the master to keep the slave JAR file and the build tools you need updated; at my shop we use a simple rsync script that runs every time the master or slave is restarted to copy the latest tools from the master to the slave and restart the slave process.

This approach also reduces the extent to which the Jenkins master is a single point of failure.

Answer 2

First of all, I'd like to point out that, like other answers describe, pegging the different build queues to different slaves would be the recommended solution, however there's a way to accomplish the same effect with just the master node.

There's a plugin called Throttle Concurrent Builds. It allows setting the build concurrency level either on a per-job basis, or across jobs.

The documentation was a bit vague, so these are the steps that worked for me:

1. After installing the plugin, go to Manage Jenkins -> Configure System.
2. In the Throttle Concurrent Builds section define the job groups and their respective concurrency levels.
3. Now for each job you want to throttle, go to its configuration, enable Throttle Concurrent Builds (General section), select Throttle this project as part of one or more categories, and assign one or more of the categories defined in the previous step (no need to set the limits here again).