Multiple Jenkins Masters using a Shared Slave pool

Question

I am trying to scale Jenkins for a large organization. Is there a way to have multiple Jenkins masters share a slave pool? For example, if I had 200 Jenkins Masters and I want them to share the same set of 50 Linux slaves.

That is, assuming each slave only has 1 executor, if Master A submits a job to the slave pool and it is running on Slave 1, if Master B submits a job to the slave pool, it would try to run on one of the other free slaves, since Slave 1 is already occupied.

I know multiple masters could share a single slave if I configured the slave to have a new workspace and executor for each master. However, I want to be able to set the slave up once, instead of having a slave.jar running on the slave for each master.

Cloudbees Op Center appears to provide this functionality, but looking for a way to do this with the Open Source version. If not, how difficult do you think it would be to extend Jenkins to have this functionality? I have Java development experience and have done a little work with Jenkins plugin development.

Thanks,

Answer 1

As you've noted it's not hard to share slaves between masters, just setup multiple workspaces and each master will install it's own slave jar. The trick is to share resources properly.

One such resource manager is Apache Mesos. A Jenkins Mesos plugin exists enabling the creation of slaves on a managed cluster.

This approach is very new and Ebay have blogged on how they've evolved their Jenkins setup to use Mesos:

• Ebay CI solutin part 1
• Ebay CI solution part 2

Hope this helps.

Answer 2

There is a Gearman Plugin developed by Open Stack to handle sharing slaves by multiple masters.