Single shared queue worker in Laravel multi-tenant app

Question

I'm building a multi-tenant Laravel application (on Laravel 5.3) that allows each tenant to have its own set of configurations for any supported Laravel settings. This is currently achieved by overriding the default Laravel Application with my own implementation that provides a custom configuration loader (overrides the default Illuminate\Foundation\Bootstrap\LoadConfiguration). The application detects the current tenant from the environment (either PHP's $_ENV or the .env file) on bootstrap and then loads the appropriate configuration files for the detected tenant.

The above approach works great for both the HTTP and Console kernels where each request/command has a limited life-cycle but I'm not sure how to approach the queue worker. I would like to have a single queue worker for all the tenants and I've already implemented a custom queue connector to add additional metadata when a queue job is scheduled, to make it possible to identify the tenant when the worker receives it.

The part on which I'm looking for your help is how to run each queue job in an isolated environment which I can bootstrap with my custom configuration.

A few possible solutions that I see would be:

• to run a custom queue worker that runs as a daemon and gets the job from the queue, but executes the job in a separate PHP process (created via exec()); once the job is executed, the worker gathers the results (status, exceptions, etc.) and finishes the job in the parent process (e.g. deletes the job, etc.)

• similar to the above, but run the job in a separate PHP thread instead of a separate process using RunKit Sandbox

• implement a solution that "reboots" the application once a queue job has been received (e.g. reloads configurations for the current tenant, resets any resolved dependencies, etc.)

What's important is that I'd like for this multi-tenant job execution to be transparent for the job itself so that jobs that are not designed to run in a multi-tenant environment (e.g. jobs from third party packages like Laravel Scout) can be handled without any modification.

Any suggestions on how to approach this?

Answer 1

We have pretty much the same situation. Here is our approach:

Service Provider

We have a ServiceProvider called BootTenantServiceProvider that bootstraps a tenant in a normal HTTP/Console request. It expects an environment variable to exist called TENANT_ID. With that, it will load all the appropriate configs and setup a specific tenant.

Trait with __sleep() and __wakeup()

We have a BootsTenant trait that we will use in our queue jobs, it looks like this:

trait BootsTenant
{
    protected $tenantId;

    /**
     * Prepare the instance for serialization.
     *
     * @return array
     */
    public function __sleep()
    {
        $this->tenantId = env('TENANT_ID');

        return array_keys(get_object_vars($this));
    }

    /**
     * Restore the ENV, and run the service provider
     */
    public function __wakeup()
    {
        // We need to set the TENANT_ID env, and also force the BootTenantServiceProvider again

        \Dotenv::makeMutable();
        \Dotenv::setEnvironmentVariable('TENANT_ID', this->tenantId);

        app()->register(BootTenantServiceProvider::class, [], true);
    }
}

Now we can write a queue job that uses this trait. When the job is serialized on the queue, the __sleep() method will store the tenantId locally. When it is unserialized the __wakeup() method will restore the environment variable and run the service provider.

Queue jobs

Our queue jobs simply need to use this trait:

class MyJob implements SelfHandling, ShouldQueue {
    use BootsTenant;

    protected $userId;

    public function __construct($userId)
    {
        $this->userId = $userId;
    }

    public function handle()
    {
        // At this point the job has been unserialized from the queue,
        // the trait __wakeup() method has restored the TENANT_ID
        // and the service provider has set us all up!

        $user = User::find($this->userId);
        // Do something with $user
    }
}

Conflict with SerializesModels

The SerializesModels trait that Laravel includes provides its own __sleep and __wakeup methods. I haven't quite figured out how to make both traits work together, or even if it's possible.

For now I make sure I never provide a full Eloquent model in the constructor. You can see in my example job above I only store IDs as class attributes, never full models. I have the handle() method fetch the models during the queue runtime. Then I don't need the SerializesModels trait at all.

Use queue:listen instead of --daemon

You need to run your queue workers using queue:listen instead of queue:work --daemon. The former boots the framework for every queue job, the latter keeps the booted framework loaded in memory.

At least, you need to do this assuming your tenant boot process needs a fresh framework boot. If you are able to boot multiple tenants in succession, cleanly overwriting the configs for each, then you might be able to get away with queue:work --daemon just fine.

Answer 2

To extend @jszobody his answer, see the TenantAwareJob trait build by multi-tenant Laravel package.

This does exactly what you want, before sleep encodes your tenant, when waking up it boots your tenant.

This trait also works with SerializesModels, so you can pass on your models.

Update

Since Laravel 6 this doesn't work anymore. The SerializeModels trait overides the __serialize and __unserialize functions.

New method is to register a service provider and hook in to the queue. This way you can add payload data and bootup the tentant before processing. Example:

    public function boot()
    {
        $this->app->extend('queue', function (QueueManager $queue) {
            // Store tenant key and identifier on job payload when a tenant is identified.
            $queue->createPayloadUsing(function () {
                return ['tenant_id' => TenantManager::getInstance()->getTenant()->id];
            });

            // Resolve any tenant related meta data on job and allow resolving of tenant.
            $queue->before(function (JobProcessing $jobProcessing) {
                $tenantId = $jobProcessing->job->payload()['tenant_id'];
                TenantManager::getInstance()->setTenantById($tenantId);
            });

            return $queue;
        });
    }

Inspired by laravel-multitenancy and tanancy queue driver