Is Orleans reminder execution interleaved?

Question

If there are two different reminders on the same grain activation to be fired at the same point, given that grain execution context is single-threaded, will both reminders be executed and interleaved at the same time?

Also, is the reminder execution limited by the default 30s timeout ?

Answer 1

Reminders are invoked using regular grain method calls: the IRemindable interface is a regular grain interface. IRemindable.ReceiveReminder(...) is not marked as [AlwaysInterleave], so it will only be interleaved if your grain class is marked as [Reentrant].

In short: no, reminder calls are not interleaved by default.

Reminders do not override the SiloMessagingOptions.ResponseTimeout value, so the default execution time will be 30s.

If you have a reminder that might need a very long time to execute, you can follow a pattern of starting the long-running work in a background task and ensuring that it is still running (not completed or faulted) whenever the relevant reminder fires.

Here is an example of using that pattern:

public class MyGrain : Grain, IMyGrain
{
    private readonly CancellationTokenSource _deactivating = new CancellationTokenSource();
    private Task _processQueueTask;
    private IGrainReminder _reminder = null;

    public Task ReceiveReminder(string reminderName, TickStatus status)
    {
        // Ensure that the reminder task is running.
        if (_processQueueTask is null || _processQueueTask.IsCompleted)
        {
            if (_processQueueTask?.Exception is Exception exception)
            {
                // Log that an error occurred.
            }

            _processQueueTask = DoLongRunningWork();
            _processQueueTask.Ignore();
        }

        return Task.CompletedTask;
    }

    public override async Task OnActivateAsync()
    {
        if (_reminder != null)
        {
            return;
        }

        _reminder = await RegisterOrUpdateReminder(
            "long-running-work",
            TimeSpan.FromMinutes(1),
            TimeSpan.FromMinutes(1)
        );
    }

    public override async Task OnDeactivateAsync()
    {
        _deactivating.Cancel(throwOnFirstException: false);

        Task processQueueTask = _processQueueTask;
        if (processQueueTask != null)
        {
            // Optionally add some max deactivation timeout here to stop waiting after (eg) 45 seconds

            await processQueueTask;
        }
    }

    public async Task StopAsync()
    {
        if (_reminder == null)
        {
            return;
        }
        await UnregisterReminder(_reminder);
        _reminder = null;
    }

    private async Task DoLongRunningWork()
    {
        // Log that we are starting the long-running work
        while (!_deactivating.IsCancellationRequested)
        {
            try
            {
                // Do long-running work
            }
            catch (Exception exception)
            {
                // Log exception. Potentially wait before retrying loop, since it seems like GetMessageAsync may have failed for us to end up here.
            }
        }
    }
}