Waiting indefinitely for a message that may never arrive

Question

I have a Java typed actor that is responsible for filter/retry logic on an external resource that may be temporarily unavailable. The actor's fields and common methods are:

public class MyActorImpl implements MyActor {
    private static final long MINWAIT = 50;
    private static final long MAXWAIT = 1000;
    private static final long DEFAULTWAIT = 0;
    private static final double BACKOFFMULTIPLIER = 1.5;

    private long updateWait(long currentWait) {
        return Math.min(Math.max((long) (currentWait * BACKOFFMULTIPLIER), MINWAIT), MAXWAIT);
    }

    // mutable
    private long opWait = DEFAULTWAIT;
    private final Queue<OpInput> opBuffer = new ArrayDeque<>();

    // called from external actor
    public void operation(OpInput opInput) {
        operation(opInput, DEFAULTWAIT);
    }

    // called internally
    public void operation(OpInput opInput, long currentWait);
}

The actor has several operations that all have more or less the same retry/buffer logic; each operation has its own [op]Wait and [op]Buffer fields.

1. The parent actor calls void operation(OpInput opInput)
2. The preceding method calls void operation(OpInput opInput, long currentWait) using DEFAULTWAIT for the second parameter
3. If the currentWait parameter does not equal opWait then the input is stored in opBuffer, else the input is sent to the external resource.
4. If the external resource returns a success, then opWait is set to DEFAULTWAIT, and the contents of opBuffer are sent back through the operation(opInput) method. If the external resource (or more likely the network) returns an error, then I update opWait = updateWait(opWait) and schedule operation(opInput, opWait) on the actor system scheduler using a delay of opWait ms.

I.e. I'm using the actor system scheduler to implement exponential backoff; I'm using the currentWait parameter to identify the message that I'm retrying, and am buffering the other messages until the primary message is successfully processed by the external resource.

The problem is that if the scheduled operation(opInput, currentWait) message is lost then I'll be buffering messages forever because the currentWait == opWait guard will fail for all other messages. I could use something like spring-retry to implement exponential backoff but I don't see a way to merge the operations' retry loops, meaning that I could be using one thread per retry loop (whereas using the actor system's scheduler doesn't put much more of a strain on the system).

I'm looking for a more fault tolerant way to implement buffering and exponential backoff on the interface between an actor and an external resource without having to allocate too many resources to the task.

Answer 1

If I'm understanding you correctly, if the only problem is losing the scheduled message, why don't you just use something like the Reliable Proxy Pattern for that particular message, and then if it fails opWait = DEFAULTWAIT;

There are somethings about your code which I get, I don't get what you mean when you say that public void operation(OpInput opInput) is called externally. Do you mean that this method is interacting with the network, which uses a resources that is sometimes unavailable?

If I may can I suggest an alternative. From what I understand your main problem is you have a resources that is unavailable at times, so you have some sort of que/buffer you implement with some sort of wait logic so that the message will be processed once it is available again, which unfortunately involves some messages which could be lost and result in an infinite wait. I think you could achieve what you want using Futures with timeouts. Then retry if the future is not completed in that certain amount of time up to say 3 re-trys. You could even adjust this time based on the server load and how long it takes to complete a message. Hope that helps.