Store and forward HTTP requests with retries?

Question

Twilio and other HTTP-driven web services have the concept of a fallback URL, where the web services sends a GET or POST to a URL of your choice if the main URL times out or otherwise fails. In the case of Twilio, they will not retry the request if the fallback URL also fails. I'd like the fallback URL to be hosted on a separate machine so that the error doesn't get lost in the ether if the primary server is down or unreachable.

I'd like some way for the secondary to:

1. Store requests to the fallback URL
2. Replay the requests to a slightly different URL on the primary server
3. Retry #2 until success, then delete the request from the queue/database

Is there some existing piece of software that can do this? I can build something myself if need be, I just figured this would be something someone would have already done. I'm not familiar enough with HTTP and the surrounding tools (proxies, reverse proxies, etc.) to know the right buzzword to search for.

Answer 1

There are couple of possibilities.

One option is to use Common Address Redundancy Protocol or carp. Brief description from the man page follows.

"carp allows multiple hosts on the same local network to share a set of IP addresses. Its primary purpose is to ensure that these addresses are always available, but in some configurations carp can also provide load balancing functionality."

It should be possible be configure IP balancing such that when a primary or master http service fails, the secondary or backup http service becomes the master. carp is host oriented as opposed to application service. So when the http service goes down, it should also take down the network interface for carp to do its thing. This means you would need more than one IP address in order to log into the machine and do maintenance. You would need a script to do the follow-up actions once the original service comes back online.

The second option is to use nginx. This is probably better suited to what you are trying to do.

Many years ago I needed something similar to what are trying to do and I ended up hacking together something that did it. Essentially it was a switch. When 'A' fails, switch over to 'B'. Re-sync process was to take time stamped logs from 'B' and play them back to 'A', once 'A' was back online.