Can I throttle requests made by a distributed app?

Question

My application makes Web Service requests; there is a max rate of requests the provider will handle, so I need to throttle them down.

When the app ran on a single server, I used to do it at the application level: an object that keeps track of how many requests have been made so far, and waits if the current request makes it exceeds the maximum allowed load.

Now, we're migrating from a single server to a cluster, so there are two copies of the application running.

• I can't keep checking for the max load at the application code, because the two nodes combined might exceed the allowed load.
• I can't simply reduce the load on each server, because if the other node is idle, the first node can send out more requests.

This is a JavaEE 5 environment. What is the best way to throttle the requests the application sends out ?

Answer 1

The N nodes need to communicate. There are various strategies:

• broadcast: each node will broadcast to everybody else that it's macking a call, and all other nodes will take that into account. Nodes are equal and maintain individial global count (each node know about every other node's call).
• master node: one node is special, its the master and all other nodes ask permission from the master before making a call. The master is the only one that know the global count.
• dedicated master: same as master, but the 'master' doesn't do calls on itslef, is just a service that keep track of calls.

Depending on how high do you anticipate to scale later, one or the other strategy may be best. For 2 nodes the simplest one is broadcast, but as the number of nodes increases the problems start to mount (you'll be spending more time broadcasting and responding to broadcats than actually doing WS requests).

How the nodes communicate, is up to you. You can open a TCP pipe, you can broadcats UDP, you can do a fully fledged WS for this purpose alone, you can use a file share protocol. Whatever you do, you are now no longer inside a process so all the fallacies of distributed computing apply.