Programmatically add nodes to a load balancer like Haproxy?

Question

I'm very sure this problem has been solved, but I can't find any information anywhere about it...

How do sysadmins programmatically add a new node to an existing and running load balancer ? Let's say I have a load balancer running and already balancing say my API server between two EC2 instances, and suddenly there's a traffic spike and I need a third node in the load balancer but I'm asleep... It would be wonderful if I had something monitoring probably RAM usage and some key performance indicators that tell me when I should have another node, and even better if it could add a new node to the load balancer alone...

I'm confident that this is possible and even trivial to do with node-http-proxy and distribute, but I'd like to know if this is possible to do with HAproxy and/or Nginx... I know Amazon's elastic load balancing is probably my best bet but I want to do it on my own (I want to spawn instances from rackspace, EC2, Joyent and probably others as it's convenient).

Once again, spawning a node is easy, I'd like to know how to add it to haproxy.cfg or something similar with Nginx without having to reload the whole proxy, and doing that programatically. Bash scripting is my best bet for this but it still does have to reload the whole proxy which is bad because it loses connections...

Answer 1

HAProxy has a Runtime API that allows you to do just that dynamically.

Please read the official documentation: Dynamic Configuration HAProxy Runtime API

Answer 2

What I do: I have a line in my backend section in haproxy.cfg which says:

# new webservers here

And with a sed script I update haproxy.cfg with something like:

  sed -i -e "/new webservers here/a\    server $ip_address $ip_address:12080 check maxconn 28 weight 100"

And then reload haproxy. Works transparently.