Emulating network disconnects to locally test distributed app partitioning

Question

I have several instances of a distributed application running on the localhost; every instance communicate with others through certain ports, all instances together make an ensemble. (I'm actually talking about ZooKeeper, running on Linux)

Now I want to write unit tests to emulate ensemble partitioning.
E.g. I have 5 instances, and I want to split them into two groups of 3 and 2 so that an instance from one group couldn't communicate to an instance from another group. It would emulate real-world situation when 3 machines are in one datacenter, 2 machines are in another one, and datacenters become partitioned.

The problem is essentially to make a socket work selectively: speak to one socket, but don't speak to another. One solution that comes to mind is to abstract communication layer and inject testing rules into it (in the form of "if I'm an instance from one group I'm not not allowed to speak to an instance from another group -- close socket or ignore data or whatever").

But maybe there exist some tools for this, maybe some testing framework? In general, how do you test such cases in your distributed apps?

P.S. Though question is tagged with "java" (since ZooKeeper is written in Java), it'd be great to hear solutions for any other language, or language-independent ones -- maybe some linux guru-tricks.

Answer 1

Maybe this answer will save a few hours of googling for someone.

Linux has a very good firewall utility, iptables. It allows you to block communication between processes, like this:

iptables -A INPUT -p tcp --sport <source port> --dport <dest port> -j DROP

While not a full-blown unit testing framework by any measure, this helps a bit with manual testing in my case.