Will using DNS failover work as a Multi-DC failover strategy?

Question

If I have a multi-DC cluster, DC1 and DC2, where DC2 is only used for failover. And in the driver on the client side, I define the contact points using the domain names (foo1.net, foo2.net, and foo3.net). I have foo* pointing to DC1 and if I ever detect any error with DC1, I will make the DNS route foo* to point to DC2.

This approach seems to work on paper, but will it actually work? Any issues with this approach?

Answer 1

In the case of the DataStax Java Driver 3.x this will not work since DNS is only evaluated at the beginning of Cluster instantiation.

The contact points provided are resolved using DNS via InetAddress.getAllByName in Cluster.Builder.addContactPoint:

public Builder addContactPoint(String address) {
    // We explicitly check for nulls because InetAdress.getByName() will happily
    // accept it and use localhost (while a null here almost likely mean a user error,
    // not "connect to localhost")
    if (address == null)
        throw new NullPointerException();

    try {
        addContactPoints(InetAddress.getAllByName(address));
        return this;
    } catch (UnknownHostException e) {
        throw new IllegalArgumentException("Failed to add contact point: " + address, e);
    }
}

If DNS is changed during the lifecycle of the Cluster, the driver will not be aware of this unless you construct a new Cluster.Builder instance and create a new Cluster from it.

I prefer a design that pushes Data Center failover outside the scope of your application and into a higher level of your architecture. Instead of making your client application responsible for failing over, you should run instances of your clients colocated in each C* data center. Your application load balancer/router/DNS could direct traffic to instances of your application in other data centers when data centers become unavailable.