How to make Java honor the DNS Caching Timeout?

Question

We use GSLB for geo-distribution and load-balancing. Each service is assigned a fixed domain name. Through some DNS magic, the domain name is resolved into an IP that's closest to the server with least load. For the load-balancing to work, the application server needs to honor the TTL from DNS response and to resolve the domain name again when cache times out. However, I couldn't figure out a way to do this in Java.

The application is in Java 5, running on Linux (Centos 5).

Answer 1

Per Byron's answer, you can't set networkaddress.cache.ttl or networkaddress.cache.negative.ttl as System Properties by using the -D flag or calling System.setProperty because these are not System properties - they are Security properties.

If you want to use a System property to trigger this behavior (so you can use the -D flag or call System.setProperty), you will want to set the following System property:

-Dsun.net.inetaddr.ttl=0

This system property will enable the desired effect.

But be aware: if you don't use the -D flag when starting the JVM process and elect to call this from code instead:

java.security.Security.setProperty("networkaddress.cache.ttl" , "0")

This code must execute before any other code in the JVM attempts to perform networking operations.

This is important because, for example, if you called Security.setProperty in a .war file and deployed that .war to Tomcat, this wouldn't work: Tomcat uses the Java networking stack to initialize itself much earlier than your .war's code is executed. Because of this 'race condition', it is usually more convenient to use the -D flag when starting the JVM process.

If you don't use -Dsun.net.inetaddr.ttl=0 or call Security.setProperty, you will need to edit $JRE_HOME/lib/security/java.security and set those security properties in that file, e.g.

networkaddress.cache.ttl = 0
networkaddress.cache.negative.ttl = 0

But pay attention to the security warnings in the comments surrounding those properties. Only do this if you are reasonably confident that you are not susceptible to DNS spoofing attacks.

Answer 2

Java has some seriously weird dns caching behavior. Your best bet is to turn off dns caching or set it to some low number like 5 seconds.

networkaddress.cache.ttl (default: -1)
Indicates the caching policy for successful name lookups from the name service. The value is specified as as integer to indicate the number of seconds to cache the successful lookup. A value of -1 indicates "cache forever".

networkaddress.cache.negative.ttl (default: 10)
Indicates the caching policy for un-successful name lookups from the name service. The value is specified as as integer to indicate the number of seconds to cache the failure for un-successful lookups. A value of 0 indicates "never cache". A value of -1 indicates "cache forever".

• http://java.sun.com/j2se/1.4.2/docs/api/java/net/InetAddress.html

Answer 3

This has obviously been fixed in newer releases (SE 6 and 7). I experience a 30 second caching time max when running the following code snippet while watching port 53 activity using tcpdump.

/**
 * http://stackoverflow.com/questions/1256556/any-way-to-make-java-honor-the-dns-caching-timeout-ttl
 *
 * Result: Java 6 distributed with Ubuntu 12.04 and Java 7 u15 downloaded from Oracle have
 * an expiry time for dns lookups of approx. 30 seconds.
 */

import java.util.*;
import java.text.*;
import java.security.*;

import java.net.InetAddress;
import java.net.UnknownHostException;
import java.io.BufferedReader;
import java.io.InputStreamReader;
import java.io.InputStream;
import java.net.URL;
import java.net.URLConnection;

public class Test {
    final static String hostname = "www.google.com";
    public static void main(String[] args) {
        // only required for Java SE 5 and lower:
        //Security.setProperty("networkaddress.cache.ttl", "30");

        System.out.println(Security.getProperty("networkaddress.cache.ttl"));
        System.out.println(System.getProperty("networkaddress.cache.ttl"));
        System.out.println(Security.getProperty("networkaddress.cache.negative.ttl"));
        System.out.println(System.getProperty("networkaddress.cache.negative.ttl"));

        while(true) {
            int i = 0;
            try {
                makeRequest();
                InetAddress inetAddress = InetAddress.getLocalHost();
                System.out.println(new Date());
                inetAddress = InetAddress.getByName(hostname);
                displayStuff(hostname, inetAddress);
            } catch (UnknownHostException e) {
                e.printStackTrace();
            }
            try {
                Thread.sleep(5L*1000L);
            } catch(Exception ex) {}
            i++;
        }
    }

    public static void displayStuff(String whichHost, InetAddress inetAddress) {
        System.out.println("Which Host:" + whichHost);
        System.out.println("Canonical Host Name:" + inetAddress.getCanonicalHostName());
        System.out.println("Host Name:" + inetAddress.getHostName());
        System.out.println("Host Address:" + inetAddress.getHostAddress());
    }

    public static void makeRequest() {
        try {
            URL url = new URL("http://"+hostname+"/");
            URLConnection conn = url.openConnection();
            conn.connect();
            InputStream is = conn.getInputStream();
            InputStreamReader ird = new InputStreamReader(is);
            BufferedReader rd = new BufferedReader(ird);
            String res;
            while((res = rd.readLine()) != null) {
                System.out.println(res);
                break;
            }
            rd.close();
        } catch(Exception ex) {
            ex.printStackTrace();
        }
    }
}

Answer 4

To expand on Byron's answer, I believe you need to edit the file java.security in the %JRE_HOME%\lib\security directory to effect this change.

Here is the relevant section:

#
# The Java-level namelookup cache policy for successful lookups:
#
# any negative value: caching forever
# any positive value: the number of seconds to cache an address for
# zero: do not cache
#
# default value is forever (FOREVER). For security reasons, this
# caching is made forever when a security manager is set. When a security
# manager is not set, the default behavior is to cache for 30 seconds.
#
# NOTE: setting this to anything other than the default value can have
#       serious security implications. Do not set it unless 
#       you are sure you are not exposed to DNS spoofing attack.
#
#networkaddress.cache.ttl=-1 

Documentation on the java.security file here.

Answer 5

To summarize the other answers, in <jre-path>/lib/security/java.security you can set the value of the property networkaddress.cache.ttl to adjust how DNS lookups are cached. Note that this is not a system property but a security property. I was able to set this using:

java.security.Security.setProperty("networkaddress.cache.ttl", "<value>");

This can also be set by the system property -Dsun.net.inetaddr.ttl though this will not override a security property if it is set elsewhere.

I would also like to add that if you are seeing this issue with web services in WebSphere, as I was, setting networkaddress.cache.ttl will not be enough. You need to set the system property disableWSAddressCaching to true. Unlike the time-to-live property, this can be set as a JVM argument or via System.setProperty).

IBM has a pretty detailed post on how WebSphere handles DNS caching here. The relevant piece to the above is:

To disable address caching for Web services, you need to set an additional JVM custom property disableWSAddressCaching to true. Use this property to disable address caching for Web services. If your system typically runs with lots of client threads, and you encounter lock contention on the wsAddrCache cache, you can set this custom property to true, to prevent caching of the Web services data.

Answer 6

According to the official oracle java properties, sun.net.inetaddr.ttl is Sun implementation-specific property, which "may not be supported in future releases". "the preferred way is to use the security property" networkaddress.cache.ttl.

Answer 7

So I decided to look at the java source code because I found official docs a bit confusing. And what I found (for OpenJDK 11) mostly aligns with what others have written. What is important is the order of evaluation of properties.

InetAddressCachePolicy.java (I'm omitting some boilerplate for readability):

String tmpString = Security.getProperty("networkaddress.cache.ttl");
if (tmpString != null) {
   tmp = Integer.valueOf(tmpString);
   return;
}
...
String tmpString = System.getProperty("sun.net.inetaddr.ttl");
if (tmpString != null) {
   tmp = Integer.valueOf(tmpString);
   return;
}
...
if (tmp != null) {
  cachePolicy = tmp < 0 ? FOREVER : tmp;
  propertySet = true;
} else {
  /* No properties defined for positive caching. If there is no
  * security manager then use the default positive cache value.
  */
  if (System.getSecurityManager() == null) {
    cachePolicy = 30;
  }
}

You can clearly see that the security property is evaluated first, system property second and if any of them is set cachePolicy value is set to that number or -1 (FOREVER) if they hold a value that is bellow -1. If nothing is set it defaults to 30 seconds. As it turns out for OpenJDK that is almost always the case because by default java.security does not set that value, only a negative one.

#networkaddress.cache.ttl=-1 <- this line is commented out
networkaddress.cache.negative.ttl=10

BTW if the networkaddress.cache.negative.ttl is not set (removed from the file) the default inside the java class is 0. Documentation is wrong in this regard. This is what tripped me over.