Resolving DNS in bulk

Question

I need to resolve a large number (hundreds of thousands) of domains to IP addresses in Java. While using InetAddress.getByName() is feasible for small numbers it is far to slow for use in large quantities (probably because it is only sending one request at a time to the DNS server and waiting for the response before moving on to the next one).

Is there a more efficient way (such as sending them to the DNS server in bulk) that would cut down the time required to resolve a large number of domains?

At fmucar's request I'm adding the code used to try a more multi-threaded approach:

Set<String> ips = Collections.synchronizedSet(new HashSet<String>());
int i = 0;
List<Set<String>> sets = new ArrayList<Set<String>>();
for (String host : domains) {
    if (i++ % 5 == 0) {
        sets.add(new HashSet<String>());
    }
    Set<String> ipset = sets.get(sets.size()-1);
    ipset.add(host);
}
for (Set<String> ipset : sets) {
    Thread t = new Thread(new DomainResolver(ips, ipset));
    t.start();
}

At 250 per thread we peaked around 700 results per minute. Which, while better than before (<300) was still not that great when needing to resolve hundreds of thousands. Lowering it to only 5 per thread greatly speeds this up to several thousand per minute. This obviously creates an insane amount of threads though, so presently investigating doing the resolving in C to make use of http://www.chiark.greenend.org.uk/~ian/adns/

Answer 1

According to the RFC for DNS Implementation you can only ask one question at a time as defined below:

4.1.2. Question section format

The question section is used to carry the "question" in most queries, i.e., the parameters that define what is being asked. The section contains QDCOUNT (usually 1) entries, each of the following format:

                                1  1  1  1  1  1
  0  1  2  3  4  5  6  7  8  9  0  1  2  3  4  5
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|                                               |
/                     QNAME                     /
/                                               /
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|                     QTYPE                     |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|                     QCLASS                    |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

where:

QNAME a domain name represented as a sequence of labels, where each label consists of a length octet followed by that number of octets. The domain name terminates with the zero length octet for the null label of the root. Note that this field may be an odd number of octets; no padding is used.

QTYPE a two octet code which specifies the type of the query. The values for this field include all codes valid for a TYPE field, together with some more general codes which can match more than one type of RR.

Mockapetris [Page 28] RFC 1035 Domain Implementation and Specification
November 1987

QCLASS a two octet code that specifies the class of the query. For example, the QCLASS field is IN for the Internet. ....

However you might get custom [ higly unlikely ] resolvers that mainitain their own caches and support bulk transfers as their specification is slightly open ended. I dont know if any exist though. Maybe you can write one :) ... For more information about resolvers look at section 5 of this RFC

The easiest solution would be to use threading as suggested before.

EDIT: The moral of the story I guess is that DNS servers are not designed to accept bulk requests. This makes sense as otherwise it might be easy for attackers to request too much information from a single DNS server