How to quickly check if domain exists? [duplicate]

Question

I have a large list of domains and I need to check if domains are available now. I do it like this:

import requests
list_domain = ['google.com', 'facebook.com']
for domain in list_domain:
    result = requests.get(f'http://{domain}', timeout=10)
    if result.status_code == 200:
        print(f'Domain {domain} [+++]')
    else:
        print(f'Domain {domain} [---]')

But the check is too slow. Is there a way to make it faster? Maybe someone knows an alternative method for checking domains for existence?

Answer 1

If you want to check which domains are available, the more correct approach would be to catch the ConnectionError from the requests module, because even if you get a response code that is not 200, the fact that there is a response means that there is a server associated with that domain. Hence, the domain is taken.

This is not full proof in terms of checking for domain availability, because a domain might be taken, but may not have appropriate A record associated with it, or the server may just be down for the time being.

The code below is asynchronous as well.

from concurrent.futures import ThreadPoolExecutor
import requests
from requests.exceptions import ConnectionError

def validate_existence(domain):
    try:
        response = requests.get(f'http://{domain}', timeout=10)
    except ConnectionError:
        print(f'Domain {domain} [---]')
    else:
        print(f'Domain {domain} [+++]')


list_domain = ['google.com', 'facebook.com', 'nonexistent_domain.test']

with ThreadPoolExecutor() as executor:
    executor.map(validate_existence, list_domain)

Answer 2

You can use the socket library to determine if a domain has a DNS entry:

>>> import socket
>>> 
>>> addr = socket.gethostbyname('google.com')
>>> addr
'74.125.193.100'
>>> socket.gethostbyname('googl42652267e.com')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
socket.gaierror: [Errno -2] Name or service not known
>>>