Geocoding in Python

Question

I am using the geocoder package in Python to obtain the coordinates for a set of addresses (around 30k). I get the following error:

Status code Unknown from https://maps.googleapis.com/maps/api/geocode/json: ERROR - HTTPSConnectionPool(host='maps.googleapis.com', port=443): Max retries exceeded with url: /maps/api/geocode/json?address=Rancho+Palos+Verdes%2CCA%2CUS&bounds=&components=&region=&language= (Caused by ProxyError('Cannot connect to proxy.', timeout('timed out',)))

The number of times I receive the error reduces if I add the time.sleep(x) function but it significantly increases the time taken to execute the code. Is there a more efficient way to run the code?

Following is a snippet of the code:

for add in clean_address:
    g=geocoder.google(add)
    time.sleep(7)
    if(g.ok==True):
        lat.append(str(g.lat))
        lon.append(str(g.lng))
    if(g.ok==False):
        lat.append("")
        lon.append("")

Answer 1

Google API has a use limitation. However you could even use geocoder library that collects a lot of geocoding services. I suggest to you to use the ArcGis api that hasn't any limitation on usage and is very accurate. The usage is very simple:

g=geocoder.arcgis(add)
lat.append(g.x)
lon.append(g.y)

Answer 2

This is an error originating in the Python requests library used by the geocoder package. The error itself is likely a symptom of exceeding a quota limit configured on your project's enabled API. There are short term quotas (per 100 seconds) and long-term quotas (per day) that you might be running into.

An improvement to automatically retry with exponential backoff may help if the error is due to short-term quota being exceeded. To do this, explicitly create a Session with a custom Retry, and use that session with the geocoder library:

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

...

# Retry for up to 65 seconds
ssn = requests.Session()
adp = HTTPAdapter(max_retries=Retry(total=16, backoff_factor=0.001))
ssn.mount('https://', adp)
ssn.mount('http://', adp)

...

geocoder.google(..., session=ssn)