How to scrape google maps for all data using python

Question

I am trying to scrape the title, phone number, website, address, rating, number of reviews of a place from google maps using python. For example, the restaurant Pike's Landing (see google maps URL below) needs all the information. I want to pull those in python.

URL: https://www.google.com/maps?cid=15423079754231040967&hl=en

I can see HTML code when I inspect but when I have used beautiful soup for scraping all codes are converted. From stack overflow, I have found a solution for the only number of review as following code,

import re
import requests
from ast import literal_eval

urls = [
'https://www.google.com/maps?cid=15423079754231040967&hl=en',
'https://www.google.com/maps?cid=16168151796978303235&hl=en']

for url in urls:
    for g in re.findall(r'\[\\"http.*?\d+ reviews?.*?]', requests.get(url).text):
        data = literal_eval(g.replace('null', 'None').replace('\\"', '"'))
        print(bytes(data[0], 'utf-8').decode('unicode_escape'))
        print(data[1])

But I need all the data. I can use Google Maps API to actual data but getting phone number, rating, review is not free now. So that I want to escape data from the frontend.

Please help me.

Answer 1

I asked the same question a long time ago on reddit. I ended up solving it myself, have a look at this NOTE - this was strictly written to extract details for my use case but you can get a gist of what's going on here.

from selenium import webdriver

options = webdriver.ChromeOptions()

options.add_argument('headless')



browser = webdriver.Chrome(options=options)



url = "https://www.google.com/maps/place/Papa+John's+Pizza/@40.7936551,-74.0124687,17z/data=!3m1!4b1!4m5!3m4!1s0x89c2580eaa74451b:0x15d743e4f841e5ed!8m2!3d40.7936551!4d-74.0124687"

# url = "https://www.google.com/maps/place/Lucky+Dhaba/@30.653792,76.8165233,17z/data=!3m1!4b1!4m5!3m4!1s0x390feb3e3de1a031:0x862036ab85567f75!8m2!3d30.653792!4d76.818712"



browser.get(url)



# review titles / username / Person who reviews

review_titles = browser.find_elements_by_class_name("section-review-title")

print([a.text for a in review_titles])

# review text / what did they think

review_text = browser.find_elements_by_class_name("section-review-review-content")

print([a.text for a in review_text])

# get the number of stars

stars = browser.find_elements_by_class_name("section-review-stars")

first_review_stars = stars[0]

active_stars = first_review_stars.find_elements_by_class_name("section-review-star-active")

print(f"the stars the first review got was {len(active_stars)}")

Answer 2

If you inspect the page source code you will find the window.APP_INITIALIZATION_STATE block, which contains JSON with all of the place data. You would just have to parse it.

You can also use a third party solution like SerpApi. It's a paid API with a free trial.

from serpapi import GoogleSearch

params = {
  "engine": "google_maps",
  "type": "place",
  "q": "Pike's Landing",
  "ll": "@40.7455096,-74.0083012,14z",
  "data": "!3m1!4b1!4m5!3m4!1s0x0:0xd609c9524d75cbc7!8m2!3d64.8299557!4d-147.8488774"
  "api_key": "API_KEY",
}

search = GoogleSearch(params)
results = search.get_dict()

Example output:

"place_results": {
  "title": "Pike's Landing",
  "data_id": "0x51325b1733fa71bf:0xd609c9524d75cbc7",
  "reviews_link": "https://serpapi.com/search.json?engine=google_maps_reviews&hl=en&place_id=0x51325b1733fa71bf:0xd609c9524d75cbc7",
  "gps_coordinates": {
    "latitude": 64.8299557,
    "longitude": -147.8488774
  },
  "thumbnail": "https://lh5.googleusercontent.com/p/AF1QipNtwheOCQ97QFrUNIwKYUoAPiV81rpiW5cIiQco=w152-h86-k-no",
  "rating": 3.9,
  "reviews": 825,
  "price": "$$",
  "type": [
    "American restaurant"
  ],
  "description": "Burgers, seafood, steak & river views. Pub fare alongside steak & seafood, served in a dining room with river views & a waterfront patio.",
  "service_options": {...},
  "extensions": [...],
  "address": "4438 Airport Way, Fairbanks, AK 99709",
  "website": "https://www.pikeslodge.com/pikes-landing",
  "phone": "(907) 479-6500",
  "hours": [...],
  "images": [...],
  "user_reviews": {
    "summary": [...],
    "most_relevant": [
      {
        "username": "Vasisht Raghavendra",
        "rating": 5,
        "description": "Restaurant with a view and good food. The Biggest Berry Blast cocktail is a highly recommended one. For food we tried the Alaska Crab with Mushrooms and the Bisque. Both were excellent. For dessert we tried the Bread Pudding - probably not the best we have had. The ambiance is very nice and the servers are friendly.",
        "images": [...],
        "date": "2 months ago"
      },
      ...
    ]
  },
  ...
}

You can check out the documentation for more details.

Disclaimer: I work at SerpApi.