Script to download Google web history

Question

How does one write a script to download one's Google web history?

I know about

https://www.google.com/history/

https://www.google.com/history/lookup?hl=en&authuser=0&max=1326122791634447

feed:https://www.google.com/history/lookup?month=1&day=9&yr=2011&output=rss

but they fail when called programmatically rather than through a browser.

Answer 1

I wrote up a blog post on how to download your entire Google Web History using a script I put together.

It all works directly within your web browser on the client side (i.e. no data is transmitted to a third-party), and you can download it to a CSV file. You can view the source code here:

http://geeklad.com/tools/google-history/google-history.js

My blog post has a bookmarklet you can use to easily launch the script. It works by accessing the same feed, but performs the iteration of reading the entire history 1000 records at a time, converting it into a CSV string, and making the data downloadable at the touch of a button.

I ran it against my own history, and successfully downloaded over 130K records, which came out to around 30MB when exported to CSV.

EDIT: It seems that number of foks that have used my script have run into problems, likely due to some oddities in their history data. Unfortunately, since the script does everything within the browser, I cannot debug it when it encounters histories that break it. If you're a JavaScript developer, use my script, and it appears your history has caused it to break; please feel free to help me fix it and send me any updates to the code.

Answer 2

I tried GeekLad's system, unfortunately two breaking changes have occurred #1 URL has changed ( I modified and hosted my own copy which led to #2 type=rss arguments no longer works.

I only needed the timestamps... so began the best/worst hack I've written in a while.

Step 1 - https://stackoverflow.com/a/3177718/9908 - Using chrome disable ALL security protocols.

Step 2 - https://gist.github.com/devdave/22b578d562a0dc1a8303

Using contentscript.js and manifest.json, make a chrome extension, host ransack.js locally to whatever service you want ( PHP, Ruby, Python, etc ). Goto https://history.google.com/history/ after installing your contentscript extension in developer mode ( unpacked ). It will automatically inject ransack.js + jQuery into the dom, harvest the data, and then move on to the next "Later" link.

Every 60 seconds, Google will force you to re-login randomly so this is not a start and walk away process BUT it does work and if they up the obfustication ante, you can always resort to chaining Ajax calls and send the page back to the backend for post processing. At full tilt, my abomination script collected 1 page a second of data.

On moral grounds I will not help anyone modify this script to get search terms and results as this process is not sanctioned by Google ( though not blocked apparently ) and recommend it only to sufficiently motivated individuals to make it work for them. By my estimates it took me 3-4 hours to get all 9 years of data ( 90K records ) @ 1 page every 900ms or faster.

While this thing is going, DO NOT browse the rest of the web because Chrome is running with no safeguards in place, most of them exist for a reason.