How to scrape AJAX website?

Question

In the past, I've used the urllib2 library to get source codes from websites. However, I've noticed that for a recent website I've been trying to play with, I can't find the information I need in the source code.

http://www.wgci.com/playlist is the site that I've been looking at, and I want to get the most recently played song and the playlist of recent songs. I essentially want to copy and paste the visible, displayed text on a website and put it in a string. Alternatively, being able to access the element that holds these values in plaintext and get them using urllib2 normally would be nice. Is there anyway to do either of these things?

Thanks kindly.

Answer 1

The website you want to scrap is using ajax calls to populate it's pages with data. You have 2 ways to scrapping data from it:

• Use a headless browser that supports javascript (ZombieJS for instance), and scrap the generated output, but that's complicated and overkill
• Understand how their API work, and call that directly, which is way simpler.

Use Chrome dev tools (network tab) to see the calls while browsing their website.

For example, the list of last played songs for a given stream is available in JSON at http://www.wgci.com/services/now_playing.html?streamId=841&limit=12