Using python to visit a link and print data [closed]

Question

I'm writing a web scraper and trying to get back Drake lyrics. My scraper has to visit one site (main metrolyrics site) and then visit each individual song link, then print out the lyrics.

I'm having trouble visiting the second link. I've searched around on BeautifulSoup and am pretty confused. I'm wondering if you can help.

# this is intended to print all of the drake song lyrics on metrolyrics

from pyquery import PyQuery as pq
from lxml import etree
import requests
from bs4 import BeautifulSoup

# this visits the website
response = requests.get('http://www.metrolyrics.com/drake-lyrics.html')

# this separates the different types of content
doc = pq(response.content)

# this finds the titles in the content
titles = doc('.title')

# this visits each title, then prints each verse
for title in titles:
    # this visits each title
  response_title = requests.get(title)
    # this separates the content
  doc2 = pq(response_title.content)
    # this finds the song lyrics
  verse = doc2('.verse')
    # this prints the song lyrics
  print verse.text

In response_title = requests.get(title), python isn't recognizing that title is a link, which makes sense. How do I get the actual in there, though? Appreciate your help.

Answer 1

Replace

response_title = requests.get(title)

with

response_title = requests.get(title.attrib['href'])

Full working script (with fixed note from the comment below)

#!/usr/bin/python

from pyquery import PyQuery as pq
from lxml import etree
import requests
from bs4 import BeautifulSoup

# this visits the website
response = requests.get('http://www.metrolyrics.com/drake-lyrics.html')

# this separates the different types of content
doc = pq(response.content)

# this finds the titles in the content
titles = doc('.title')

# this visits each title, then prints each verse
for title in titles:
    # this visits each title
  #response_title = requests.get(title)
  response_title = requests.get(title.attrib['href'])

    # this separates the content
  doc2 = pq(response_title.content)
    # this finds the song lyrics
  verse = doc2('.verse')
    # this prints the song lyrics
  print verse.text()