Extract table from web

Question

need to extract data from <a href="#">Data</a> from url below. Any clue how to extract this table into DataFrames?

from bs4 import BeautifulSoup
import requests

url = 'https://docs.google.com/spreadsheets/d/1dgOdlUEq6_V55OHZCxz5BG_0uoghJTeA6f83br5peNs/pub?range=A1:D70&gid=1&output=html#'

r = requests.get(url)
html_doc = r.text
soup = BeautifulSoup(html_doc, features='html.parser')

#print(soup.prettify())
print(soup.title)

Answer 1

It might be easier to start of with a multi-dimensional list, then port it to a DataFrame, that way we aren't assuming sizes. The "Data" hyperlink references the div id=0, so we select all elements in that, then parse each column in each row into a an list array (in which I call elements) that gets appended to a full list array (which I call fullelements) and reset for each new row.

from bs4 import BeautifulSoup
import pandas as pd
import requests

url = 'https://docs.google.com/spreadsheets/d/1dgOdlUEq6_V55OHZCxz5BG_0uoghJTeA6f83br5peNs/pub?range=A1:D70&gid=1&output=html#'

r = requests.get(url)
html_doc = r.text
soup = BeautifulSoup(html_doc, features='html.parser')

#print(soup.prettify())
print(soup.title.text)
datadiv=soup.find("div", {"id": "0"})
elementsfull =[]
row=0
for tr in datadiv.findAll("tr"):
    elements=[]
    column=0
    for td in tr.findAll("td"):
        if(td.text!=''):
            elements.append(td.text)
            column+=1
            #print('column: ', column)   

    elementsfull.append(elements)        
    #print('row: ', row)        
    row+=1

mydf = pd.DataFrame(data=elementsfull)
print(mydf)

I tested this code and checked it against the table, so I guarantee it works.