Beautifulsoup extract <li> and <ul> tags and write results to CSV

Question

I'm trying to extract the text inside all lines ('li') from the following:

<ul id="tco_detail_data">
        <li>
            <ul class="list-title">
                <li class="first"> </li>
                <li>Year 1</li>
                <li>Year 2</li>
                <li>Year 3</li>
                <li>Year 4</li>
                <li>Year 5</li>
                <li class="last">5 Yr Total</li>
            </ul>
        </li>
        <hr class="loose-dotted" />
        <li class="first">
            <ul class="first">
                <li class="first">Depreciation</li>
                <li>$5,390</li>
                <li>$1,658</li>
                <li>$1,459</li>
                <li>$1,293</li>
                <li>$1,161</li>
                <li class="last">$10,961</li>
            </ul>
        </li>
        <hr class="loose-dotted" />
        <li>
            <ul>
                <li class="first">Taxes &amp; Fees</li>
                <li>$1,424</li>
                <li>$61</li>
                <li>$61</li>
                <li>$61</li>
                <li>$61</li>
                <li class="last">$1,668</li>
            </ul>
        </li>
        <hr class="loose-dotted" />
        <li>
            <ul>
                <li class="first">Financing</li>
                <li>$1,022</li>
                <li>$817</li>
                <li>$603</li>
                <li>$375</li>
                <li>$135</li>
                <li class="last">$2,952</li>
            </ul>

To get to this point I used the following:

import requests
from bs4
import BeautifulSoup
import csv
page = requests.get('https://www.edmunds.com/ford/escape/2017/cost-to-own/')
soup = BeautifulSoup(page.content, 'html.parser')
data = soup.find_all("ul", {"id": "tco_detail_data"})

Now, to extract the all the lines under class="first", I used:

details = soup.find_all("li", {"class":"first"})

However, it with only get the firs parent li tag and the child li tags under it. How can I repeat the process to select each li class"first" section and write results to CSV? I would appreciate any guidance.

Answer 1

Here's a similar approach to the previous answer, which will give you the table from the web page in nested list form (i.e. [[table row], [table row], ...':

data = soup.find_all("ul", {"id": "tco_detail_data"})

# get all list elements
lis = data[0].find_all('li')

# add a helper lambda, just for readability
find_ul = lambda x: x.find_all('ul')
uls = [find_ul(elem) for elem in lis if find_ul(elem) != []]

# use a nested list comprehension to iterate over the <ul> tags
# and extract text from each <li> into sublists
text = [[li.text.encode('utf-8') for li in ul[0].find_all('li')] for ul in uls]

# [
#   ['\xc2\xa0', 'Year 1', 'Year 2', 'Year 3', 'Year 4', 'Year 5', '5 Yr Total'],
#   ['Depreciation', '$4,853', '$1,658', '$1,459', '$1,293', '$1,161', '$10,424'],
#   ['Taxes & Fees', '$2,057', '$21', '$66', '$21', '$66', '$2,231'],
#   ['Financing', '$1,026', '$821', '$605', '$376', '$136', '$2,964'],
#   ['Fuel', '$1,606', '$1,654', '$1,704', '$1,755', '$1,808', '$8,527'],
#   ['Insurance', '$764', '$791', '$818', '$847', '$877', '$4,097'],
#   ['Maintenance', '$230', '$601', '$385', '$1,653', '$1,504', '$4,373'],
#   ['Repairs', '$0', '$0', '$109', '$257', '$374', '$740'],
#   ['Tax Credit', '$0', '', '', '', '', '$0'],
#   ['True Cost to Own \xc2\xae', '$10,536', '$5,546', '$5,146', '$6,202', '$5,926', '$33,356']
# ]

# write "text" list to csv
with open('ford_escape_2017.csv', 'w') as f:
    writer = csv.writer(f)
    writer.writerows(text)