Simple forvalues loop in python?

Question

is there a simple way in Python to loop over a simple list of numbers? I want to scrape some data from different URLs that only differ in 3 numbers?

I'm quite new to python and couldn't figure out an easy way to do it. Thanks a lot! Here's my code:

import csv
from urllib.request import urlopen
from bs4 import BeautifulSoup

html = urlopen("http://www.example.com/3322")
bsObj = BeautifulSoup(html)
table = bsObj.findAll("table",{"class":"MainContent"})[0]
rows=table.findAll("td")

csvFile = open("/Users/Max/Desktop/file1.csv", 'wt')
writer = csv.writer(csvFile)
try:
    for row in rows:
        csvRow=[]
        for cell in row.findAll(['tr', 'td']):
            csvRow.append(cell.get_text())
            writer.writerow(csvRow)
finally:
    csvFile.close()

In Stata this would be like:

 foreach i of 13 34 55 67{
     html = urlopen("http://www.example.com/`i'")
      ....
      }

Thanks a lot! Max

Answer 1

I've broken your original code into functions simply to make clearer what I think is the answer to your question: use a simple loop, and .format() to construct urls and filenames.

import csv
from urllib.request import urlopen
from bs4 import BeautifulSoup

def scrape_url(url):
    html = urlopen(url)
    bsObj = BeautifulSoup(html)
    table = bsObj.findAll("table",{"class":"MainContent"})[0]
    rows=table.findAll("td")
    return rows

def write_csv_data(path, rows):
    csvFile = open(path, 'wt')
    writer = csv.writer(csvFile)
    try:
        for row in rows:
            csvRow=[]
            for cell in row.findAll(['tr', 'td']):
                csvRow.append(cell.get_text())
                writer.writerow(csvRow)
    finally:
        csvFile.close()

for i in (13, 34, 55, 67):
    url = "http://www.example.com:3322/{}".format(i)
    csv_path = "/Users/MaximilianMandl/Desktop/file-{}.csv".format(i)

    rows = scrape_url(url)
    write_csv_data(csv_path, rows)

Answer 2

i would use set.intersection() for that:

mylist=[1,16,8,32,7,5]
fieldmatch=[5,7,16]

intersection = list(set(mylist).intersection(fieldmatch))