Trouble getting PythonAnywhere to scrape web for me

Question

I've been experimenting around on PythonAnywhere trying to get some python to work on a webserver. I initially switched from Arvixe because they were running 2.4 and PythonAnywhere's name was just too appealing.

My application consists of two files: phones.py and phonesearch.py. Together they are supposed to scrape craigslist for phone prices.

I test locally in 2.7 and it runs just fine, generating an html page (celly.html) with a table and all of the prices. When I upload it, it generates the html just fine, but refuses to to add anything to my prices list ([intprices]).

My suspicions: (a) because it works fine locally, PythonAnywhere isn't allowing it to communicate with craigslist; or (b) because I'm doing this like a caveman and not using a microframework, PythonAnywhere is denying me; or (c) I am blind to my errors and I have missed something obvious.

My python scripts are located in /home/tseymour/mysite and the html is generated at same/mysite/static/celly.html. The file is served up at http://tseymour.pythonanywhere.com/static/celly.html

You'll notice that all of my cells are filled with "N/A" which means it raised an IndexError in the try:" in SearchPhone.py. This means my lists are being filled!

But WHY is that?! I believe it is because I am PythonAnywhere n00b.

Please advise.

SearchPhone.py

from BeautifulSoup import BeautifulSoup
import urllib
import re

def SearchPhone(phone):

    y = "http://losangeles.craigslist.org/search/moa?query=" + phone + "+-%22buy%22+-%22fix%22+-%22unlock%22+-%22broken%22+-%22cracked%22+-%22parts%22&srchType=T&minAsk=&maxAsk="

    site = urllib.urlopen(y)
    html = site.read()
    site.close()
    soup = BeautifulSoup(html)


    prices = soup.findAll("span", {"class":"itempp"})
    prices = [str(j).strip('<span class="itempp"> $</span>') for j in prices]

    for k in prices[:]:
        if k == '': #left price blank
            prices.remove(k)
        elif int(k) <= 75: #less than $50: probably a service (or not true)
            prices.remove(k)
        elif int(k) >= 999: #probably not true
            prices.remove(k)

    #Find Average Price
    intprices = []
    newprices = prices[:]
    total = 0
    for k in newprices:
        total += int(k)
        intprices.append(int(k))

    intprices = sorted(intprices)

    try:
        del intprices[0]
        del intprices[-1]


        avg = total/len(newprices)
        low = intprices[0]
        high = intprices[-1]

        if len(intprices) % 2 == 1:
            median = intprices[(len(intprices)+1)/2-1]
        else:
            lower = intprices[len(intprices)/2-1]
            upper = intprices[len(intprices)/2]
            median = (float(lower + upper)) / 2



        namestr = str(phone)
        medstr = "Median: $" + str(median)
        avgstr = "Average: $" + str(avg)
        lowstr = "Low: $" + str(intprices[0])
        highstr = "High: $" + str(intprices[-1])
        samplestr = "# of samples: " + str(len(intprices))
        linestr = "-------------------------------"

    except IndexError:
        namestr = str(phone)
        medstr = "N/A"
        avgstr = "N/A"
        lowstr = "N/A"
        highstr = "N/A"
        samplestr = "N/A"
        linestr = "-------------------------------"

    return (namestr, medstr, avgstr, lowstr, highstr, samplestr, linestr)

phones.py

from SearchPhone import SearchPhone

phones = ["Iphone 4", "Iphone 5","Galaxy s3", "Galaxy s2", "LG Lucid", "LG Esteem", "HTC One S", "Droid 4",
          "Droid RAZR MAXX", "HTC EVO", "Galaxy Nexus", "LG Optimus 2", "LG Ignite",
          "Galaxy Note", "HTC Amaze", "HTC Rezound", "HTC Vivid", "HTC Rhyme", "Motorola Photon",
          "Motorola Milestone", "myTouch slide", "HTC Status", "Droid 3", "HTC Evo 3d", "HTC Wildfire",
          "LG Optimus 3d", "HTC ThunderBolt", "Incredible 2", "Kyocera Echo", "Galaxy S 4g",
          "HTC Inspire", "LG Optimus 2x", "Samsung Gem", "HTC Evo Shift", "Nexus S", "LG Axis", "Droid 2",
          "G2", "Droid x", "Droid Incredible"
          ]

f = open('/home/tseymour/mysite/static/celly.html','w')


f.write("""<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>Celly Blue Book</title>
</head>

<body>
</body>
</html>
""")

#table
f.write('<table width="100%" border="1">')
for x in phones:
    print "SEarchphone0"
    y = SearchPhone(x)
    print "SEarchphone"
    f.write( "\t<tr>")
    f.write( "\t\t<td>" + str(y[0]) + "</td>")
    f.write( "\t\t<td>" + str(y[1]) + "</td>")
    f.write( "\t\t<td>" + str(y[2]) + "</td>")
    f.write( "\t\t<td>" + str(y[3]) + "</td>")
    f.write( "\t\t<td>" + str(y[4]) + "</td>")
    f.write( "\t</tr>")

f.write('</table>')

f.close()

Also, I did upload beautifulsoup just incase

Answer 1

PythonAnywhere dev here. You don't say if you're using a free or a paid PythonAnywhere account, but if it's a free one then I think you're running into our whitelist. For free accounts, we only allow access to a particular set of websites -- this is because people were using us to do Bad Things.

We put sites on the whitelist so that free accounts can use them if they have an official publicly-accessible API, and unfortunately Craigslist doesn't have one -- quite the opposite, unfortunately.

If you sign up for a paid account then you'll probably be able to do what you want, but if the article I just linked to is correct then you might want to make sure you have good lawyers...