Python and urllib

Question

I'm trying to download a zip file ("tl_2008_01001_edges.zip") from an ftp census site using urllib. What form is the zip file in when I get it and how do I save it?

I'm fairly new to Python and don't understand how urllib works.

This is my attempt:

import urllib, sys

zip_file = urllib.urlretrieve("ftp://ftp2.census.gov/geo/tiger/TIGER2008/01_ALABAMA/Autauga_County/", "tl_2008_01001_edges.zip")

If I know the list of ftp folders (or counties in this case), can I run through the ftp site list using the glob function?

Thanks.

Answer 1

Use urllib2.urlopen() for the zip file data and directory listing.

To process zip files with the zipfile module, you can write them to a disk file which is then passed to the zipfile.ZipFile constructor. Retrieving the data is straightforward using read() on the file-like object returned by urllib2.urlopen().

Fetching directories:

>>> files = urllib2.urlopen('ftp://ftp2.census.gov/geo/tiger/TIGER2008/01_ALABAMA/').read().splitlines()
>>> for l in files[:4]: print l
... 
drwxrwsr-x    2 0        4009         4096 Nov 26  2008 01001_Autauga_County
drwxrwsr-x    2 0        4009         4096 Nov 26  2008 01003_Baldwin_County
drwxrwsr-x    2 0        4009         4096 Nov 26  2008 01005_Barbour_County
drwxrwsr-x    2 0        4009         4096 Nov 26  2008 01007_Bibb_County
>>> 

Or, splitting for directory names:

>>> for l in files[:4]: print l.split()[-1]
... 
01001_Autauga_County
01003_Baldwin_County
01005_Barbour_County
01007_Bibb_County