WSGI file streaming with a generator

Question

I have the following code:

def application(env, start_response):
    path = process(env)
    fh = open(path,'r')
    start_response('200 OK', [('Content-Type','application/octet-stream')])
    return fbuffer(fh,10000)


def fbuffer(f, chunk_size):
    '''Generator to buffer file chunks'''  
    while True:
        chunk = f.read(chunk_size)      
        if not chunk: break
        yield chunk

I'm not sure that it's right but the scraps of information I've found on the internet have led me to think it ought to work. Basically I want to stream a file out in chunks, and to do that I'm passing a generator back from my application function. However this only prints out the headers and doesn't actually send back any data, can anyone tell me why this is?

Alternatively, if this is completely wrong, what's the best way to do this? I can't buffer the whole file in memory as the files I'll be working with are potentially gigabytes large.

Tertiary question: What's the best way to close the file once I'm done outputting it? In the code I posted I can't see anyway to actually close the file.

(I'm running python 3.2.3 with uWSGI 1.2.4)

Answer 1

Without some care, uwsgi is careful not to allow errors to leak, but a if you run your application in a stricter implementation, say the one provided with python as wsgiref.simple_server, you can more easily see the problem.

Serving <function application at 0xb65848> http://0.0.0.0:8000
Traceback (most recent call last):
  File "/usr/lib64/python3.2/wsgiref/handlers.py", line 138, in run
    self.finish_response()
  File "/usr/lib64/python3.2/wsgiref/handlers.py", line 179, in finish_response
    self.write(data)
  File "/usr/lib64/python3.2/wsgiref/handlers.py", line 264, in write
    "write() argument must be a bytes instance"
AssertionError: write() argument must be a bytes instance
localhost.localdomain - - [04/Aug/2012 16:27:08] "GET / HTTP/1.1" 500 59

The problem is that wsgi requires that data transmitted to and from the HTTP gateway must be served as bytes, but when you use open(path, 'r'), python 3 conveniently converts the data read to unicode, what in python 3 is str, using the default encoding.

changing

fh = open(path, 'r')

to

fh = open(path, 'rb')
#                 ^

fixes it.