Iterable object and Django StreamingHttpResponse

Question

I want to connect to internal http services with django and I need to buffer the output the http response of those services because some contents are very large.

I am using python 3.6, django 2.0, http.client and the following code:

class HTTPStreamIterAndClose():
    def __init__(self, conn, res, buffsize):
        self.conn = conn
        self.res = res
        self.buffsize = buffsize
        self.length = 1

        bytes_length = int(res.getheader('Content-Length'))

        if buffsize < bytes_length:
            self.length = math.ceil(bytes_length/buffsize)

    def __iter__(self):
        return self

    def __next__(self):
        buff = self.res.read(self.buffsize)

        if buff is b'':
            self.res.close()
            self.conn.close()

            raise StopIteration
        else:

            return buff

    def __len__(self):
        return self.length


def passthru_http_service(request, server, timeout, path):
    serv = HTTPService(server, timeout)
    res = serv.request(path)

    response = StreamingHttpResponse(
        HTTPStreamIterAndClose(serv.connection, res, 200),
        content_type='application/json'
    )
    response['Content-Length'] = res.getheader('Content-Length')

    return response

And the reponse is empty, I test the iterator with:

b''.join(HTTPStreamIterAndClose(serv.connection, res, 200)

And everything works fine, I don't know why is not working.

Answer 1

https://andrewbrookins.com/django/how-does-djangos-streaminghttpresponse-work-exactly/

First, some conditions must be true:

• The client must be speaking HTTP/1.1 or newer
• The request method wasn’t a HEAD
• The response does not include a Content-Length header
• The response status wasn’t 204 or 304

If these conditions are true, then Gunicorn will add a Transfer-Encoding: chunked header to the response, signaling to the client that the response will stream in chunks.

In fact, Gunicorn will respond with Transfer-Encoding: chunked even if you used an HttpResponse, if those conditions are true!

To really stream a response, that is, to send it to the client in pieces, the conditions must be true, and your response needs to be an iterable with multiple items.

Basically, you need to decide: streaming or Content-Length.

If you want resumable downloads use Range.