Django - Stream request from external site as received

Question

How can Django be used to fetch data from an external API, triggered by a user request, and stream it directly back in the request cycle without (or with progressive/minimal) memory usage?

Background

As a short-term solution to connect with externally hosted micro-services, there is a need to limit user accessibility (based off the Django application's authentication system) to a non-authenticated API. Previous developers exposed these external IPs in Javascript and we need a solution to get them out of the public eye.

Requirements

• We are not bound to using the requests library and are open to using any others if it can help speed up the response time.
• Responses from the external API may be somewhat large (5-10MB) and being able to shorten the request cycle (User request via Ajax > Django > External API > Django > User) is crucial.

Is this possible? If so, can you suggest a method?

from django.shortcuts import Http404, HttpResponse
import requests

def api_gateway_portal(request, path=''):
    # Determine whether to grant access

    # If so, fetch and return data
    r = requests.get('http://some.ip.address/%s?api_key=12345678901234567890' % (path,))

    # Return as JSON 
    response = HttpResponse(r.content, content_type='application/json')
    response['Content-Length'] = len(r.content)

    return response

Please note - I am fully aware this is a poor long-term solution, but is necessary short-term for demo purposes until a new external authentication system is completed.

Answer 1

import requests

from django.http import StreamingHttpResponse

def api_gateway_portal(request, path=''):
    url = 'http://some.ip.address/%s?api_key=12345678901234567890' % (path,)
    r = requests.get(url, stream=True)

    response = StreamingHttpResponse(
        (chunk for chunk in r.iter_content(512 * 1024)),
        content_type='application/json')

    return response

Documentation:

• Body content workflow (stream=True explained)
• StreamingHttpResponse
• iter_content()