How to serve huge file over streaming rest?

Question

I have 2 Spring Web applications: Application1 and Application2. In Application1, I have an endpoint at "http://application1/getbigcsv" that uses streaming in order to serve a gigantic 150MB CSV file back to the user if they hit that URL.

I dont want users to hit Application1 directly, but hit Application2 instead. If I have the following method in my controller in Application2

@RequestMapping(value = "/large.csv", method = GET, produces = "text/csv")
@ResponseStatus(value = HttpStatus.OK)
public String streamLargeCSV() {
    // Make an HTTP Request to http://application1/getbigcsv
    // Return its response
}

My worry is the above is not doing "streaming" whereas Application1 is doing streaming. Is there some way I can make sure that the application2 will be serving back the same data from application1's rest endpoint in a streaming fashion? Or is the method above actually returning things in a "Streaming" method already because Application1 is serving its endpoint as streaming?

Answer 1

First of all: you can but not with that method signature.

Unfortunately, you have not shown how you produce that CSV file in app1, whether this is truly streaming. Let's assume it is.

You signature will look like this:

@RequestMapping(value = "/large.csv", method = GET, produces = "text/csv")
@ResponseStatus(value = HttpStatus.OK)
public void streamLargeCSV(OutputStream out) {
    // Make an HTTP Request to http://application1/getbigcsv
    // Return its response
}

Now we have to grab the input stream from app1 first. Use Apache HttpClient to get your HttpEntity. This entity has a writeTo(OutputStream) method which will receive your out parameter. It will block until all bytes are consumed/streamed. When you are done, free all HttpClient resources.

Complete code:

@RequestMapping(value = "/large.csv", method = GET, produces = "text/csv")
@ResponseStatus(value = HttpStatus.OK)
public void streamLargeCSV(OutputStream out) {
    // Make an HTTP Request to http://application1/getbigcsv
    CloseableHttpClient httpclient = HttpClients.createDefault();
    HttpGet httpGet = new HttpGet("http://application1/getbigcsv");
    CloseableHttpResponse response = httpclient.execute(httpGet);
    try {
        HttpEntity entity = response.getEntity();
        // Return its response
        entity.writeTo(out);
    } finally {
        response.close();
    }
}

Here is my real world example. Start reading from "Interesting to say what I have achieved in particular with this:"