Spring restTemplate execute( ) POST large files and obtain a response

Question

This took me quite a while to work out so I wanted to share it. Most information came from SO and I wanted to consolidate into this one place.

My requirements are to upload files using a RESTFul POST. Due to possibly large files I wanted to stream the files. I obviously want to be able to read the response.

I planned to use Jersey as the REST Server and Spring's RestTemplate as the client (and for testing).

The problem I faced was streaming POSTs and receiving a response. How can I do that? (Rhetorical question - I answer this!)

Answer 1

It's unnecessary to go through all these hoops with a RequestCallback. Simply use a PathResource.

PathResource pathResource = new PathResource(theTestFilePath);
ResponseEntity<String> response = restTemplate.exchange(url, HttpMethod.POST, new HttpEntity<>(pathResource), String.class);

Spring will use a ResourceHttpMessageConverter to serialize the file identified by the given Path to the request body. Internally, the Spring 4.x implementation uses a buffer size of 4096 bytes (which is also what IOUtils#copy(..) uses).

Obviously, you can provide the response type you want. The example above expects the response body as a String. With a ResponseEntity, you can access all the response headers with

HttpHeaders responseHeaders = response.getHeaders();

Answer 2

I am using SpringBoot 1.2.4.RELEASE with Jersey being pulled in by:

compile("org.springframework.boot:spring-boot-starter-jersey")

I created the project with the brilliant Spring Starter Project (Spring Tool Suite > New or you can do through a website I believe and no doubt IntelliJ has this capability also). And chose 'Jersey (JAX-RS)' option. In the gradle build.gradle I also added the dependency:

compile('commons-io:commons-io:2.4')

I wrote this server side code.

import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.URI;
import java.net.URISyntaxException;

import javax.ws.rs.Consumes;
import javax.ws.rs.GET;
import javax.ws.rs.POST;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.QueryParam;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.RestController;

import org.me.fileStore.service.FileStoreService;

@RestController
@Path("/filestore")
public class FileStoreRestService {
    private static Logger logger = LoggerFactory.getLogger(FileStoreRestService.class);

    @Autowired
    private FileStoreService fileStoreService;


    @POST
    @Path("upload")
    @Consumes(MediaType.APPLICATION_OCTET_STREAM)
    @Produces(MediaType.APPLICATION_JSON)
    public Response Upload(InputStream stream) throws IOException, URISyntaxException { //
        String location = fileStoreService.upload(stream);  // relative path
        URI loc = new URI(location);
        Response response = Response.created(loc).build();
        System.out.println("POST - response: " + response + ", :" + response.getHeaders());
        return response;
    }

Where i had most troubles was in getting a Response with a location.

Firstly I had to handle streaming large files. I followed https://stackoverflow.com/a/15785322/1019307 as you can see in the test below. I was NOT obtaining a Response no matter what I tried with the HttpMessageConverterExtractor as per that post:

final HttpMessageConverterExtractor<String> responseExtractor =
new HttpMessageConverterExtractor<String>(String.class, restTemplate.getMessageConverters());

After finding https://stackoverflow.com/a/6006147/1019307 I wrote:

private static class ResponseFromHeadersExtractor implements ResponseExtractor<ClientHttpResponse> {

    @Override
    public ClientHttpResponse extractData(ClientHttpResponse response) {
        System.out.println("StringFromHeadersExtractor - response headers: " + response.getHeaders());
        return response;
    }
}

This gave me this test:

import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.io.InputStream;
import java.net.URI;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.HashMap;
import java.util.Map;

import org.apache.commons.io.IOUtils;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.hamcrest.MatcherAssert;
import org.hamcrest.Matchers;
import org.junit.Assert;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.boot.test.IntegrationTest;
import org.springframework.boot.test.SpringApplicationConfiguration;
import org.springframework.boot.test.TestRestTemplate;
import org.springframework.http.HttpMethod;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.http.client.ClientHttpRequest;
import org.springframework.http.client.ClientHttpResponse;
import org.springframework.http.client.SimpleClientHttpRequestFactory;
import org.springframework.test.context.junit4.SpringJUnit4ClassRunner;
import org.springframework.test.context.web.WebAppConfiguration;
import org.springframework.web.client.RequestCallback;
import org.springframework.web.client.ResponseExtractor;
import org.springframework.web.client.RestTemplate;


@RunWith(SpringJUnit4ClassRunner.class)
@SpringApplicationConfiguration(classes = FileStoreApplication.class)
@WebAppConfiguration
@IntegrationTest("server.port:9000")
public class FileStoreRestServiceTest {
    private static Logger logger = LoggerFactory.getLogger(FileStoreRestServiceTest.class);
    protected final Log logger2 = LogFactory.getLog(getClass());

    String base = "http://localhost:9000/filestore";
    private RestTemplate restTemplate = new TestRestTemplate();

@Test
public void testMyMethodExecute() throws IOException {
    String content = "This is file contents\nWith another line.\n";
    Path theTestFilePath = TestingUtils.getTempPath(content);
    InputStream inputStream = Files.newInputStream(theTestFilePath);

    String url = base + "/upload";
    final RequestCallback requestCallback = new RequestCallback() {
        @Override
        public void doWithRequest(final ClientHttpRequest request) throws IOException {
            request.getHeaders().setContentType(MediaType.APPLICATION_OCTET_STREAM);
            IOUtils.copy(inputStream, request.getBody());
        }
    };
    final RestTemplate restTemplate = new RestTemplate();
    SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
    requestFactory.setBufferRequestBody(false);
    restTemplate.setRequestFactory(requestFactory);
    ClientHttpResponse response = restTemplate.execute(url, HttpMethod.POST, requestCallback,
            new ResponseFromHeadersExtractor());
    URI location = response.getHeaders().getLocation();
    System.out.println("Location: " + location);
    Assert.assertNotNull(location);
    Assert.assertNotEquals(0, location.getPath().length());

}

private static class ResponseFromHeadersExtractor implements ResponseExtractor<ClientHttpResponse> {

    @Override
    public ClientHttpResponse extractData(ClientHttpResponse response) {
        System.out.println("StringFromHeadersExtractor - response headers: " + response.getHeaders());
        return response;
    }
}

I need to refactor much in that test out into some services.