Sending file between systems A->B->C without storing whole file in B

Question

I have 3 separate spring web applications

• A uses spring 4.x
• B uses spring 3.2.0
• C uses spring 4.x

B and C exposes REST controllers for uploading files

• A reads file and uploads it to B
• B sends the request to C without any need to read file content
• and then C does whatever it wants with the file.

So the flow would be A->B->C

My question is - is it possible to setup B in such a way so that B wouldn't store whole file in the memory, but would read incoming stream and forward it to C?

What I managed to do is: A

public void sendFileFromA() throws FileNotFoundException {
    final InputStream fis = new FileInputStream(new File("someFile"));
    final RequestCallback requestCallback = new RequestCallback() {
        @Override
        public void doWithRequest(final ClientHttpRequest request) throws IOException {
            request.getHeaders().add("Content-type", "application/octet-stream");
            IOUtils.copy(fis, request.getBody());
        }
    };
    final RestTemplate restTemplate = new RestTemplate();
    SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
    requestFactory.setBufferRequestBody(false);
    restTemplate.setRequestFactory(requestFactory);

    final HttpMessageConverterExtractor<String> responseExtractor = new HttpMessageConverterExtractor<>(
            String.class, restTemplate.getMessageConverters());
    restTemplate.execute("http://b_url/upload", HttpMethod.POST, requestCallback, responseExtractor);
}

B

@RequestMapping(value = "/upload", method = RequestMethod.POST)
public @ResponseBody String handleFileUpload(HttpServletRequest request) throws IOException {
    final ServletInputStream input = request.getInputStream();

    final RequestCallback requestCallback = new RequestCallback() {
        @Override
        public void doWithRequest(final ClientHttpRequest request) throws IOException {
            request.getHeaders().add("Content-type", "application/octet-stream");
            try (OutputStream body = request.getBody()) {
                IOUtils.copy(input, body);
            }
        }
    };
    final RestTemplate restTemplate = new RestTemplate();
    SimpleClientHttpRequestFactory requestFactory = new SimpleClientHttpRequestFactory();
    requestFactory.setBufferRequestBody(false);
    restTemplate.setRequestFactory(requestFactory);

    final HttpMessageConverterExtractor<String> responseExtractor = new HttpMessageConverterExtractor<>(
            String.class, restTemplate.getMessageConverters());
    restTemplate.execute("http://c_url/upload", HttpMethod.POST, requestCallback, responseExtractor);

    return "success";
}

C

@RequestMapping(value = "/upload", method = RequestMethod.POST)
public @ResponseBody String handleFileUpload(HttpServletRequest request) throws IOException {
    ServletInputStream input = request.getInputStream();

    try (BufferedOutputStream output = new BufferedOutputStream(new FileOutputStream("zibiTest"))) {
        IOUtils.copy(input, output);
    }
    return "success";
}

I can easily copy files over >10GB from A to C using B.

With such a solution we can try to stop A while transferring, B and C should be notified about the error, but sometimes it happens that the error message doesn't reach C - it gets closed with socket timeout exception, any idea why this happens and how to implement it properly?

Is this a valid approach or can it be handled better?

Answer 1

I would try setting smaller socket timeout on C than you have on B. Currently it seems that both have some default value, so if A hangs, both B and C will stop getting data almost the same time. Both start timing out, and maybe it is a race condition, where it depends on the timeout accuracy which one times out first.