Large file transfer from HTTP server running Java Jersey Rest API

Question

I am writing a web service using Java JDK 1.7 and Jersey Web service Framework. One of the things I need to provide is a way to allow authenticated clients to download certain large data files ( 1-3 GB). Ideally I would like this to be a pause and resume type downloadable option. I tried the jersey multi-part API and was able to get it to work on my client machine upto 400 MB but beyond that it ran into out-of memory issues. I am also worried that the server might fail when faced with simultaneous download requests. Any thoughts on how this can be done? Is Netty an option? Any pointers on how Netty can be integrated into a existing Jersey based web service? Are there other frame works available to help accomplish this? I do have to use java for the web service. Any pointers will be helpful.

Answer 1

If you are getting stuck on out-of-memory issues, you should check how you are handling the data you are downloading. If you are using Jersey's ClientResponse, make sure you are using getEntityInputStream() and not getEntity(). This way, you can stream the data, write it to file, and toss it aside, rather than letting it build up in the Java heap space.

I can't really speak about your simultaneous download concerns, but if you are using the web services framework, then it should be handled properly.

For both issues, more info on your specific implementation, especially code, will help you get a better response.

Answer 2

The server and the client must both support HTTP chunked encoding which allows one to stream data using HTTP. The code below should work with Jersey 2.11.

For downloading large files, try this on the server:

@GET
@Path("/files/{fileName}")
@Produces(MediaType.APPLICATION_OCTET_STREAM)

public StreamingOutput getFile(@PathParam("fileName") final String fileName)  throws Exception  {
  //create instance of StreamingOutput here
  return streamingOutput;
}

Try this for a client GET request using steams to download a file.

public String getFileReq(File outFile) throws IOException {

  client = ClientBuilder.newClient(new ClientConfig());
  client.property(ClientProperties.REQUEST_ENTITY_PROCESSING, "CHUNKED");
  WebTarget target = client.target(URI)
  OutputStream fileOutputStream = new FileOutputStream(outFile);
  InputStream fileInputStream = target.request().get(InputStream.class);
  writeFile(fileInputStream, fileOutputStream);    

}

public static void writeFile(InputStream fileInputStream, OutputStream outputStream) throws IOException {
try {
  byte[] buffer = new byte[1024];
  int bytesRead;

  while((bytesRead = fileInputStream.read(buffer)) !=-1) {
      outputStream.write(buffer, 0, bytesRead);
  }

  fileInputStream.close();
  outputStream.flush();
 } catch (IOException e) {
   e.printStackTrace();
 } finally {
   outputStream.close();
}