Jersey service file upload causes OutOfMemoryError

Question

I'm developing form submission service with Jersey 2.0. The form includes several text fields and one file field. I need to extract file, file name, file media type and file content type and save them in object store.

@Path("upload")
@Consumes({MediaType.MULTIPART_FORM_DATA})
@Produces({MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON})
public class UploadService {
    @POST
    public BlobDo uploadFile(FormDataMultiPart uploadedBody) {
        String accountSid = uploadedBody.getField("account-sid").getValue();
        String apiToken = uploadedBody.getField("api-token").getValue();
        String checksum = uploadedBody.getField("checksum").getValue();

        FormDataBodyPart bodyPart = uploadedBody.getField("file");
        MySwiftObject obj = new MySwiftObject(bodyPart.getValueAs(InputStream.class));
        obj.setName(bodyPart.getContentDisposition().getFileName());
        obj.setContentType(bodyPart.getMediaType().toString());
        obj.setContentDisposition(bodyPart.getContentDisposition().toString());
   ...
}

pom.xml

<jersey.version>2.17</jersey.version>

<dependency>
    <groupId>org.glassfish.jersey.containers</groupId>
    <artifactId>jersey-container-servlet-core</artifactId>
</dependency>
<dependency>
    <groupId>org.glassfish.jersey.media</groupId>
    <artifactId>jersey-media-json-jackson</artifactId>
</dependency>
<dependency>
    <groupId>org.glassfish.jersey.test-framework.providers</groupId>
    <artifactId>jersey-test-framework-provider-inmemory</artifactId>
</dependency>
<dependency>
    <groupId>org.glassfish.jersey.media</groupId>
    <artifactId>jersey-media-multipart</artifactId>
</dependency>
<dependency>
    <groupId>org.glassfish.jersey.core</groupId>
    <artifactId>jersey-server</artifactId>
</dependency>
<dependency>
    <groupId>org.glassfish.jersey.ext</groupId>
    <artifactId>jersey-bean-validation</artifactId>
</dependency>

form submission request

POST /nbs/v2/upload HTTP/1.1
Host: 127.0.0.1:8080
Cache-Control: no-cache
Postman-Token: a4c1d4e9-5f71-2321-3870-e9cac0524f8d
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryA2Z9pPMA7y3da8BG

------WebKitFormBoundaryA2Z9pPMA7y3da8BG
Content-Disposition: form-data; name="account-sid"

Q45Ppm5ukvdqjTQ6eW0O5ztTXipwnjKQx1p6cf+fbCQ=
------WebKitFormBoundaryA2Z9pPMA7y3da8BG
Content-Disposition: form-data; name="api-token"

6397cd691909fdc14cef67dbc1dc2dc3
------WebKitFormBoundaryA2Z9pPMA7y3da8BG
Content-Disposition: form-data; name="file"; filename="screen_4_100155.jpg"
Content-Type: image/jpeg

......Exif..MM.*.............................b...........j
------WebKitFormBoundaryA2Z9pPMA7y3da8BG
Content-Disposition: form-data; name="checksum"

6a3381b1d16bded4a3dfc325a8bb800e
------WebKitFormBoundaryA2Z9pPMA7y3da8BG

JVM heap size

-Xmx=1024mb

---

The problem

When uploading ~50MB file two temporary files with similar MD5 sums are created in the directory /tmp/tomcat7-tomcat7-tmp with name FileBackedOutputStream7949386530699987086.tmp and MIME8234229766850016150.tmp

Before upload is complete server throws exception

javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: java.lang.OutOfMemoryError: Java heap space
    org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:421)
    org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:386)
    org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:335)
    org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:222)

and deletes the file with name MIME8234229766850016150.tmp but leaves the other one. And undeleted FileBackedOutputStream....tmp files fill the whole space on hard drive.

---

What I did

1. Increase heap space to 7GB, but ~200MB files can't be uploaded.

2. Run job on server to delete old temporary files.

3. Created file with name jersey-multipart-config.properties and content

   jersey.config.multipart.bufferThreshold = -1

The file MIME[random numbers].tmp is no longer created, but FileBackedOutputStream[random number].tmp still hangs on hard drive unless tomcat is restarted.

---

Question

1. How can Jersey handle large files (maybe 1GB) without leaving temporary files on my hard disk? Best case would be not to use hard drive at all and transfer small chunks through memory.

2. Why do I get heap overflow if input stream is backed with files?

---

Materials I read

• The closes explanation I found so far. read
• This guy has similar problem but on the client side. read
• May contain solution, but couldn't understand the answer. read
• Pretty close to my problem but can't work it out. read
• bufferThreshold idea was taken from here. read

Answer 1

It seems that problem #1 was solved by adding following lines in my web.xml under the <servlet> tag

<multipart-config>
        <location>/tmp</location>
        <max-file-size>1000000000</max-file-size>
        <max-request-size>1500000000</max-request-size>
        <file-size-threshold>0</file-size-threshold>
</multipart-config>

and removed jersey-multipart-config.properties file.

Now I can upload files that are over 200Mb. No more temporary files are created.

But I still can't explain problem #2.