Multipart file transfer from Java to Coldfusion - no leading boundary: %PDF-1.4

Question

A little history before my question, I posted a question about uploading multiple files to coldfusion using HTML5's multipart/form-data. And its worked beautifully. Can you isolate code from being seen from CF10 compiler?

Our client has finally requested some unit tests for the RESTful functions that I have put together, and I've been able to get quite a lot done, but I've hit a roadblock with the massUpload function that I designed above.

Sorry for the long question, putting down what is relevant to the problem.

Here is the code in question:

Unit Test Code:

//Outside class calling sendHTTPrequest
HashMap<String,String> map = new HashMap<String,String>();
HashMap<String,File> getFiles = getFirstFileList();
map.put("testMethod", "massUploadTest");
map.put("method", "massUpload");
map.put("valueString1", valueString1);
map.put("valueString2", valueString2);
map.put("valueNumeric3", valueNumeric3);
map.put("valueBoolean4", valueBoolean4);
map.put("valueString5", valueString5);
map.put("valueBoolean6", valueBoolean6);
map.put("valueString7", valueString7);

try {                    
    sendHTTPrequest(map, getFiles);
} catch(RuntimeException e) {
    throw new RuntimeException("Fatal error in massUpload\n"
            + e.getMessage());
}
//End Call class code

Coldfusion function:

<cffunction name="massUpload" access="remote" returntype="string">    
    <cfargument name="valueString1"  type="string"  required="false">
    <cfargument name="valueString2"  type="string"  required="false">
    <cfargument name="valueNumeric3" type="numeric" required="false" default=0>
    <cfargument name="valueBoolean4" type="boolean" required="true"  default="false">
    <cfargument name="valueString5"  type="string"  required="false">
    <cfargument name="valueBoolean6" type="boolean" required="false" default="true">
    <cfargument name="valueString7"  type="string"  required="true">

    <!--- massUpload code --->
</cffunction>

So here is the code that is causing the major problems. I've tried different approaches to get the POST to work so obviously the code has problems. I'm trying to do this without downloading multiple software/libraries, but if there is no other way, then I'll make the necessary arrangements. Otherwise, would love to get this done with the standard Java library.

Function code:

//sendHTTPrequest method code
protected static final String BASE_URI = "<webaddress>/rest.cfc";
protected static final String CHARSET = "UTF-8";
protected String response;
protected int status;
protected String statusMessage;

protected void sendHTTPrequest(Map<String,String> map, Map<String, File> fileList) {

        Set<String> keys = map.keySet();
        status = 0;
        response = null;
        String boundary = "----" + System.currentTimeMillis();

        try {            
            URL_CONNECTION = BASE_URL.openConnection();
            HTTP_CONNECTION = (HttpURLConnection) (URL_CONNECTION);

            //Set the request headers
            HTTP_CONNECTION.setRequestMethod("POST");
            URL_CONNECTION.setRequestProperty("Accept-Charset", CHARSET);            
            URL_CONNECTION.setRequestProperty("Content-type", "multipart/form-data; boundary=" + boundary);

            //Set up a post request
            URL_CONNECTION.setDoOutput(true);

            OutputStream output = URL_CONNECTION.getOutputStream();
            ByteArrayOutputStream bOutput = new ByteArrayOutputStream();
            PrintWriter writer = new PrintWriter(new OutputStreamWriter(output, CHARSET), true);

            for(String key : keys) {
                writer.write("--" + boundary);
                writer.write(lineFeed);
                writer.write("Content-Disposition: form-data; name=\"" + key + "\"");
                writer.write(lineFeed);
                writer.write(lineFeed);
                writer.write(map.get(key));
                writer.write(lineFeed);
            }

            FileInputStream inputStream;
            for(Map.Entry<String, File> entry : fileList.entrySet()){
                String name = entry.getKey();
                writer.write("--" + boundary);
                writer.write(lineFeed);
                writer.write("Content-Disposition: form-data; "
                        + "name=\"allfiles\"; filename=\"" + name + "\"");
                writer.write(lineFeed);

                String contentType = URLConnection.guessContentTypeFromName(name);
                writer.write("Content-Type: " + contentType);
                writer.write(lineFeed);    

                writer.write("Content-Transfer-Encoding: binary");
                writer.write(lineFeed);    

                writer.write("Content-Id: <" + boundary + ">");
                writer.write(lineFeed);  
                writer.write(lineFeed);  

                File temp = entry.getValue();

                byte[] buffer = FileUtils.readFileToByteArray(temp);
                output.write(buffer, 0, buffer.length);
                output.flush();

                writer.write(lineFeed);    
            }

            writer.write("--" + boundary + "--");
            writer.write(lineFeed);
            writer.flush();

            writer.close();

            status = HTTP_CONNECTION.getResponseCode();
            statusMessage = HTTP_CONNECTION.getResponseMessage();
            if(status == SUCCESS) {
                response = IOUtils.toString(URL_CONNECTION.getInputStream(), URL_CONNECTION.getContentEncoding());
            }
            System.out.println("Finished test " + map.get("testMethod") + " Status: " + status);
            System.out.println("Response: " + response);
            HTTP_CONNECTION.disconnect();

        } catch(UnknownServiceException use) {
            throw new RuntimeException("Protocol for the output is not supported");
        } catch(IOException ioe) {
            throw new RuntimeException("Unable to create the output stream");
        }
    }
//End sendHTTPrequest method code

I see in the output for status, response and statusMessage, that I get a 500 error, null, Internal Server Error.

Looking on the Coldfusion server I see:

SEVERE: Servlet.service() for servlet [CFCServlet] in context with path [/] threw exception 
java.io.IOException: Corrupt form data: no leading boundary: %PDF-1.4 != ------1429222349902
at com.oreilly.servlet.multipart.MultipartParser.<init>(MultipartParser.java:182)
...

massUpload can handle different types of files, but is looking specifically for PDFs. So the unit test needs to be able to send different file types through to massUpload, not just PDFs.

Any insight into the problem would be welcome. Thank you.

Answer 1

(From comments)

Have you used a packet sniffer to see what the above actually generates?

Try flushing the writer just before you write the file bytes:

...
byte[] buffer = FileUtils.readFileToByteArray(temp);
writer.flush(); // flush stream here
output.write(buffer, 0, buffer.length);
...

Update:

Just to clarify, the reason for the original error is that even though the code creates a PrintWriter with autoFlush=true, it still does not automatically save text to the underlying OutputStream when write() is invoked. The autoFlush setting only affects these methods:

autoFlush - if true, the println, printf, or format methods will [automatically] flush the output buffer

As a result, the code writes the file content to the stream before the boundary markers, which creates an invalid HTTP POST. Hence the 500 error. You can verify this using a package sniffer like Fiddler:

Original - (Invalid):

POST http://localhost:8888/test.cfm HTTP/1.1
Accept-Charset: UTF-8
Content-Type: multipart/form-data; boundary=14cd4c75e24
...
Content-Length: 65570

%PDF-1.4
%    
2 0 obj <</Type/XObject/ColorSpace/DeviceGray/Subtype/Image/BitsPerComponent 8/Width 612/Length 11876/Height 792/Filter/FlateDecode>>stream

... more binary

--14cd4c75e24
Content-Disposition: form-data; name="allfiles"; filename="file0.pdf"
Content-Type: application/pdf
Content-Transfer-Encoding: binary
Content-Id: <14cd4c75e24>


--14cd4c75e24--

The solution is to flush the writer before adding the file content. That ensures the boundary markers are added in the correct location, so the generated POST content is valid.

New Code (Valid)

POST http://localhost:8888/test.cfm HTTP/1.1
Accept-Charset: UTF-8
Content-Type: multipart/form-data; boundary=14cd4c91d29
...
Content-Length: 65570

--14cd4c91d29
Content-Disposition: form-data; name="allfiles"; filename="file0.pdf"
Content-Type: application/pdf
Content-Transfer-Encoding: binary
Content-Id: <14cd4c91d29>

%PDF-1.4
%    
2 0 obj <</Type/XObject/ColorSpace/DeviceGray/Subtype/Image/BitsPerComponent 8/Width 612/Length 11876/Height 792/Filter/FlateDecode>>stream
...

As an aside, I believe Content-Id must be unique. So "boundary" probably is not a good choice for the Content-Id value.