Downloading large file in ColdFusion using CFHTTP

Question

I'm trying to download a large (600MB) binary file to the server using CFHTTP on Coldfusion 8:

<cfhttp 
  method="get" 
  url="#fileURL#" 
  path="#filePath#" 
  file="#fileName#" 
  timeout="600" 
  getasbinary="auto"
>

It's working fine for smaller files (100something MB) but for the large one's I'm getting the Server 500 error: "java.lang.OutOfMemoryError: Java heap space".

The file is being downloaded from a Dropbox folder - so only available option is to use HTTP GET.

Does anyone have idea how to download it, so it wouldn't kill the server or timeout?

Answer 1

You can do this by calling Java from CF code. The buffered input and output stream classees are designed to hold onto chunks of data, rather than the whole thing, avoiding OutOfMemory errors.

getByteArray() is a helper method because there's not way to declare something like byte buf[]=new byte[1024]; in CF directly.

In the example change the source and destination variables.

Example

<cfset source = 'protocol://domain/path/to/file.ext'>
<cfset destination = getDirectoryFromPath(getCurrentTemplatePath()) & listlast(source, "/")>
<cffunction name="getByteArray" access="private" returnType="binary" output="no">
    <cfargument name="size" type="numeric" required="true"/>
    <cfset var emptyByteArray =
        createObject("java", "java.io.ByteArrayOutputStream").init().toByteArray()/>
    <cfset var byteClass = emptyByteArray.getClass().getComponentType()/>
    <cfset var byteArray =
        createObject("java","java.lang.reflect.Array").newInstance(byteClass, arguments.size)/>
    <cfreturn byteArray/>
</cffunction>

<cfscript>
uri = createObject("java", "java.net.URL").init(source);
uis = uri.openStream();
bis = createObject("java", "java.io.BufferedInputStream").init(uis);
fos = createObject("java", "java.io.FileOutputStream").init(destination);
bos = createObject("java", "java.io.BufferedOutputStream").init(fos);
buffer = getByteArray(1024);
len = bis.read(buffer);
while(len > 0) {
    bos.write(buffer,0,len);
    len = bis.read(buffer);
}
bos.close();
bis.close();
fos.close();
uis.close();
</cfscript>

Answer 2

The problem is is that it's too large. ColdFusion reads the entire thing into memory before writing it to disk.

You'll be better off using some other method to get the file. wget can do http requests from a command-line. That with judicious use of CFEXECUTE is probably a good way to go.