Writing to CSV Files and then Zipping it up in Appengine (Java)

Question

I'm currently working on a project that is done in Java, on google appengine.

Appengine does not allow files to be stored so any on-disk representation objects cannot be used. Some of these include the File class.

I want to write data and export it to a few csv files, and then zip it up, and allow the user to download it.

How may I do this without using any File classes? I'm not very experienced in file handling so I hope you guys can advise me.

Thanks.

Answer 1

You can create a zip file and add to it while the user is downloading it. If you are using a servlet, this is straigthforward:

protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {

    // ..... process request

    // ..... then respond
    response.setContentType("application/zip");
    response.setStatus(HttpServletResponse.SC_OK);

    // note : intentionally no content-length set, automatic chunked transfer if stream is larger than the internal buffer of the response
    ZipOutputStream zipOut = new ZipOutputStream(response.getOutputStream());

    byte[] buffer = new byte[1024 * 32];
    try {
            // case1: already have input stream, typically ByteArrayInputStream from a byte[] full of previoiusly prepared csv data

            InputStream in = new BufferedInputStream(getMyFirstInputStream());
            try {
                zipOut.putNextEntry(new ZipEntry("FirstName"));

                int length;
                while((length = in.read(buffer)) != -1) {
                    zipOut.write(buffer, 0, length);
                }

                zipOut.closeEntry();
            } finally {
                in.close();
            }

            // case 2: write directly to output stream, i.e. you have your raw data but need to create csv representation

            zipOut.putNextEntry(new ZipEntry("SecondName"));

            // example setup, key is to use the below outputstream 'zipOut' write methods
            Object mySerializer = new MySerializer(); // i.e. csv-writer
            Object myData = getMyData(); // the data to be processed by the serializer in order to make a csv file

            mySerizalier.setOutput(zipOut);

            // write whatever you have to the zipOut
            mySerializer.write(myData);

            zipOut.closeEntry();

            // repeat for the next file.. or make for-loop

        }
    } finally { 
        zipOut.close();
    }
}

There is no reason to store your data in files unless you have memory constraints. Files give you InputStream and OutputStream, both which have in-memory equivalents.

Note that creating a csv writer usually means doing something like this, where the point is to take a piece of data (array list or map, whatever you have) and make it into byte[] parts. Append the byte[] parts into an OutputStream using a tool like DataOutputStream (make your own if you like) or OutputStreamWriter.

Answer 2

If your data is not huge, meaning can stay in memory then exporting to CSV and zipping up and streaming it for downloading can all be done on-they-fly. Caching can be done at any of these steps which greatly depends on your application's business logic.