Multiple file upload using GWT and AppEngine Blobstore?

Question

How would I go about creating a modern, Gmail-like, multiple file upload in GWT and AppEngine Blobstore?

The solution most commonly proposed is gwtupload, an excellent GWT component written by Manolo Carrasco. However, the latest version 0.6.6 does not work with blobstore (at least I can't get it to work), and it does not support multiple file select. There's a patch for multiple file select in the latest 0.6.7 snapshot, but although it allows selection of multiple files (using the "multiple" attribute in HTML5), it stills sends them in one huge POST request (and progress is shown for the whole bunch of files).

There are also other questions on SO for this (for example here or here), but the answers usually uses the HTML5 "multiple" attribute and sends them as one big POST request. It works, but its not what I am after.

Answer 1

Nick Johnson wrote some great blog posts about this. He uses the common and well-accepted JavaScript upload component called Plupload, and uploads files to an AppEngine-app written in Python. Plupload supports different backends (runtimes) for supporting multiple file selection (HTML5, flash, Silverlight, etc) and handles upload progress and other upload related client-side events.

The problem with his solution is (1) it's in Python, and (2) it's in JavaScript. This is where gwt-plupload enters the picture. It is a JSNI-wrapper for Plupload written by Samuli Järvelä, which enables use of Plupload inside the GWT environment. However, the project is outdated (no commits since 2010), but we can use it for inspiration.

So, step-by-step instructions for building the multiple file upload component follows. This will all be in one project, but it (especially the JSNI-wrapper) could be extracted to its own .jar-file or library to be reused in other projects. The source code is available on Bitbucket here.

The application is available on AppEngine (non-billable, so don't count on it being available or working) at http://gwt-gaemultiupload-example.appspot.com/.

Screenshots

Step 1 - Servlets

Blobstore works in the following way:

1. Client asks blobstore for a URL it can use for uploading a file.
2. Client POSTs the file to the received URL.
3. When the whole POST is received, blobstore will redirect the client to a success URL (specified when creating the upload URL).

To support this, we will need two servlets. One for generating URLs for file uploads (note that each file upload will need an unique URL), and one for receiving finished uploads. Both will be quite simple. Below is the URL generator servlet, which will just write the URL in plain text to the HTTP response.

public class BlobstoreUrlGeneratorServlet extends HttpServlet {     
    private static BlobstoreService blobstore = BlobstoreServiceFactory.getBlobstoreService();

    @Override
    protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
        resp.setHeader("Content-Type", "text/plain");
        resp.getWriter().write(blobstore.createUploadUrl("/uploadfinished"));
    }
}

And then, the servlet for receiving successful uploads, which will print the blobkey to System.out:

public class BlobstoreUploadFinishedServlet extends HttpServlet {
    private static BlobstoreService blobstore = BlobstoreServiceFactory.getBlobstoreService();

    @Override
    protected void doPost(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
        Map<String, List<BlobKey>> blobs = blobstore.getUploads(req);
        List<BlobKey> blobKeyList = blobs.get("file");

        if (blobKeyList.size() == 0)
            return;

        BlobKey blobKey = blobKeyList.get(0);

        System.out.println("File with blobkey " + blobKey.getKeyString() + " was saved in blobstore.");
    }
}

We also need to register these in web.xml.

<servlet>
    <servlet-name>urlGeneratorServlet</servlet-name>
    <servlet-class>gaemultiupload.server.BlobstoreUrlGeneratorServlet</servlet-class>
</servlet>
<servlet-mapping>
    <servlet-name>urlGeneratorServlet</servlet-name>
    <url-pattern>/generateblobstoreurl</url-pattern>
</servlet-mapping>

<servlet>
    <servlet-name>uploadFinishedServlet</servlet-name>
    <servlet-class>gaemultiupload.server.BlobstoreUploadFinishedServlet</servlet-class>
</servlet>
<servlet-mapping>
    <servlet-name>uploadFinishedServlet</servlet-name>
    <url-pattern>/uploadfinished</url-pattern>
</servlet-mapping>

If we run the app now and visit http://127.0.0.1:8888/generateblobstoreurl, we will see something like

http://<computername>:8888/_ah/upload/ahpnd3QtZ2FlbXVsdGl1cGxvYWQtZXhhbXBsZXIbCxIVX19CbG9iVXBsb2FkU2Vzc2lvbl9fGAEM 

If we were to post a file to that URL, it would be saved in blobstore. Note however that the default URL for the local development web server is http://127.0.0.1:8888/ while the URL generated by blobstore is http://<computername>:8888/. This will cause problems later on, as for security reasons Plupload won't be able to POST files to another domain. This only happens with the local development server, the published app will have only one URL. Fix it by editing the Run Configurations in Eclipse, add -bindAddress <computername> to the arguments. This will cause the local development server to host the web app on http://<computername>:8888/ instead. You might need to allow <computername> in the GWT browser plugin for it to load the app after this change.

So far so good, we have the servlets we need.

Step 2 - Plupload

Download Plupload (I used the latest version, 1.5.4), unzip, and copy the js folder to the war directory in our GWT application. For this example, we won't be using jquery.plupload.queue or jquery.ui.plupload as we'll create our own GUI. We also need jQuery, which I downloaded from Google APIs.

Next, we need to include the JavaScripts in our application, so edit index.html and add the following to the <head> tag.

<script type="text/javascript" language="javascript" src="js/jquery.min.js"></script>
<script type="text/javascript" language="javascript" src="js/plupload.full.js"></script>

So now we have Plupload included in our application. Next, we need to wrap it to be able to use it with GWT. This is where gwt-plupload is used. I didn't use the jar file from the project, but instead copied the source files to be able to make modifications to them. The wrapper's main object is the Plupload class, which is constructed by PluploadBuilder. There's also the interface PluploadListener, which can be implemented to receive client-side events.

Step 3 - Putting it together

So now we need to actually use Plupload in our GWT application. I added the following to an Index.ui.xml UIBinder:

<g:Button text="Browse" ui:field="btnBrowse" />
<g:Button text="Start Upload" ui:field="btnStart" /><br />
<br />
<h:CellTable width="600px" ui:field="tblFiles" />

There's a button for browsing files, a button to start uploading and a CellTable which we will use to display upload status. In Index.java, we initialize Plupload as follows:

btnBrowse.getElement().setId("btn-browse");
PluploadBuilder builder = new PluploadBuilder();
builder.runtime("html5");
builder.useQueryString(false);
builder.multipart(true);
builder.browseButton(btnBrowse.getElement().getId());
builder.listener(this);
plupload = builder.create();
plupload.init();

The runtime attribute tells Plupload which backends to use (I have only tested HTML5, but the others should work as well). Blobstore requires multipart to be enabled. We also need to set an ID for the browse button, and then tell Plupload to use that ID. Clicking this button will popup Plupload's file selection dialog. Last, we add ourselves as listener (implementing PluploadListener) and create() and init() Plupload.

To display the files ready to upload, we just need to add data to the tblFilesDataProvider list data provider in the events from UploadListener.

@Override
public void onFilesAdded(Plupload p, List<File> files) {
    tblFilesDataProvider.getList().addAll(files);
}

To display progress, we simply update the list whenever we are notified progress have changed:

@Override
public void onFileUploadProgress(Plupload p, File file) {
    tblFilesDataProvider.refresh();
}

We also implement a click handler for btnStart, which justs tells Plupload to start uploading.

@UiHandler("btnStart")
void btnStart_Click(ClickEvent event) {
    plupload.start();
}

It is now possible to select files, they will be added to the pending uploads list and we can start the upload. The only piece left is to actually use the servlets we implemented earlier. Currently, Plupload does not know which URL to POST uploads to, so we need to tell it. This is where I have made a change to the gwt-plupload source code (apart from minor bug fixes); I added a function to Plupload called fetchNewUploadUrl. What it does is it performs an Ajax GET request at the servlet we definied earlier to fetch an upload URL. It does this synchronously (why will be clear later). When the requests returns, it sets this URL as the POST URL for Plupload.

private native void fetchNewUploadUrl(Plupload pl) /*-{
    $wnd.$.ajax({
        url: '/generateblobstoreurl',
        async: false,
        success: function(data) {
          pl.settings.url = data;
        },
    });
}-*/;

public void fetchNewUploadUrl() {
    fetchNewUploadUrl(this);
}

Plupload will post each file in its own POST request. This means we need to give it a new URL before each upload starts. Luckily, there's an event for that in PluploadListener which we can implement. And here's the reason why the request has to be synchronous: otherwise the upload would have started before we received the upload URL in the event handler below (pl.fetchNewUploadUrl() would have returned immediately).

@Override
public void onBeforeUpload(Plupload pl, File cast) {
    pl.fetchNewUploadUrl();
}

And that's it! You now have GWT HTML5 multiple file upload functionality placing files in AppEngine Blobstore!

Passing parameters

If you want to additional parameters (such as an ID for the entity to which the uploaded files belong), I added an example on how to add one. There's a method on Plupload called setExtraValue() I implemented as:

public native void setExtraValue(String value) /*-{
    this.settings.multipart_params = {extravalue: value}
}-*/;

Extra values can be passed as multipart_params. This is a map, so the functionality could be extended to allow many arbitrary key-value pairs. The value can be set in the onBeforeUpload() event handler

@Override
public void onBeforeUpload(Plupload pl, File cast) {
    pl.setExtraValue(System.currentTimeMillis() + " is unique.");
    pl.fetchNewUploadUrl();
}

and retrieved in the servlet receiving finished uploads as

String value = req.getParameter("extravalue");

The example project contains sample code for this.

Final words

I am no way an expert GWT developer. This is what I came up with after hours of frustration not finding the functionality I was after. After I got it working, I thought I should write up a complete example, as every component/blog post/etc I used/followed had left some part out. I do not imply this to be best practice code in any way. Comments, improvements and suggestions are welcome!