Efficient large file upload with Yesod

Question

I want to implement large file upload with my Yesod application. Right now I have:

module Handler.File where

import Import

import System.Random
import System.FilePath
import Control.Monad
import qualified Data.ByteString.Lazy as LBS
import qualified Data.Text.Encoding

-- upload

uploadDirectory :: FilePath -- FIXME: make this configurable
uploadDirectory = "incoming"

randomFileName :: IO FilePath
randomFileName = do
  fname'base <- replicateM 20 (randomRIO ('a','z'))
  let fname = uploadDirectory </> fname'base <.> "bin"
  return fname

fileUploadForm :: Form (FileInfo, Textarea)
fileUploadForm = renderDivs $ (,)
    <$> fileAFormReq "Choose a file"
    <*> areq textareaField "What's on the file?" Nothing

getFileNewR :: Handler RepHtml
getFileNewR = do
  (formWidget, formEnctype) <- generateFormPost fileUploadForm
  defaultLayout $ do
       setTitle "Upload new file."
       $(widgetFile "file-new")

postFileNewR :: Handler RepHtml
postFileNewR = do
  user <- requireAuth
  ((result, formWidget), formEnctype) <- runFormPost fileUploadForm
  case result of
    FormSuccess (fi,info) -> do
                 fn <- liftIO randomFileName
                 liftIO (LBS.writeFile fn (fileContent fi))
                 let newFile = File (entityKey user) fn info (fileName fi) (fileContentType fi)
                 fid <- runDB $ insert newFile
                 redirect (FileViewR fid)
    _ -> return ()

  defaultLayout $ do
       setTitle "Upload new file."
       $(widgetFile "file-new")

It is mostly fine, except few issues:

1. Maximum size of a file is around 2 megabytes. I have a fix, but is it right way to do this? My fix is overriding default implementation of maximumContentLength method in Yesod instance for my app, like this:

   maximumContentLength _ (Just (FileNewR _)) = 2 * 1024 * 1024 * 1024 -- 2 gigabytes maximumContentLength _ _ = 2 * 1024 * 1024 -- 2 megabytes

2. The amount of memory used is equal to the size of a file. This is really suboptimal. I would like to use tempFileBackEnd from http://hackage.haskell.org/packages/archive/wai-extra/1.2.0.4/doc/html/Network-Wai-Parse.html but I have no idea how to actually wire that into my request and make it work with forms logic (hidden _token field etc.).

3. The upload is 'single shot': any examples of how to make it work with Flash/Html5 based uploaders that show progress to user?

Answer 1

1. Your solution is correct. The purpose of the maximumContentLength setting is to allow you to override this value for specific routes that need larger uploads.

2. This is a shortcoming of the current setup of file handling in yesod-core. It's currently hard-coded to use in-memory file uploads. We've discussed on the mailing list in the past that this is suboptimal. I've just created a Github issue for this, and the fix will be included in Yesod 1.1 (no timetable on release though).

3. I don't have an example of this, sorry.