Uploading large files with Django: How should one go about doing this?

Question

I have to upload >= 20 GBs of data with Django. Should I break the file into chunks and then upload it with some kind of a checksum to maintain integrity or does Django implicitly does it?

Will it be better if I use FTP instead of regular HTTP for such large files?

Answer 1

Django uses so-called Upload Handlers to upload files, and has a related setting called FILE_UPLOAD_MAX_MEMORY_SIZE (default value of 2.5Mb). Files smaller than this threshold will be handled in memory, larger files will be streamed into a temporary file on disk. I haven't yet tried uploading files larger than about 1Gb, but I would expect you can just use django without problems.