How secure directories? And how to create new directories when a user registers? [closed]

Question

I'm creating a video sharing website in django.

Currently, if a user registers, and upload a video, it will be uploaded to media/vid/uploaded-vid. Then i converted using ffmpeg to flv. What i would like to do is this:

Someone with the username of alex1 registers, i would like to create a directory for him when he confirms his email, called /media/vid/members-vid/alex1

If he uploaded a video, it will be converted to flv in media/vid/uploaded-vid then copied to /media/vid/members-vid/alex1. And the video in media/vid/uploaded-vid should be deleted.

And I would like to secure /media/vid/. How do you secure django directories? Or is just an apache chmod?

i was wondering if i can use celery/rabbitqm to copy files from one folder to another, or to create new folders..

Answer 1

To avoid someone uploading a shell or some other malicious code you should not give access to uploaded file to the outside clients. So user uploads a file, than celery gets that file and processes it, putting the result to another, webserver accessible path. If processing fails (i.e. file is not a valid video), then noone can access it. But with correct webserver setup (i.e. deny running any scripts from locations availble for upload) it should be no big problem anyway.

To avoid users see other's private files, you can also put the files outside of site's mediafiles, and use django's views for checking access rights + special webserver directive to actually server the file without proxying it through Django. These directives are different for diffeent servers:

• Apache - mod_xsendfile
• Lighttpd - XSendfile
• Nginx - X-Accel-Redirect

Answer 2

Your use of a holding directory seems to be a good one. However having directories named after users is risky (what if someone provides a name which is an executable command name?) so using the user id might be better. Indeed if you are going to have lots of users you might want to consider splitting up user directories under some general directories to make finding the directories easier for admin purposes, and because some file systems have rather low limits on the number of entries under a directory -- e.g. ext3 allows roughly 32K entries.

For instance, a crude user directory scheme with outer directories named 1 to 9 for subdirectories starting with those numbers will give you a bit more flexibility here.

So, assuming you have your uploaded files in /tmp/upload/1015/, and you want to move them to /var/userdata/01/1015/ and process files in /var/userdata/01/1015/, the following might be a sensible approach:

• /tmp/upload/1015/ will need to be apache user or group writeable
• record the need to process files in /tmp/upload/1015/ in a database, AMQP scheduling service such as RabbitMQ or an RPC/web services call
• a scheduling service client running under its own userid picks up the job from db/AMQP/web call and does the following
  • run a virus check
  • runs a file sanity check to see if the source file is what it expects
  • copies the file to the user folder
  • if successful, removes the original file (or schedules a job to do that if it does not have permission to do so, or relies on a cronjob to delete files in /tmp/upload that are more than 24 hours old)
  • attempts to convert the files, updating relevant success/failure flags in the database as it goes along
  • deletes temporary files/unsuccessful source/output files on completion

The list obviously would go on further for a while, depending on your needs. In the end you will have a user directory with files readable by apache but not writeable by the apache user. If your service grows you can move the video processing components of this service to another server.

By the way there is a very good description of how to do this sort of thing (very simply) by Cal Henderson of Flickr in his O'Reilly book "Building Scalable Web Sites". It was written in 2006 but his approach is refreshingly direct and straight-forward.

Answer 3

May be you should store your videos in mongoDB GridFS or something similar. So you will not have any trouble with directories, storage, executable files ...