How to add multiple images to a django form asynchronously before form submit

Question

Intro: I have a python Django web app where users are allowed to create posts. Each post has 1 main image and followed by extra images (max 12 & min 2) that are associated with that post. I want to let users add a total of 13 images. 1 main image and 12 extra images.

The issue: Usually users take photos with their smart phones. which makes image size upto 10MB . with 13 images that can become 130MB form. My django server can accept a max of 10MB form. So I cannot reduce the images ServerSide

What I want to do: I want such that when the user uploads each image to a form. The size of that image is reduced on client side and it is asynchronously saved in a temporary place on my server using Ajax. When the post is created all these images are linked to the post. So basically when the user hits submit on the post create form. Its a super light form with no images. Sounds too ambitious.. ha ha maybe

What I have so far:

1. I have the models/views (all django parts that create a post) without the asynchronous part. As in, if the form after all images are added is less than 10MB. My post is created with how many ever extra images
2. I have the Javascript code that reduces the size of the images on the client side and asynchronously adds it to my server. All I need to do is give it a endpoint, which is a simple url
3. I have a rough idea of how I plan to achieve this

Now to show you my code

My Models (Just the django part no asynchronous part added as yet)

class Post(models.Model):
    user = models.ForeignKey(User, related_name='posts')
    title = models.CharField(max_length=250, unique=True)
    slug = models.SlugField(allow_unicode=True, unique=True, max_length=500)
    message = models.TextField()
    post_image = models.ImageField()

class Extra (models.Model): #(Images)
    post = models.ForeignKey(Post, on_delete=models.CASCADE, related_name='post_extra')
    image = models.ImageField(upload_to='images/', blank=True, null=True, default='')
    image_title = models.CharField(max_length=100, default='')
    image_description = models.CharField(max_length=250, default='')
    sequence = models.SmallIntegerField(validators=[MaxValueValidator(12), MinValueValidator(1)])

My views (Just the django part no asynchronous part added as yet)

@login_required
def post_create(request):
    ImageFormSet = modelformset_factory(Extra, fields=('image', 'image_title', 'image_description'), extra=12, max_num=12,
                                        min_num=2)
    if request.method == "POST":
        form = PostForm(request.POST or None, request.FILES or None)
        formset = ImageFormSet(request.POST or None, request.FILES or None)
        if form.is_valid() and formset.is_valid():
            instance = form.save(commit=False)
            instance.user = request.user
            instance.save()
            for index, f in enumerate(formset.cleaned_data):
                try:
                    photo = Extra(sequence=index+1, post=instance, image=f['image'],
                                 image_title=f['image_title'], image_description=f['image_description'])
                    photo.save()
                except Exception as e:
                    break   

            return redirect('posts:single', username=instance.user.username, slug=instance.slug)

Now Just to keep things simple I will not add any Javascript in this question. Adding the below script tag to my form makes the image saved asynchronously to the server. You can read more about Filepond if you wish

'''See the urls below to see where the **new_image** is coming from'''
    FilePond.setOptions({ server: "new_image/",
                          headers: {"X-CSRF-Token": "{% csrf_token %}"}}
    }); #I need to figure how to pass the csrf to this request Currently this is throwing error

My plan to make it work

Add a new model below the existing 2 models

class ReducedImages(models.Model):
    image = models.ImageField()
    post = models.ForeignKey(Post, blank=True, null=True, upload_to='reduced_post_images/')

Change the view as below (only working on the main image for now. Not sure how to get the Extra images )

''' This could be my asynchronous code  '''
@login_required
def post_image_create(request, post):
    image = ReducedImages.objects.create(image=request.FILES)
    image.save()
    if post:
        post.post_image = image


@login_required
def post_create(request):
    ImageFormSet = modelformset_factory(Extra, fields=('image', 'image_title', 'image_description'), extra=12, max_num=12,
                                        min_num=2)
    if request.method == "POST":
        form = PostForm(request.POST or None)
        formset = ImageFormSet(request.POST or None, request.FILES or None)
        if form.is_valid() and formset.is_valid():
            instance = form.save(commit=False)
            instance.user = request.user
            post_image_create(request=request, post=instance) #This function is defined above
            instance.save()
            for index, f in enumerate(formset.cleaned_data):
                try:
                    photo = Extra(sequence=index+1, post=instance, image=f['image'],
                                 image_title=f['image_title'], image_description=f['image_description'])
                    photo.save()

                except Exception as e:
                    break
            return redirect('posts:single', username=instance.user.username, slug=instance.slug)
    else:
        form = PostForm()
        formset = ImageFormSet(queryset=Extra.objects.none())
    context = {
        'form': form,
        'formset': formset,
    }
    return render(request, 'posts/post_form.html', context)

my urls.py

url(r'^new_image/$', views.post_image_create, name='new_image'),

Any suggestions on how I can make this work

My Templates

{% extends 'posts/post_base.html' %}
{% load bootstrap3 %}
{% load staticfiles %}

{% block postcontent %}
<head>

    <meta http-equiv="X-UA-Compatible" content="ie=edge">
    <link href="https://unpkg.com/filepond/dist/filepond.css" rel="stylesheet" type="text/css"/>
    <link href="https://unpkg.com/filepond-plugin-image-edit/dist/filepond-plugin-image-edit.css" rel="stylesheet" type="text/css"/>
    <link href="https://unpkg.com/filepond-plugin-image-preview/dist/filepond-plugin-image-preview.css" rel="stylesheet" type="text/css"/>
    <link href="{% static 'doka.min.css' %}" rel="stylesheet" type="text/css"/>
    <style>
    html {
        font-family: -apple-system,BlinkMacSystemFont,Segoe UI,Roboto,Helvetica,Arial,sans-serif;
        font-size: 1em;
    }

    body {
        padding: 2em;
        max-width: 30em;
    }
    </style>
</head>
<body>
<div class="container">
    <h2> Add a new Recipe</h2>
    <form action="" method="post" enctype="multipart/form-data" id="form">
        {% csrf_token %}
        {% bootstrap_form form %}
        <img alt="" id="preview" src="" width="100" />
        <img alt="" id="new_image" src="" style="display: none;"  />
        {{formset.management_form}}
          <h3 class="text-danger">You must be present in at least 1 image making the dish. With your face clearly visible and
            matching your profile picture
        </h3>
        <h5>(Remember a picture is worth a thousand words) try to add as many extra images as possible
            <span class="text-danger"><b>(Minimum 2)</b></span>.
            People love to see how its made. Try not to add terms/language which only a few people understand.

         Please add your own images. The ones you took while making the dish. Do not copy images</h5>
        {% for f in formset %}
            <div style="border-style: inset; padding:20px; display: none;" id="form{{forloop.counter}}" >
                <p class="text-warning">Extra Image {{forloop.counter}}</p>
                {% bootstrap_form f %}

                <img alt="" src="" width="60" id="extra_image{{forloop.counter}}"  />
            </div>
        {% endfor %}

        <br/><button type="button" id="add_more" onclick="myFunction()">Add more images</button>

        <input type="submit" class="btn btn-primary" value="Post" style="float:right;"/>

    </form>

</div>
<script>
    [
        {supported: 'Promise' in window, fill: 'https://cdn.jsdelivr.net/npm/promise-polyfill@8/dist/polyfill.min.js'},
        {supported: 'fetch' in window, fill: 'https://cdn.jsdelivr.net/npm/[email protected]/fetch.min.js'},
        {supported: 'CustomEvent' in window && 'log10' in Math && 'sign' in Math &&  'assign' in Object &&  'from' in Array &&
                    ['find', 'findIndex', 'includes'].reduce(function(previous, prop) { return (prop in Array.prototype) ? previous : false; }, true), fill: 'doka.polyfill.min.js'}
    ].forEach(function(p) {
        if (p.supported) return;
        document.write('<script src="' + p.fill + '"><\/script>');
    });
    </script>

    <script src="https://unpkg.com/filepond-plugin-image-edit"></script>
    <script src="https://unpkg.com/filepond-plugin-image-preview"></script>
    <script src="https://unpkg.com/filepond-plugin-image-exif-orientation"></script>
    <script src="https://unpkg.com/filepond-plugin-image-crop"></script>
    <script src="https://unpkg.com/filepond-plugin-image-resize"></script>
    <script src="https://unpkg.com/filepond-plugin-image-transform"></script>
    <script src="https://unpkg.com/filepond"></script>

    <script src="{% static 'doka.min.js' %}"></script>

    <script>

    FilePond.registerPlugin(
        FilePondPluginImageExifOrientation,
        FilePondPluginImagePreview,
        FilePondPluginImageCrop,
        FilePondPluginImageResize,
        FilePondPluginImageTransform,
        FilePondPluginImageEdit
    );

// Below is my failed attempt to tackle the csrf issue

const csrftoken = $("[name=csrfmiddlewaretoken]").val();


FilePond.setOptions({
    server: {
        url: 'http://127.0.0.1:8000',
        process: {
            url: 'new_image/',
            method: 'POST',
            withCredentials: false,
            headers: {
                headers:{
        "X-CSRFToken": csrftoken
            },
            timeout: 7000,
            onload: null,
            onerror: null,
            ondata: null
        }
    }
}});


// This is the expanded version of the Javascript code that uploads the image


    FilePond.create(document.querySelector('input[type="file"]'), {

        // configure Doka
        imageEditEditor: Doka.create({
            cropAspectRatioOptions: [
                {
                    label: 'Free',
                    value: null
                }                   
            ]
        })

    });

The below codes are exacty like the one above. I have just minimised it

FilePond.create(document.querySelector('input[type="file"]'), {...});
FilePond.create(document.querySelector('input[type="file"]'), {...});
FilePond.create(document.querySelector('input[type="file"]'), {...});
FilePond.create(document.querySelector('input[type="file"]'), {...});
FilePond.create(document.querySelector('input[type="file"]'), {...});
FilePond.create(document.querySelector('input[type="file"]'), {...});
FilePond.create(document.querySelector('input[type="file"]'), {...});
FilePond.create(document.querySelector('input[type="file"]'), {...});
FilePond.create(document.querySelector('input[type="file"]'), {...});
FilePond.create(document.querySelector('input[type="file"]'), {...});
FilePond.create(document.querySelector('input[type="file"]'), {...});
FilePond.create(document.querySelector('input[type="file"]'), {...});


// ignore this part This is just to have a new form appear when the add more image button is pressed. Default is 3 images


<script>
    document.getElementById("form1").style.display = "block";
    document.getElementById("form2").style.display = "block";
    document.getElementById("form3").style.display = "block";   

    let x = 0;
    let i = 4;
    function myFunction() {

          if( x < 13) {
            x = i ++
          }
      document.getElementById("form"+x+"").style.display = "block";
    }
</script>
</body>


{% endblock %}

I have not added the forms.py as they were not relevant

Answer 1

According to your problem there are four things to do.

1. Make temporary files storage tracker.
2. Uploading files immediately after user selects image (Somewhere on the storage may be temporary location) server responds with link of reduced image.
3. When user Posts form that pass just references to those images then save Post with given references.
4. Handle temporary location efficiently. (By some batch processing or some celery tasks.)

Solution

1. Make temporary files storage tracker for files those are uploaded asynchronously.

Your temporary uploaded files will be stored in TemporaryImage model in the temp_folder as following structure.

Update your models.py

models.py

class TemporaryImage(models.Model):
    image = models.ImageField(upload_to="temp_folder/")
    reduced_image = models.ImageField(upload_to="temp_thumb_folder/")
    image_title = models.CharField(max_length=100, default='')
    image_description = models.CharField(max_length=250, default='')
    sequence = models.SmallIntegerField(validators=[MaxValueValidator(12), MinValueValidator(1)])


class Post(models.Model):
    user = models.ForeignKey(User, related_name='posts')
    title = models.CharField(max_length=250, unique=True)
    slug = models.SlugField(allow_unicode=True, unique=True, max_length=500)
    message = models.TextField()
    post_image = models.ImageField()

class Extra (models.Model): #(Images)
    post = models.ForeignKey(Post, on_delete=models.CASCADE, related_name='post_extra')
    image = models.ImageField(upload_to='images/', blank=True, null=True, default='')
    image_thumbnail = models.ImageField(upload_to='images/', blank=True, null=True, default='')
    image_title = models.CharField(max_length=100, default='')
    image_description = models.CharField(max_length=250, default='')
    sequence = models.SmallIntegerField(validators=[MaxValueValidator(12), MinValueValidator(1)])

Here TemporaryImage contains temporary uploaded files the field raw_image represent original uploaded file and reduced_image stands for thumbnails which is generated after file upload.

In order to send asynchronous java script request you need to install django-restframewrok by following command.

pip install djangorestframework

After installing restframework add serializers.py with following code.

serializers.py

from rest_framework import serializers


class TemporaryImageUploadSerializer(serializers.ModelSerializer):
    class Meta:
        model = TemporaryImage
        field = ('id', 'image',)

    def create(self, validated_data):
        raw_image = validated_data['raw_image']
        # Generate raw image's thumbnail here
        thumbnail = generate_thumbnail(raw_image)
        validated_data['reduced_image'] = thumbnail
        return super(TemporaryImageUploadSerializer, self).create(validated_data)

This serializer generates thumbnail when user asynchronously uploads file. generate_thumbnail function will do this job. Implementation of this method can be found from here.

Add this serializer in viewset as below

apis.py

from rest_framework.generics import CreateAPIView, DestroyAPIView
from .serializers import TemporaryImageUploadSerializer

# This api view is used to create model entry for temporary uploaded file
class TemporaryImageUploadView(CreateAPIView):
    serializer_class = TemporaryImageUploadSerializer
    queryset = TemporaryImage.objects.all()

class TemporaryImageDeleteView(DestroyAPIView):
    lookup_field = 'id'
    serializer_class = TemporaryImageUploadSerializer
    queryset = TemporaryImage.objects.all()

This TemporaryImageUploadViewSet creates POST, PUT, PATCH, DELETE methods for your uploads.

Update your urls.py as below

urls.py

from .apis import TemporaryImageUploadView, TemporaryImageDeleteView

urlpatterns = [
  ...
  url(r'^ajax/temp_upload/$', TemporaryImageUploadView.as_view()),
  url(r'^ajax/temp_upload/(?P<user_uuid>[0-9]+)/$', TemporaryImageDeleteView.as_view()),
  ...
]

This will create following endpoints for to handle asynchronous uploads

• <domain>/ajax/temp_upload/ POST
• <domain>/ajax/temp_upload/{id}/ DELETE

Now those endpoints are ready to handle file uploads

2. Uploading files immediately after user selects image

For this you need to update your template.py to handle iamge uploads when user select extra images and post with image field upload this to <domain>/ajax/temp_upload/ with POST method this would return you following sample json data.

{
    "id": 12,
    "image": "/media/temp_folder/image12.jpg",
    "reduced_image": "/media/temp_thumb_folder/image12.jpg",
}

You can preview image from reduced_image key inside json.

id is reference for your temporary uploaded file you need to store it somewhere to pass in the Post create form. i.e As hidden field.

I am not writing javascript code because answer will become more lengthy.

3. When user Posts form that pass just references to those images.

The uploaded files' id is set as Hidden field on formset in the HTML page. In order to handle formset you need to do following.

forms.py

from django import forms

class TempFileForm(forms.ModelForm):
    id = forms.HiddenInput()
    class Meta:
        model = TemporaryImage
        fields = ('id',)

    def clean(self):
        cleaned_data = super().clean()
        temp_id = cleaned_data.get("id")
        if temp_id and not TemporaryImage.objects.filter(id=temp_id).first():
            raise forms.ValidationError("Can not find valida temp file")

This is single uploaded temporary file form.

You could handle this by using formset in django as below

forms.py

from django.core.files.base import ContentFile

@login_required
def post_create(request):
    ImageFormSet = formset_factory(TempFileForm, extra=12, max_num=12,
                                        min_num=2)
    if request.method == "POST":
        form = PostForm(request.POST or None)
        formset = ImageFormSet(request.POST or None, request.FILES or None)
        if form.is_valid() and formset.is_valid():
            instance = form.save(commit=False)
            instance.user = request.user
            post_image_create(request=request, post=instance) #This function is defined above
            instance.save()
            for index, f in enumerate(formset.cleaned_data):
                try:
                    temp_photo = TemporaryImage.objects.get(id=f['id'])

                    photo = Extra(sequence=index+1, post=instance,
                                 image_title=f['image_title'], image_description=f['image_description'])
                    photo.image.save(ContentFile(temp_photo.image.name,temp_photo.image.file.read()))
                    
                    # remove temporary stored file
                    temp_photo.image.file.close()
                    temp_photo.delete()
                    photo.save()

                except Exception as e:
                    break
            return redirect('posts:single', username=instance.user.username, slug=instance.slug)
    else:
        form = PostForm()
        formset = ImageFormSet(queryset=Extra.objects.none())
    context = {
        'form': form,
        'formset': formset,
    }
    return render(request, 'posts/post_form.html', context)

This would save Post with given references(temporary uploaded files).

4. Handle temporary location efficiently.

You need to handle temp_folder and temp_thumb_folder to keep your file system clean.

Suppose user uploads file and does not submit post form than you need to remove that files.

I know the answer became too lengthy to read, apologise for that yet edit this post if any improvements

Refer https://medium.com/zeitcode/asynchronous-file-uploads-with-django-forms-b741720dc952 for post related to this

Answer 2

Below is the answer I feel may be simpler to solve the above problem

How I got this idea

I wanted to send someone a email. I clicked compose I did not type anything. I got distracted by something and accidentally closed the browser. When I opened the email again. I saw there was a draft. It did not have anything in it. I was like Eureka!

What does a email have

sender = (models.ForeignKey(User))
receiver =  models.ForeignKey(User
subject =  models.CharField()
message = models.TextFied()
created_at = models.DateTimefield()


#Lets assume that Multiple attachments are like my model above.

Now the thing to notice is when I clicked compose and closed the window. It had only 2 of the above attributes

  sender = request.user
  created_at = timezone.now()

It created the email object with just these 2 things. So all the remaining attributes were optional. Also it saved it as a Draft hence there was another attribute called

is_draft = models.BooleanField(default=True)

I am sorry I have typed so much stuff and I still haven't got to the point(I have been watching a lot of court room drama. Its all relevant)

Now Lets apply all this to my problem.(I am sure some of you have already guessed the solution)

My models

'''I have made a lot of attributes optional'''
class Post(models.Model):
    user = models.ForeignKey(User, related_name='posts') #required
    title = models.CharField(max_length=250, unique=True, blank=True, null=True,) #optional
    slug = models.SlugField(allow_unicode=True, unique=True, max_length=500, blank=True, null=True,) #optional
    message = models.TextField(blank=True, null=True,) #optional
    post_image = models.ImageField(blank=True, null=True,) #optional
    created_at = models.DateTimeField(auto_now_add=True) #auto-genetrated
    is_draft = models.BooleanField(default=True) #I just added this new field

class Extra (models.Model): #(Images)
    post = models.ForeignKey(Post, on_delete=models.CASCADE, related_name='post_extra') #This is required 
    image = models.ImageField(upload_to='images/', blank=True, null=True, default='') #optional
    image_title = models.CharField(max_length=100, default='') #optional
    image_description = models.CharField(max_length=250, default='') #optional
    sequence = models.SmallIntegerField(validators=[MaxValueValidator(12), MinValueValidator(1)]) #optional

Now in my code above the only thing needed to create this post is a logged_in user

I created a tab on my navbar called Drafts

Before: When the user clicked on add a post. A Blank form was rendered. which the user filled and when all the requirements were satisfied the post object was created. The create_post function above managed the view for creation of this post

Now: When the user clicks add a post. A post is created immediately and the Blank form that the user now sees is the post_edit form. I am adding Javascript barriers to stop the form from submitting unless all my previously required fields are satisfied.

The images are added asynchronously from my post_edit form. They are not orphaned images anymore. I don't need another model like previously to temporarily save the images. when the user adds the images they will get send one by one to the server. If everything is done properly. After all the images are added asynchronously. The user submits a super light form when he/she clicks submit. If the user abandons the form It is left on the users Navbar as Draft(1). You can let the user delete this draft. if he does not need it. Or have a simple code like

delete the draft after1 week if its still a draft. YOu can add this at user sign in

if post.is_draft and post.created_at > date__gt=datetime.date.today() + datetime.timedelta(days=6)

I will try and make a github code for exact execution with the javascript components.

Please let me know what you think of this approach. how I can better this. Or ask me doubts if something is not clear