Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Remove duplicates in a Django query

Tags:

sql

django

Is there a simple way to remove duplicates in the following basic query:

email_list = Emails.objects.order_by('email') 

I tried using duplicate() but it was not working. What is the exact syntax for doing this query without duplicates?

like image 221
David542 Avatar asked May 04 '11 00:05

David542


People also ask

How to remove duplicates from query in django?

In each loop, you launch a first query to see if i not in d , and possibly another one to delete the affected address records. You can do this in the ORM directly without the Python loop by doing something like: Address. objects. exclude(pk__in=d.

How do you prevent duplicates in Django?

Use the get_or_create() Method in Django When we create duplicate objects multiple times, this method helps us avoid creating them multiple times.


1 Answers

This query will not give you duplicates - ie, it will give you all the rows in the database, ordered by email.

However, I presume what you mean is that you have duplicate data within your database. Adding distinct() here won't help, because even if you have only one field, you also have an automatic id field - so the combination of id+email is not unique.

Assuming you only need one field, email_address, de-duplicated, you can do this:

email_list = Email.objects.values_list('email', flat=True).distinct() 

However, you should really fix the root problem, and remove the duplicate data from your database.

Example, deleting duplicate Emails by email field:

for email in Email.objects.values_list('email', flat=True).distinct():     Email.objects.filter(pk__in=Email.objects.filter(email=email).values_list('id', flat=True)[1:]).delete() 

Or books by name:

for name in Book.objects.values_list('name', flat=True).distinct():      Book.objects.filter(pk__in=Artwork.objects.filter(name=name).values_list('id', flat=True)[3:]).delete() 
like image 124
Daniel Roseman Avatar answered Oct 08 '22 11:10

Daniel Roseman