How do I convert a django QuerySet to numpy record array?

Question

How do I convert a django QuerySet to numpy record array?

PS: I know you can iterate and construct it and but is there any other cleaner solution?

Answer 1

import numpy as np
qs = MyModel.objects.all()
vlqs = qs.values_list()
r = np.core.records.fromrecords(vlqs, names=[f.name for f in MyModel._meta.fields])

This uses the QuerySet iterator directly and avoids the time-and-garbage-wasting step of creating a python list. It also uses MyModel._meta.fields to get the actual field names from the model, as explained at Get model's fields in Django

If you just want a single field (e.g. the 'votes' field of the model) extracted into a one-dimensional array, you can do:

vlqs = qs.values_list('votes', flat=True)
votes = np.fromiter(vlqs, numpy.dtype('int_'))

Answer 2

This is like asking "how do I convert the contents of my fridge into dinner?". It depends on what you have in your fridge and what you'd like to eat. The short answer (equivalent to saying "by cooking") is to iterate over the queryset, constructing objects of whatever composite data types you'd like to instantiate the array with (generally an iterable and a dictionary). The long answer depends on what you'd actually like to accomplish.

Answer 3

If you want to get all of your objects and create a numpy array with objects as elements of array:

import numpy as np
qs = MyModel.objects.all()
numpy_array = np.array(list(qs))

According to my work, I use something as below:

import numpy as np    
qs = MyModel.objects.values_list('id','first_name','last_name').filter(gender='male').order_by('id')
numpy_array = np.array(list(qs))

Rows of array corresponds to records and columns of array corresponds to values that I defined above (id, first name, last name).

Answer 4

What I was looking for:

1. From QuerySet qs get vlqs (django.db.models.query.ValuesListQuerySet)

   vlqs = qs.values_list()
   

2. Covert vlqs to list

   mylist = list(vlqs)
   

3. Create numpy record array

   # Names are the model fields
   r = np.core.records.array(mylist, names='field1, field2, field3') 
   

Answer 5

what you could do is:

[index[0] for index in qs.values_list('votes')]

and ready...XD

Answer 6

And to put it into a neat little function to which you just pass any Django Queryset:

import pandas as pd
import numpy as np

def qs_to_df(qs):
    """ QuerySet to DataFrame """
    Model = qs.model
    np_array = np.core.records.fromrecords(qs.values_list(), names=[f.name for f in Model._meta.fields])
    return pd.DataFrame(np_array)