Firestore write to document invalid type errors for numpy

Question

I have a dictionary with over 1000 fields that I'm trying to write to Firestore as a document. However, I keep getting invalid type errors like below:

'Cannot convert to a Firestore Value', 3, 'Invalid type', <class 'numpy.int64'>
'Cannot convert to a Firestore Value', False, 'Invalid type', <class 'numpy.bool_'>

Most of my fields are of these numpy types. I'm new to Firestore so I'm not sure if I missing something simple or If I have to convert all my fields? Does the Firestore python library not allow numpy types? I have multiple nested dictionaries within the main dictionary so it won't be easy to convert everything to different types.

Here is the python code I'm using to write to Firestore:

firestore_db.collection("collection_name").document("doc_name").set(allVarsDict)

Here is a sample of the dictionary:

{
'price_date': '2020-08-22 00:00:00',
'time_ms': 1598054400000,
'price1min': 11526.32,
'price_open1min': 11529.99,
'volume1min': 12.75662516,
'price': 11526.32,
'price1min_dict': {
    'price_date': '2020-08-22 00:00:00',
    'ms': 1598054400000,
    '50ma': 11525.190199999994,
    '128ma': 11547.188046874997,
    '200ma': 11569.5485,
    '200ema': 11552.031164880687,
    'heightPer': 0.0013139635732870457,
    'pinbarPer': 0.24224422442245286,
    'hammer': True,
    'star': False,
    'green': False,
    'pinbar': True,
    'height': True,
    'rsi': 49.58870472682622,
    'Mflow': -3.0986389827593954,
    'cmf': -0.39267912421581946,
    'obv': 12.09637204,
    'stoch': 85.27999999999884,
},

Answer 1

NumPy has a slightly different way to represent data. In other words, numpy.int64 is not the same as int and is therefore not one of the acceptable data types in Firestore

There are many ways to overcome this problem. Depending on how you are obtaining the data, there could be a more efficient approach. However, with the information provided, this is the only way I could suggest which seems to work for me, which involves using ndarray.item method:

def np_to_fs(og_dict):
    for k, v in og_dict.items():
        if type(v).__module__ == 'numpy':
            og_dict[k] = v.item()

This function would convert all instances of values of types originating from numpy and replace them with python-native variables.

Keep in mind that since you have nested dictionaries, you will have to call np_to_fs multiple times, once for each dictionary, and that the function will replace the values in-place.