Cache one large object and read from cache fast for flask

Question

I tried flask-cache to cache an object(about 800MB) like this:

@cache.memoize(50000)
def get_nmf_result():
    return NMF_Recommendation(basis_path = app.config['BASIS_PATH'],
                              coef_path = app.config['COEF_PATH'],
                              mask_path = app.config['MASK_PATH'])

But I found that (1) It takes quite a long time to do the caching

(2) Even I cached it, it still need quite a while to read from the cache. Does that mean when I read the cache, actually I'm getting a copyinstead of reference of the cache?

Does anyone have ideas about quick storing and reading for cache? Thanks!

Answer 1

Flask-Cache uses the Werkzeug cache, which uses the pickling library to serialize any cache values to a binary blob, so the value can be saved in any backend. Pickling and unpickling a 800MB object could take quite some time, especially if it's not just one big string or something "simple" like that.

At this point, i think it's better to write your own cache, tailored to the backend you use and the type of data you store.