How can I serialise a Flask app or request context, or a subset of that context (i.e. whatever can be successfully serialised) so that I can access that context from another process, rather than a thread?
I have some functions that require access to the Flask request context, or the App context, that I want to run in the background.
Flask has a built-in @copy_current_request_context
decorator to wrap a function in a copy of the request context, so you can run it in a different thread:
from threading import Thread
from flask import Flask, request, copy_current_request_context
app = Flask(__name__)
@app.route('/')
def index():
request.foo = 'bar'
@copy_current_request_context
def baz():
print(request.foo)
thr = Thread(target=baz)
thr.start()
return 'ok'
Whilst Flask doesn't provide a built-in decorator to copy the app context, it provides the machinery to do it - a solution is described on Access flask.g inside greenlet
First of all, I was hoping to solve with Pickle. This is what is used by concurrent.futures.ProcessPoolExecutor
. Unfortunately, Pickle fails due to the presence of thread lock objects inside the app context:
>>> with app.test_request_context('/'):
... pickle.dumps(appctx)
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
TypeError: can't pickle _thread.lock objects
It also can't copy decorated/wrapped functions:
>>> with app.test_request_context('/'):
... bar = copy_current_request_context(pow)
... pickle.dumps(bar)
...
Traceback (most recent call last):
File "<stdin>", line 3, in <module>
_pickle.PicklingError: Can't pickle <function pow at 0x110bee8c8>: it's not the same object as builtins.pow
Next I tried dill
. It passes the above two tests, but it can't serialise a lot of other things that might end up in a context. App contexts especially are prone to having extensions link themselves in: SQLAlchemy is a great example. Here's what happens when you're using SQLAlchemy and you try to serialise your app context:
>>> from flask_sqlalchemy import SQLAlchemy
>>> db = SQLAlchemy(app)
>>> with app.test_request_context('/'):
... from flask.globals import _app_ctx_stack
... appctx = _app_ctx_stack.top
... dill.dumps(appctx)
...
Traceback (most recent call last):
File "<stdin>", line 4, in <module>
File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 294, in dumps
dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 287, in dump
pik.dump(obj)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 437, in dump
self.save(obj)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
save(state)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 856, in save_dict
self._batch_setitems(obj.items())
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 882, in _batch_setitems
save(v)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
save(state)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 856, in save_dict
self._batch_setitems(obj.items())
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 882, in _batch_setitems
save(v)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 816, in save_list
self._batch_appends(obj)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 843, in _batch_appends
save(tmp[0])
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 1386, in save_function
obj.__dict__), obj=obj)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 638, in save_reduce
save(args)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 786, in save_tuple
save(element)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 771, in save_tuple
save(element)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 1129, in save_cell
pickler.save_reduce(_create_cell, (f,), obj=obj)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 638, in save_reduce
save(args)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 771, in save_tuple
save(element)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
save(state)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 856, in save_dict
self._batch_setitems(obj.items())
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 882, in _batch_setitems
save(v)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
save(state)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 856, in save_dict
self._batch_setitems(obj.items())
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 882, in _batch_setitems
save(v)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 549, in save
self.save_reduce(obj=obj, *rv)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 662, in save_reduce
save(state)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 902, in save_module_dict
StockPickler.save_dict(pickler, obj)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 856, in save_dict
self._batch_setitems(obj.items())
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 882, in _batch_setitems
save(v)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 504, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/dchevell/Development/python3/sandbox/env/lib/python3.7/site-packages/dill/_dill.py", line 1330, in save_type
StockPickler.save_global(pickler, obj)
File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pickle.py", line 957, in save_global
(obj, module_name, name)) from None
_pickle.PicklingError: Can't pickle <class 'sqlalchemy.orm.session.SignallingSession'>: it's not found as sqlalchemy.orm.session.SignallingSession
So: is there any way around this? One thought I had was that forking a new process "just in time" - i.e. immediately before I want to run my background task - may mean I don't even need to copy a context at all, it should already have it as long as the fork happens after the context is set up. I don't know how to make this work with a ProcessPoolExecutor
however, where processes are a long lived pool. The only idea I have there is to somehow force worker processes to shut down at the end of each task but I'm pretty sure that this will just result in a broken pool.
Not sure if this is suitable for you, but there is a possibility to use Flask application context from other processes. It requires you to utilize the application factory pattern.
# your_project/app.py
def create_app(config_file):
# Load the config file somehow
# Create your app instance
app = Flask(__name__)
# Initialize your extensions
db.init_app(app)
# Mount routes, etc
app.register_blueprint(...)
# Return your app
return app
# your_project/wsgi.py
from .app import create_app
app = create_app("./config/local.cfg")
You define an application factory and use it to create your app as an entry point in wsgi.py
or another convenient location, and you can later access this app instance from your background script which would run in another process.
# your_project/background_script.py
from .wsgi import app
with app.app_context():
# Do some stuff
BTW, this is also a way to use Flask with Celery, for example, as described in this question. Hope this helps you.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With