Here is what we have currently:
cache_instance = cache.get_cache("cache_entry")
in static code. During investigation, I found that at the moment the bug happens cache_instance.get(key)
returns wrong value, although get_cache("cache_entry").get(key)
on the next line returns correct one. This means either bug disappears too quickly or for some reason cache_instance object got corrupted.
Isn't cache instance object returned by django's cache thread safe?MemoryError
was logged these daysI know, all of this sounds like some sort of magic.. And really, any ideas how that's possible or how to debug this would be very appreciated.
PS: My current assumption is that this is connected with multiprocessing: as soon as cache instance is created in static code and before Worker process fork this would lead to all workers sharing same socket (Does it sound plausibly?)
Solved it finally:
from django.core.cache import cache
this object stores pre-connected memcached socket. Don't use it when your process could be dynamically forked.. and don't use stored connections, pools and other. This has been bugging me for a while until I found this question and answer. I just want to add some things I've learnt.
You can easily reproduce this problem with a local memcached instance:
from django.core.cache import cache
import os
def write_read_test():
pid = os.getpid()
cache.set(pid, pid)
for x in range(5):
value = cache.get(pid)
if value != pid:
print "Unexpected response {} in process {}. Attempt {}/5".format(
value, pid, x+1)
os._exit(0)
cache.set("access cache", "before fork")
for x in range(5):
if os.fork() == 0:
write_read_test()
What you can do is close the cache client as Django does in the request_finished
signal:
https://github.com/django/django/blob/master/django/core/cache/init.py#L128
If you put a cache.close()
after the fork, everything works as expected.
For celery you could connect to a signal that is fired after the worker is forked and execute cache.close()
.
This also affects gunicorn when preload is active and the cache is initialized before forking the workers.
For gunicorn, you could use post_fork
in your gunicorn configuration:
def post_fork(server, worker):
from django.core.cache import cache
cache.close()
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With