Pickle is refusing to serialize content with celery reporting ContentDisallowed: Refusing to deserialize untrusted content of type pickle

Question

I am trying to put some python object mostly json serializable except datetime.datetime in rabbitmq queue and so using pickle to serialize.

celery_config file:

CELERY_TASK_SERIALIZER = 'pickle'
CELERY_RESULT_SERIALIZER = 'pickle'

It is throwing an exception saying:

 File "/usr/local/lib/python2.7/dist-packages/kombu/serialization.py", line 174, in loads
    raise self._for_untrusted_content(content_type, 'untrusted')
ContentDisallowed: Refusing to deserialize untrusted content of type pickle (application/x-python-serialize)

This link suggests I do message signing about which I have no clue.

Can someone please guide me through how do I work it out?

Answer 1

Have you tried, this:

CELERY_ACCEPT_CONTENT = ['pickle']

As indicated in this link ( http://docs.celeryproject.org/en/latest/userguide/configuration.html#std:setting-accept_content) this setting accepts a list of serializer names and content-types, so you could either white-list the serializer or the content-types you expect to serialize.

So either do the above, or use SSL message signing… which is basically, building a ssh-key pair, and enabling celery to use your keys to get a secure connection.

You can activate message signing, by registering your "KEY" and "CERTIFICATE" with:

CELERY_SECURITY_KEY = '/etc/ssl/private/worker.key'
CELERY_SECURITY_CERTIFICATE = '/etc/ssl/certs/worker.pem'
CELERY_SECURITY_CERT_STORE = '/etc/ssl/certs/*.pem'
from celery.security import setup_security
setup_security()

As far as what that stuff means… and how it works, see: http://www.tldp.org/HOWTO/SSL-Certificates-HOWTO/x64.html

Also, for how to generate keys (and enable secure passwordless logins), see: https://help.github.com/articles/generating-ssh-keys/ or http://mah.everybody.org/docs/ssh for more general links referenced therein.