celery: "Substantial drift from"

Question

I have quite a problem with celery on my distribted system. I have couple of machines among different localizations and I've got a lot of warnings in my log files like:

"Substantial drift from celery@host [...]"

I was able to set date to return the same values (even that the machines are in different countries) but python print(utcoffset()) returns different results on main server and nodes.

How to fix that issue? I was unable to find any good solution except that utcoffset() should return the same value - but how?

Thanks for help.

Answer 1

I encountered a similar issue while starting flower:

[W 161208 08:42:00 state:74] Substantial drift from [email protected] may mean clocks are out of sync.  Current drift is 10800 seconds.  [orig: 2016-12-08 08:42:00.722560 recv: 2016-12-08 05:42:00.719938]

versions used:

celery==3.1.23 (and 3.1.25)
flower==0.9.1
rabbitmq 3.5.6

with settings:

CELERY_TIMEZONE='US/Pacific'
TIME_ZONE = 'US/Pacific'
USE_TZ = True

Looked through all available issue reports and suggestions on stackoverflow and git, but all indicated fixes that I should already have in place with > 3.1.23

This is how I solved it

My system and subsequent bash environment was running EST for timezone. As you can see above, the django app runs 'US/Pacific' (PST). This is is a 3 hour difference, or 10800 seconds.

By setting the bash timezone to PST as well, to match the django app, the drift error went away:

export TZ="US/Pacific"