Using Celery Chunks with sequences of objects or integers

Question

When using task.chunks with a sequence of sequences (e.g: list of strings)

my_task.chunks(['a', 'b', 'c', 'd'], 2).delay() 

things work fine. However, passing a sequence of anything else (e.g: integers or objects)

my_task.chunks([1, 2, 3, 4], 2).delay()

raises this exception

[2014-06-22 16:50:27,970: ERROR/MainProcess] Task celery.starmap[44b20b4c-ef69-4c68-bd9c-e1de42c34c49] raised unexpected: TypeError('my_task object argument after * must be a sequence, not int',)
Traceback (most recent call last):
  File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/venv/local/lib/python2.7/site-packages/celery/app/trace.py", line 437, in protected_call
    return self.run(*args, **kwargs)
  File "/venv/local/lib/python2.7/site-packages/celery/app/builtins.py", line 125, in xstarmap
    return [task(*item) for item in it]
TypeError: my_task object argument after * must be a sequence, not int

my_task is currently trivial

@shared_task
def my_task(word):
    print word

How can I use task.chunks with a sequence of non-sequences ?

Answer 1

Change it to:

my_task.chunks([(1,), (2,), (3,), (4,)], 2).delay()

As the error trace complains, each argument in your iterable is expected to be a sequence, which can be unpacked into arguments of the function you want to call.

How it comes, the ['a', 'b', 'c', 'd'] works?

It is just a coincidence resulting from the fact, string is also iterable. So "a" is turned into ("a", ) sequence.

If you try to use any value with more then one character, you will get another error as your task expects just one parameter.

You shall change it to:

my_task.chunks([('a',), ('b',), ('c',), ('d',)], 2).delay() 

If you need to turn your existing lists into lists of tuples:

>>> nums = [1, 2, 3, 4]
>>> [(num, ) for num in nums]
[(1,), (2,), (3,), (4,)]
>>> chars = ["a", "b", "c", "d"]
>>> [(c,) for c in chars]
[('a',), ('b',), ('c',), ('d',)]