Python Reddis Queue ValueError: Functions from the __main__ module cannot be processed by workers

Question

I'm trying to enqueue a basic job in redis using python-rq, But it throws this error

"ValueError: Functions from the main module cannot be processed by workers"

Here is my program:

import requests

def count_words_at_url(url):
    resp = requests.get(url)
    return len(resp.text.split())

from rq import Connection, Queue
from redis import Redis
redis_conn = Redis()
q = Queue(connection=redis_conn)
job = q.enqueue(count_words_at_url, 'http://nvie.com')
print job

Answer 1

Break the provided code to two files: count_words.py:

import requests

def count_words_at_url(url):
    resp = requests.get(url)
    return len(resp.text.split())

and main.py (where you'll import the required function):

from rq import Connection, Queue
from redis import Redis
from count_words import count_words_at_url # added import!
redis_conn = Redis()
q = Queue(connection=redis_conn)
job = q.enqueue(count_words_at_url, 'http://nvie.com')
print job

I always separate the tasks from the logic running those tasks to different files. It's just better organization. Also note that you can define a class of tasks and import/schedule tasks from that class instead of the (over-simplified) structure I suggest above. This should get you going.. Also see here to confirm you're not the first to struggle with this example. RQ is great once you get the hang of it.

Answer 2

Currently there is a bug in RQ, which leads to this error. You will not be able to pass functions in enqueue from the same file without explicitly importing it.

Just add from app import count_words_at_url above the enqueue function:

import requests

def count_words_at_url(url):
    resp = requests.get(url)
    return len(resp.text.split())

from rq import Connection, Queue
from redis import Redis
redis_conn = Redis()
q = Queue(connection=redis_conn)

from app import count_words_at_url
job = q.enqueue(count_words_at_url, 'http://nvie.com')
print job

The other way is to have the functions in a separate file and import them.