Flask Redis Queue (RQ) worker cannot import module named app

Question

I've a flask application running called modeling-manager, it's receiving a post request from another flask application. 'till there everything works fine. (all files are located in the same folder)

Modeling-manager when triggered with post request registers a job in Redis for execution by the worker. The job to execute is called run_model() which is stored in a different file and importer in the main.py (flask app)

the worker is launched on the same docker as the modeling-manager app. using script : worker.py

main.py -> flask app

from flask import Flask, request, jsonify
import logging as lg
from datetime import datetime as dt
import os
import boto3
import redis
import json
from rq import Worker, Queue, Connection
from .model import run_model

app = Flask(__name__)

@app.route('/ModelManager/register',  methods=['POST'])
def register_philjob_to_queue():
    lg.info("triggered register cahce on model manager")
    job_data = request.get_json()
    lg.info("json data " + job_data["InstanceNumber"])
    try:
        with Connection(redis.Redis(host="redis-caching", port="6379")):
            lg.info("adding job to queue")
            q = Queue()
            task = q.enqueue(run_model, job_data)
        response_object = {
            'status': 'pending',
            'data': {
                'task_id': task.get_id()
            }
        }
        return jsonify(response_object), 202
        except Exception as e:
          lg.error(e)

worker.py

import redis
from rq import Worker, Queue, Connection

conn = redis.Redis(host="redis-caching", port="6379")
listen = ['default']

if __name__ == '__main__':
    with Connection(conn):
        worker = Worker(list(map(Queue, listen)))
        worker.work()

finally model.py

import logging as lg
def run_model(job_data):

    # Run phil on started instance
    try:

        lg.info("executing the model") 
        return 'Model Executed'

    except Exception as e:
        print(e)
        print("RUN_MODEL:")
        lg.error(e)

Error message I get from the worker, where "app" probably refers to the root folder of my flask application (named app) :

10:35:15 RQ worker 'rq:worker:7555cdf9c1ad.9' started, version 0.12.0
10:35:15 *** Listening on default...
10:35:15 Cleaning registries for queue: default
10:35:15 default: app.model.run_model({'InstanceNumber': '002365984', 
'CallBackUrl': 'http://blabla.be'}) (5e83d96b-de11-4ddb-9c8f- 
b7b256580bfc)
10:35:15 ModuleNotFoundError: No module named 'app'
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/rq/worker.py", line 
793, in perform_job
    rv = job.perform()
  File "/usr/local/lib/python3.7/site-packages/rq/job.py", line 599, in perform
    self._result = self._execute()
  File "/usr/local/lib/python3.7/site-packages/rq/job.py", line 605, in _execute
return self.func(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.7/site-packages/rq/job.py", line 213, 
in func
return import_attribute(self.func_name)
  File "/usr/local/lib/python3.7/site-packages/rq/utils.py", line 152, in import_attribute
    module = importlib.import_module(module_name)
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in 
import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in 
_find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'app'
Traceback (most recent call last):
 File "/usr/local/lib/python3.7/site-packages/rq/worker.py", line 793, in perform_job
rv = job.perform()
 File "/usr/local/lib/python3.7/site-packages/rq/job.py", line 599, in perform
self._result = self._execute()
  File "/usr/local/lib/python3.7/site-packages/rq/job.py", line 605, in _execute
return self.func(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.7/site-packages/rq/job.py", line 213, in func 
return import_attribute(self.func_name)
  File "/usr/local/lib/python3.7/site-packages/rq/utils.py", line 152, in import_attribute
module = importlib.import_module(module_name)
  File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
  ModuleNotFoundError: No module named 'app'   
10:35:15 Moving job to 'failed' queue

this is all executed on a docker container, redis is on a different container. Communication between those 2 containers works fine.

flask application is launched on docker using command : flask run --host=0.0.0.0 --port=8081

Answer 1

I'm running a flask application and walked against the same issue here. The solution above did not work for me. However my issue was related to a incorrect import.

When debugging on windows in terminal my redis server is running in cygwin. I just started my worker with my static project path. Now my import error is solved. Before:

rq worker flask-task

After:

rq worker flask-task --path /mnt/c/Users/<USER>/Desktop/projects/flask_app

Answer 2

Found a solution, issue is due to the relative path I use for my module "model" (.model) it was only working that way because of a init.py file in the root directory. So I've deleted the init.py file, change the import to

from model import run_model

And now everything works fine, the function can be found by the worker and is being executed correctly !