Deferred tasks creates new instances that can't access some python modules

Question

I am using the latest version of GAE with automated scaling, endpoints API, and deferred.defer() tasks.

The problem is that since adding the API, there have been some instances that will spin up automatically that always throw permanent task failures:

Permanent failure attempting to execute task
Traceback (most recent call last):
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 310, in post
    self.run_from_request()
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 305, in run_from_request
    run(self.request.body)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 145, in run
    raise PermanentTaskFailure(e)
PermanentTaskFailure: No module named app.Report

The permanent task failures are unique for a single instance though, in which every deferred tasks on that instance fail. These deferred tasks all throw the same error, even though the tasks aren't using the Api.py module. On other instances, the same deferred tasks will run just fine if they aren't routed to a failing instance.

The app.yaml handlers looks like this:

handlers:
# Api Handler
- url: /_ah/api/.*
  script: main.api
- url: /_ah/spi/.*
  script: main.api
# All other traffic
- url: .*
  script: main.app

builtins:
- deferred: on

The main.py looks like:

import Api, endpoints, webapp2

api = endpoints.api_server([Api.AppApi])

app = webapp2.WSGIApplication( 
    [(misc routes)]
,debug=True)

The Api.py looks like :

import endpoints
from protorpc import messages
from protorpc import message_types
from protorpc import remote
from google.appengine.ext import deferred
from app.Report import ETLScheduler

@endpoints.api(...)
class AppApi(remote.Service):
    @endpoints.method(...)
    def reportExtract(self, request):
        deferred.defer(
            ETLScheduler,
            params
        )

I'm not doing any path modification, so I'm curious why the new instance is having trouble finding the python modules for the API, even though the deferred tasks are in another module using other functions. Why would it throw these errors for that instance only?

Edit:

So after looking at some other SO issues, I tried doing path modification in appengine_config.py. I moved all my folders to a lib directory, and added this to the config file:

import os,sys
sys.path.append(os.path.join(os.path.dirname(__file__), 'lib'))

Now the error I get on the failing instance is:

Permanent failure attempting to execute task
Traceback (most recent call last):
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 310, in post
    self.run_from_request()
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 305, in run_from_request
    run(self.request.body)
  File "/base/data/home/runtimes/python27/python27_lib/versions/1/google/appengine/ext/deferred/deferred.py", line 145, in run
    raise PermanentTaskFailure(e)
PermanentTaskFailure: cannot import name ETLScheduler

So it seems to be finding the module, but same as before, none of the deferred tasks on the instance can import the method.

Answer 1

So I figured out a way to make it work, but am not sure why it works.

By importing the entire module, rather than a method from the module, the new instances that spin up for deferred tasks no longer throw the PermanentTaskFailure: cannot import name ETLScheduler error.

I tried importing the whole module instead of the method, so that the Api.py looks like this:

import endpoints
from protorpc import messages
from protorpc import message_types
from protorpc import remote
from google.appengine.ext import deferred

# Import the module instead of the method
#from app.Report import ETLScheduler
import app.Report

@endpoints.api(...)
class AppApi(remote.Service):
    @endpoints.method(...)
    def reportExtract(self, request):
        deferred.defer(
            app.Report.ETLScheduler,
            params
        )

Now I am no longer getting instances that throw the PermanentTaskFailure: cannot import name ETLScheduler. Might be a circular dependency by import Api.py in main.py (I'm not sure) but at least it works now.