Max. number of parallel background threads (backend)

Question

How many parallel background threads can I start in one google app engine backend? I didn't find any information to the amount of parallel allowed threads. I'm using Java for the GAE.

I start a new thread as explained in the docs: [1]

return ThreadManager.createBackgroundThread(new Runnable() { ... });

If I run my application, after a while the following exception is thrown (at the creation of a new thread):

com.google.appengine.api.system.SystemFailureException: An unknown error occurred

This [2] issue mentioned that this exception appears if the API is out of quota. So i can create threads, but after a certain time the exception appears. That's why i think there is a limit of threads in a backend.

[1] https://developers.google.com/appengine/docs/java/backends/overview#background_threads

[2] http://code.google.com/p/googleappengine/issues/detail?id=7398

Answer 1

For the record, App Engine distinguishes between normal threads, which can't outlive the HTTP request that started them, and background threads, which can.

For Python, at least, it looks like the production (Python 2.7) runtime and dev_appserver both impose a fixed limit of 10 background threads per backend, independent of other settings like e.g. max_concurrent_requests in backends.yaml.

I've talked with a few other old App Engine team members, and while they weren't 100% sure, they said this sounded right. I've confirmed empirically with the test backend config and code below. I also tried starting more background threads from a separate HTTP request, and from yet another background thread. No luck; same overall limit of 10 total.

Here's where the SDK sets that limit in dev_appserver (specifically devappserver2 in SDK 1.8.8) for Python: instance.py:449, python_runtime.py:61. It looks like background threads are disabled altogether for Go and Java: go_runtime.py:99, java_runtime.py:61.

One interesting quirk: inside a background thread, it looks like you can start as many normal threads as you want, at least until you hit the memory limit. They don't hold any HTTP request open, and they don't seem to get cut off by a deadline either. I'm currently doing this to work around the background thread limit.

I wish this was documented! Would have saved me a lot of time.

---

backends.yaml

- name: test
  instances: 1
  start: threadtest.application

test.py

def test():
  for i in range(100):
    logging.info('Starting #%d', i)
    background_thread.start_new_background_thread(time.sleep, [20])

class Start(webapp2.RequestHandler):
  def get(self):
    background_thread.start_new_background_thread(test, [])

application = webapp2.WSGIApplication([('/_ah/start', Start)], debug=True)

Answer 2

My problem was the API limit of the backends. I could start as many threads as I want, but when I reached the API limit (100 simultanous calls) the creation of new threads was not possible. Quotas and limits are described in their docs.

I reduced my API calls (database, google cloud storage) to speed up the performance of the application and the exception is gone :) .