CherryPy with additional threads for custom jobs

Question

We're designing a system based on CherryPy that in addition to serving web requests needs to do tasks/jobs in parallel. We want it to be a single process running as a daemon and create threads for all the parallel jobs like scheduled tasks or collecting data online.

I've been browsing through the CherryPy documentation and know that it's thread-pooled creating threads for all user requests. However I can't seem to find documentation on how to create and manage threads for custom jobs. Does CherryPy have a thread-handler that we can hook in to or could/should we write our own handler that hooks into CherryPy?

Answer 1

Subscribe a Monitor instance:

from cherrypy.process.plugins import Monitor

def foo():
    my.store.collect_data('things', 'stuff')

Monitor(cherrypy.engine, foo, frequency=300).subscribe()

This will run the foo function every 300 seconds in its own thread, and that thread will start when you call engine.start and stop when you call engine.stop (or at process exit).

Answer 2

I would like to complete fumanchu's excellent answer by sharing the script I wrote for testing this before using it. It uses python bottle, proving the advice is also working with it.

#!/usr/bin/env python3

import bottle
import time
import threading
import cherrypy
from cherrypy.process.plugins import Monitor

srv = bottle.Bottle()
lock = threading.Lock()
x = 0
y = 0

def increment ():
    print("incrementing...")
    with lock:
        global x
        time.sleep(1)
        x += 1
    print("Done.")

monitor = Monitor(cherrypy.engine, increment, frequency=1)

@srv.get("/")
def read ():
    with lock:
        return "Got: %d %d." % (x, y)

@srv.post("/periodic/<x:int>")
def periodic (x):
    if x: monitor.start()
    else: monitor.stop()

@srv.put("/<V:int>")
def write (V):
    print("Serving")
    with lock:
        global x
        global y
        x = V
        time.sleep(5)
        y = V
        print("gtfo")
    return "OK. Current: %d." % x

srv.run(server='cherrypy')

Usage:

After enabling the server, use curl http://localhost:8080 to read, curl http://localhost:8080/<value> to write some value (takes 5 seconds, while all the reads will be hanging), and finally curl http://localhost:8080/periodic/0 and curl http://localhost:8080/periodic/1 to disable/enable respectively the periodic write. Each write will take 1 second, during which reads and modifications will be hanging.

P.s. Working with python2 and python3