Daemonizing python's BaseHTTPServer

Question

I am working on a daemon where I need to embed a HTTP server. I am attempting to do it with BaseHTTPServer, which when I run it in the foreground, it works fine, but when I try and fork the daemon into the background, it stops working. My main application continues to work, but BaseHTTPServer does not.

I believe this has something to do with the fact that BaseHTTPServer sends log data to STDOUT and STDERR. I am redirecting those to files. Here is the code snippet:

# Start the HTTP Server
server = HTTPServer((config['HTTPServer']['listen'],config['HTTPServer']['port']),HTTPHandler)

# Fork our process to detach if not told to stay in foreground
if options.foreground is False:
    try:
        pid = os.fork()
        if pid > 0:
            logging.info('Parent process ending.')
            sys.exit(0)            
    except OSError, e:
        sys.stderr.write("Could not fork: %d (%s)\n" % (e.errno, e.strerror))
        sys.exit(1)

    # Second fork to put into daemon mode
    try: 
        pid = os.fork() 
        if pid > 0:
            # exit from second parent, print eventual PID before
            print 'Daemon has started - PID # %d.' % pid
            logging.info('Child forked as PID # %d' % pid)
            sys.exit(0) 
    except OSError, e: 
        sys.stderr.write("Could not fork: %d (%s)\n" % (e.errno, e.strerror))
        sys.exit(1)


    logging.debug('After child fork')

    # Detach from parent environment
    os.chdir('/') 
    os.setsid()
    os.umask(0) 

    # Close stdin       
    sys.stdin.close()

    # Redirect stdout, stderr
    sys.stdout = open('http_access.log', 'w')
    sys.stderr = open('http_errors.log', 'w')    

# Main Thread Object for Stats
threads = []

logging.debug('Kicking off threads')

while ...
  lots of code here
...

server.serve_forever()

Am I doing something wrong here or is BaseHTTPServer somehow prevented from becoming daemonized?

Edit: Updated code to demonstrate the additional, previously missing code flow and that log.debug shows in my forked, background daemon I am hitting code after fork.

Answer 1

After a bit of googling I finally stumbled over this BaseHTTPServer documentation and after that I ended up with:

from BaseHTTPServer import BaseHTTPRequestHandler, HTTPServer
from SocketServer import ThreadingMixIn

class ThreadedHTTPServer(ThreadingMixIn, HTTPServer):
  """Handle requests in a separate thread."""

server = ThreadedHTTPServer((config['HTTPServer']['listen'],config['HTTPServer']['port']), HTTPHandler)
server.serve_forever()

Which for the most part comes after I fork and ended up resolving my problem.

Answer 2

Here's how to do this with the python-daemon library:

from BaseHTTPServer import (HTTPServer, BaseHTTPRequestHandler)
import contextlib

import daemon

from my_app_config import config

# Make the HTTP Server instance.
server = HTTPServer(
    (config['HTTPServer']['listen'], config['HTTPServer']['port']),
    BaseHTTPRequestHandler)

# Make the context manager for becoming a daemon process.
daemon_context = daemon.DaemonContext()
daemon_context.files_preserve = [server.fileno()]

# Become a daemon process.
with daemon_context:
    server.serve_forever()

As usual for a daemon, you need to decide how you will interact with the program after it becomes a daemon. For example, you might register a systemd service, or write a PID file, etc. That's all outside the scope of the question though.

In particular, it's outside the scope of the question to ask: once it's become a daemon process (necessarily detached from any controlling terminal), how do I stop the daemon process? That's up to you to decide, as part of defining the program's behaviour.