Flask app unable to query data from mongodb using mongoengine in a dockerized setup

Question

I am running a website application and am using Flask which queries a mongodb database using mongoengine. Am getting an error when the application tries to query the database.

I have mongodb installed in one of the containers and I can connect to it successfully.

mongodb: 2019-09-06T03:45:11.801+0000 I  NETWORK  [conn1] received client metadata from 172.20.0.3:43918 conn1: { driver: { name: "PyMongo", version: "3.9.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.10.0-957.27.2.el7.x86_64" }, platform: "CPython 3.7.2.final.0" }

Docker Setup

version: "3.7"

services:

  portfolio:
    build: ./portfolio
    container_name: portfolio
    restart: always
    environment:
      - APP_NAME=portfolio
    expose:
      - 8080
    links:
      - mongodb

  admin:
    build: ./admin
    container_name: admin
    restart: always
    environment:
      - APP_NAME=admin
    expose:
      - 8089
    links:
      - mongodb

  mongodb:
    image: mongo
    container_name: mongodb
    ports:
      - "27017:27017"

  nginx:
    build: ./nginx
    depends_on:
      - portfolio
      - admin
    container_name: nginx
    restart: always
    ports:
      - "80:80"

Python setup:

from flask import Flask
from mongoengine import connect
app = Flask(__name__)
connect('spees_db', host='mongodb://mongodb:27017/spees_db')

Error occurs when I do

email_ex = User.objects(email=email).first()

Error stack:

nginx        | 102.140.206.140 - - [06/Sep/2019:03:53:09 +0000] "GET /register HTTP/1.1" 200 4145 "http://admin.shemoirere.com/login?next=%2F" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" "-"
 [pid: 13|app: 0|req: 4/5] 102.140.206.140 () {44 vars in 967 bytes} [Fri Sep  6 03:53:09 2019] GET /register => generated 4145 bytes in 10 msecs (HTTP/1.1 200) 4 headers in 192 bytes (1 switches on core 1)
 Traceback (most recent call last):
   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2463, in __call__
     return self.wsgi_app(environ, start_response)
   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2449, in wsgi_app
     response = self.handle_exception(e)
   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1866, in handle_exception
     reraise(exc_type, exc_value, tb)
   File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
     raise value
   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2446, in wsgi_app
     response = self.full_dispatch_request()
   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1951, in full_dispatch_request
     rv = self.handle_user_exception(e)
   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1820, in handle_user_exception
     reraise(exc_type, exc_value, tb)
   File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 39, in reraise
     raise value
   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1949, in full_dispatch_request
     rv = self.dispatch_request()
   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1935, in dispatch_request
     return self.view_functions[rule.endpoint](**req.view_args)
   File "./admin/views.py", line 37, in do_register
     email_ex = User.objects(email=email).first()
   File "/usr/local/lib/python3.7/site-packages/mongoengine/queryset/manager.py", line 37, in __get__
     queryset = queryset_class(owner, owner._get_collection())
   File "/usr/local/lib/python3.7/site-packages/mongoengine/document.py", line 207, in _get_collection
     db.client.is_primary:
   File "/usr/local/lib/python3.7/site-packages/pymongo/mongo_client.py", line 1006, in is_primary
     return self._server_property('is_writable')
   File "/usr/local/lib/python3.7/site-packages/pymongo/mongo_client.py", line 831, in _server_property
     writable_server_selector)
   File "/usr/local/lib/python3.7/site-packages/pymongo/topology.py", line 231, in select_server
nginx        | 2019/09/06 03:53:50 [error] 6#6: *5 upstream prematurely closed connection while reading response header from upstream, client: 102.140.206.140, server: admin.shemoirere.com, request: "POST /register HTTP/1.1", upstream: "uwsgi://172.20.0.4:8089", host: "admin.shemoirere.com", referrer: "http://admin.shemoirere.com/register"
nginx        | 102.140.206.140 - - [06/Sep/2019:03:53:50 +0000] "POST /register HTTP/1.1" 502 559 "http://admin.shemoirere.com/register" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36" "-"
     address))
   File "/usr/local/lib/python3.7/site-packages/pymongo/topology.py", line 189, in select_servers
     selector, server_timeout, address)
   File "/usr/local/lib/python3.7/site-packages/pymongo/topology.py", line 205, in _select_servers_loop
     self._error_message(selector))
 pymongo.errors.ServerSelectionTimeoutError: No servers found yet

Answer 1

I played around with your repo and figured out your issue. This is your wsgi config-

[uwsgi] wsgi-file = run.py callable = app socket = :8089 processes = 4 threads = 2 master = true chmod-socket = 660 vacuum = true die-on-term = true

The problem is in the processes argument. uwsgi launches a master process and then uses fork() to spawn workers. You were doing your connect during the initial startup- so the connection lived in your master process, while your children were getting a bad copy. You can actually see the log on mongo when the connection happens... just 1! pymongo specifically calls out fork() as being unsafe with a connection.

The easiest fix for this is to add lazy-apps = true to the end of that wsgi file. The system works as intended then. You can also set up your app to avoid actually calling connect until the first time it is needed (probably the before first request hook?)

https://api.mongodb.com/python/3.9.0/faq.html#is-pymongo-fork-safe