Mongodb replica set auto reconect don't work after down and up for nginx + uwsgi with several processes

Question

Hi everyone I have next envirement for python2.7.5:

flask==0.10.1
flask-wtf==0.8.4
jinja2==2.7
werkzeug==0.9.1
flask-mongoengine==0.7.0
mongoengine==0.8.2
pymongo==2.5.2
uwsgi==1.9.13

and have next application 'app.py':

from flask import Flask
from flask.ext.mongoengine import Document, MongoEngine
from mongoengine import StringField  

class Config(object):
    DEBUG = True
    MONGODB_HOST = ('mongodb://localhost:27017,localhost:27018/'
                    'test?replicaSet=rs0')
    MONGODB_DB = True

app = Flask(__name__)
app.config.from_object(Config)
MongoEngine(app)

class Test(Document):
    test = StringField(default='test')

    meta = {
        'allow_inheritance': False,
    }

    def __unicode__(self):
        return self.test

Test(test='test1').save()

@app.route('/')
def hello_world():
    return unicode(Test.objects.first())

if __name__ == '__main__':
    app.run('0.0.0.0', 8080, True)

I have next nginx config:

server {
    listen       80;
    server_name  localhost;
    location / {
        include uwsgi_params;
        uwsgi_pass unix:/tmp/uwsgi.sock;
    }

}

I start uwsgi as:

/path/to/env/bin/uwsgi \
  --module app:app \
  --env /path/to/env/ \
  --pythonpath /path/to/app/ \
  --socket /tmp/uwsgi.sock \
  --pidfile /tmp/uwsgi.pid \
  --daemonize /tmp/uwsgi.log \
  --processes 2 \
  --threads 2 \
  --master

I have two mongodb instances:

mongod --port 27017 --dbpath /path/to/mongo/data/rs0-0 --replSet rs0 \
  --smallfiles --oplogSize 128

and

mongod --port 27018 --dbpath /path/to/mongo/data/rs0-1 --replSet rs0 \
  --smallfiles --oplogSize 128

And configured replica set in mongo console as:

rsconf = {
    _id: "rs0",
    members: [{_id: 0, host: "127.0.0.1:27017"}]
};
rs.initiate(rsconf);
rs.add("127.0.0.1:27018");

So it's working good. But when I down and up primary or secondary mongo instance then my application can't restore connection and I have next exceptions every time after:

...
  File "/path/to/app/replica.py", line 33, in hello_world
    return unicode(Test.objects.first())
  File "/path/to/env/local/lib/python2.7/site-packages/mongoengine/queryset/queryset.py", line 325, in first
    result = queryset[0]
  File "/path/to/env/local/lib/python2.7/site-packages/mongoengine/queryset/queryset.py", line 211, in __getitem__
    return queryset._document._from_son(queryset._cursor[key],
  File "/path/to/env/local/lib/python2.7/site-packages/pymongo/cursor.py", line 470, in __getitem__
    for doc in clone:
  File "/path/to/env/local/lib/python2.7/site-packages/pymongo/cursor.py", line 814, in next
    if len(self.__data) or self._refresh():
  File "/path/to/env/local/lib/python2.7/site-packages/pymongo/cursor.py", line 763, in _refresh
    self.__uuid_subtype))
  File "/path/to/env/local/lib/python2.7/site-packages/pymongo/cursor.py", line 700, in __send_message
    **kwargs)
  File "/path/to/env/local/lib/python2.7/site-packages/pymongo/mongo_replica_set_client.py", line 1546, in _send_message_with_response
    raise AutoReconnect(msg, errors)
pymongo.errors.AutoReconnect: No replica set primary available for query with ReadPreference PRIMARY

When I use mongoengie==0.7.10 which use ReplicaSetConnection instead MongoReplicaSetClient in mongoengine==0.8.2 then I have next exceptions:

1. Down secondary, get requests, up secondary, get requests:

   I have at first time:

   pymongo.errors.AutoReconnect: 127.0.0.1:27017: [Errno 104] Connection reset by peer
   

   after:

   pymongo.errors.AutoReconnect: No replica set primary available for query with ReadPreference PRIMARY
   

2. Down primary, get requests, up primary, get requests:

   I have at first time:

   pymongo.errors.AutoReconnect: 127.0.0.1:27017: [Errno 111] Connection refused
   

   after:

   pymongo.errors.AutoReconnect: No replica set primary available for query with ReadPreference PRIMARY
   

3. Down primary or secondary, up primary or secondary, get requests I have always:

   pymongo.errors.AutoReconnect: not master and slaveOk=false
   

So two mongo instances just simple example. If I add one more instance (total 3), then:

1. If I down any secondary then all work fine. If I down and up one secondary and then down or down and up second secondary then all work fine too.

2. If I down and up two secondaries - some problem.

3. If I down and up primary or just down one primary (two secondaries availible) - some problem, despite that mongo elect new primary!!!

If I start one uwsgi process (without --master for two or three mongo instances) as:

/path/to/env/bin/uwsgi \
  --module app:app \
  --env /path/to/env/ \
  --pythonpath /path/to/app/ \
  --socket /tmp/uwsgi.sock \
  --pidfile /tmp/uwsgi.pid \
  --daemonize /tmp/uwsgi.log \
  --processes 1 \
  --threads 2

or run application with dev server:

/path/to/env/bin/python app.py

then application restore connection without problems after down and up mongo instances.

I have some deployment in production and sometimes connection to mongo instances can disapire (to several seconds). After that my application don't work normal before uwsgi will be restarted.

So I have two questions:

1. Why it's happenes with several uwsgi process?
2. How fix normal application work after down and up mongo instances?

---

UPD1: I trying understand problem and now I have different behaviour for self.__schedule_refresh() when get connection exception when I down one mongo node:

1. For one process:

   1. Before this statement: rs_state have two members: active with up == True, downed with up == False.
   2. After this statement: rs_state have one active member with up == True.

2. For two processes:

   1. Before this statement: rs_state have two members: active with up == True, downed with up == False.
   2. After this statement: rs_state have two members: active with up == True, downed with up == False (no changed).

When I up mongo node then self.__schedule_refresh(sync=sync) also have different behaviour:

1. For one process:

   1. Before this statement: rs_state have one active member with up == True.
   2. After this statement: rs_state have two members active with up == True, upped with up == True.

2. For two processes:

   1. Before this statement: rs_state have two members active with up == True, upped with up == False.
   2. After this statement: rs_state have two members active with up == True, upped with up == False (no changed).

So look like mongo can't update replica set state (see __schedule_refresh):

def __schedule_refresh(self, sync=False):
    """Awake the monitor to update our view of the replica set's state.

    If `sync` is True, block until the refresh completes.

    If multiple application threads call __schedule_refresh while refresh
    is in progress, the work of refreshing the state is only performed
    once.
    """
    self.__monitor.schedule_refresh()
    if sync:
        self.__monitor.wait_for_refresh(timeout_seconds=5)

Answer 1

Try using the uwsgi --lazy-apps option. MongoReplicaSetClient spawns a replicaset MonitorThread, and this thread does not survive the uwsgi worker process fork. --lazy-apps will initialize the pymongo MonitorThread in each worker process.