MongoDB databases disappeared

Question

My entire mongo database seems to be gone (this is for my thesis which is due in a month so kind of desperate here).

I'm running some code fetching tweets (it's been working for a month without any edits) and writing to mongo using pymongo and I get this error:

Traceback (most recent call last):
   File "linguistic_feats.py", line 100, in <module>
write_all_user_tweets(conn, dataset)
   File "linguistic_feats.py", line 89, in write_all_user_tweets
[get_all_tweets(user, conn) for user in parse]
   File "linguistic_feats.py", line 89, in <listcomp>
[get_all_tweets(user, conn) for user in parse]
   File "/Users/zeerakw/Documents/Education/Master/Projects/Gender/collect.py", line 58, in get_all_tweets
[conn.write_user_collection(tweet) for tweet in  all_tweets]
   File "/Users/zeerakw/Documents/Education/Master/Projects/Gender/collect.py", line 58, in <listcomp>
[conn.write_user_collection(tweet) for tweet in  all_tweets]
   File "/Users/zeerakw/Documents/Education/Master/Projects/Gender/db.py", line 49, in write_user_collection
table.update({'_id': tweet['id_str']}, {'$setOnInsert': tweet}, upsert = True)
   File "/usr/local/lib/python3.5/site-packages/pymongo/collection.py", line 2230, in update
with self._socket_for_writes() as sock_info:
   File "/usr/local/Cellar/python3/3.5.0/Frameworks/Python.framework/Versions/3.5/lib/python3.5/contextlib.py", line 59, in __enter__
return next(self.gen)
   File "/usr/local/lib/python3.5/site-packages/pymongo/mongo_client.py", line 712, in _get_socket
server = self._get_topology().select_server(selector)
   File "/usr/local/lib/python3.5/site-packages/pymongo/topology.py", line 141, in select_server
address))
   File "/usr/local/lib/python3.5/site-packages/pymongo/topology.py", line 117, in select_servers
self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: localhost:27017: [Errno 61] Connection refused

I tried to access the mongo console but receive:

MongoDB shell version: 3.2.0
connecting to: test
2016-06-24T03:02:05.622+0200 W NETWORK  [thread1] Failed to connect to 127.0.0.1:27017 after 5000 milliseconds, giving up.
2016-06-24T03:02:05.623+0200 E QUERY    [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
connect@src/mongo/shell/mongo.js:224:14
@(connect):1:6

exception: connect failed

Then I called mongod manually, which allowed me to access the console but now instead of having multiple show dbs simply gives:

$ mongo
MongoDB shell version: 3.2.0
connecting to: test
Server has startup warnings:
2016-06-24T03:07:43.638+0200 I CONTROL  [initandlisten]
2016-06-24T03:07:43.638+0200 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
> use thesis 
> show dbs
local  0.000GB
> show collections
>

Log file:

2016-06-24T01:17:51.074+0200 E STORAGE  [thread1] WiredTiger (24) [1466723871:73125][473:0x70000020a000], log-server: /usr/local/var/mongodb/journal: opendir: Too many open files
2016-06-24T01:17:51.074+0200 E STORAGE  [thread1] WiredTiger (24) [1466723871:74536][473:0x70000020a000], log-server: dirlist journal prefix WiredTigerPreplog: Too many open files
2016-06-24T01:17:51.074+0200 E STORAGE  [thread1] WiredTiger (24) [1466723871:74568][473:0x70000020a000], log-server: log pre-alloc server error: Too many open files
 2016-06-24T01:17:51.074+0200 E STORAGE  [thread1] WiredTiger (24) [1466723871:74588][473:0x70000020a000], log-server: log server error: Too many open files
 2016-06-24T01:17:51.078+0200 W FTDC     [ftdc] Uncaught exception in 'FileNotOpen Failed to open interim file /usr/local/var/mongodb/diagnostic.data/metrics.interim.temp' in full-time diagnostic data capture subsystem. Shutting down the full-time diagnostic data capture subsystem.
 2016-06-24T01:18:14.124+0200 E STORAGE  [thread2] WiredTiger (24) [1466723894:124125][473:0x700000393000], file:WiredTiger.wt, WT_SESSION.checkpoint: WiredTiger.turtle: fopen: Too many open files
 2016-06-24T01:18:14.127+0200 E STORAGE  [thread2] WiredTiger (24) [1466723894:126992][473:0x700000393000], checkpoint-server: checkpoint server error: Too many open files
 2016-06-24T01:18:14.127+0200 E STORAGE  [thread2] WiredTiger (-31804) [1466723894:127084][473:0x700000393000], checkpoint-server: the process must exit and restart: WT_PANIC: WiredTiger library panic
2016-06-24T01:18:14.127+0200 I -        [thread2] Fatal Assertion 28558
 2016-06-24T01:18:14.127+0200 I -        [thread2] 

***aborting after fassert() failure


2016-06-24T01:18:14.163+0200 F -        [thread2] Got signal: 6 (Abort trap: 6).

0x10788d449 0x10788cddf 0x7fff8deb952a 0x7fff94e7ba26 0x7fff9ce4e6e7 0x10782d53a 0x1076679a1 0x10800e381 0x10800e4c9 0x10800ea74 0x107f9ac5d 0x7fff98d5199d 0x7fff98d5191a 0x7fff98d4f351
----- BEGIN BACKTRACE -----
{
  "backtrace": [
    {
      "b": "106FA5000",
  "o": "8E8449"
},
{
  "b": "106FA5000",
  "o": "8E7DDF"
},
{
  "b": "7FFF8DEB7000",
  "o": "252A"
},
{
  "b": "7FFF94E79000",
  "o": "2A26"
},
{
  "b": "7FFF9CDF0000",
  "o": "5E6E7"
},
{
  "b": "106FA5000",
  "o": "88853A"
},
{
  "b": "106FA5000",
  "o": "6C29A1"
},
{
  "b": "106FA5000",
  "o": "1069381"
},
{
  "b": "106FA5000",
  "o": "10694C9"
},
{
  "b": "106FA5000",
  "o": "1069A74"
},
{
  "b": "106FA5000",
  "o": "FF5C5D"
},
{
  "b": "7FFF98D4E000",
  "o": "399D"
},
{
  "b": "7FFF98D4E000",
  "o": "391A"
},
{
  "b": "7FFF98D4E000",
  "o": "1351"
}
  ],
  "processInfo": {
"mongodbVersion": "3.2.0",
"gitVersion": "45d947729a0315accb6d4f15a6b06be6d9c19fe7",
"compiledModules": [

],
"uname": {
  "sysname": "Darwin",
  "release": "15.5.0",
  "version": "Darwin Kernel Version 15.5.0: Tue Apr 19 18:36:36 PDT 2016; root:xnu-3248.50.21~8\/RELEASE_X86_64",
  "machine": "x86_64"
},
"somap": [
  {
    "path": "\/usr\/local\/opt\/mongodb\/bin\/mongod",
    "machType": 2,
    "b": "106FA5000",
    "buildId": "A553ECE2755B341A850927E8DEE071E3"
  },
  {
    "path": "\/usr\/lib\/libSystem.B.dylib",
    "machType": 6,
    "b": "7FFF92227000",
    "buildId": "C5D09FE1CC70383EAC2718602F2EDEC4"
  },
  {
    "path": "\/usr\/lib\/libc++.1.dylib",
    "machType": 6,
    "b": "7FFF9112A000",
    "buildId": "8FC3D139805534989AC56467CB7F4D14"
  },
  {
    "path": "\/usr\/lib\/system\/libcache.dylib",
    "machType": 6,
    "b": "7FFF9ACA9000",
    "buildId": "9548AAE92AB735259ECEA2A7C4688447"
  },
  {
    "path": "\/usr\/lib\/system\/libcommonCrypto.dylib",
    "machType": 6,
    "b": "7FFF8B009000",
    "buildId": "9373226134B43914B7A290A81A182DBA"
  },
  {
    "path": "\/usr\/lib\/system\/libcompiler_rt.dylib",
    "machType": 6,
    "b": "7FFF9DD3D000",
    "buildId": "A13ECF69F59F38AE86097B731450FBCD"
  },
  {
    "path": "\/usr\/lib\/system\/libcopyfile.dylib",
    "machType": 6,
    "b": "7FFF8E288000",
    "buildId": "A48637BCF3F234F2BB684C65FD012832"
  },
  {
    "path": "\/usr\/lib\/system\/libcorecrypto.dylib",
    "machType": 6,
    "b": "7FFF9F808000",
    "buildId": "B5C05FD7A540345A87BF8E41848A3C17"
  },
  {
    "path": "\/usr\/lib\/system\/libdispatch.dylib",
    "machType": 6,
    "b": "7FFF8BEA7000",
    "buildId": "C749985761A53D7DA5EA65DCC8C3DF92"
  },
  {
    "path": "\/usr\/lib\/system\/libdyld.dylib",
    "machType": 6,
    "b": "7FFF8B915000",
    "buildId": "CC088C2AD40733E7A6B6B06E0D4AD999"
  },
  {
    "path": "\/usr\/lib\/system\/libkeymgr.dylib",
    "machType": 6,
    "b": "7FFF9B28B000",
    "buildId": "8371CE545FDD3CE9B3DFE98C761B6FE0"
  },
  {
    "path": "\/usr\/lib\/system\/liblaunch.dylib",
    "machType": 6,
    "b": "7FFF92532000",
    "buildId": "834ED60551143641AA4DECF31B801C50"
  },
  {
    "path": "\/usr\/lib\/system\/libmacho.dylib",
    "machType": 6,
    "b": "7FFF92499000",
    "buildId": "318264FA58F139D882851F6254EE410E"
  },
  {
    "path": "\/usr\/lib\/system\/libquarantine.dylib",
    "machType": 6,
    "b": "7FFF8C27F000",
    "buildId": "0F4169F00C843A25B3AEE47B3586D908"
  },
  {
    "path": "\/usr\/lib\/system\/libremovefile.dylib",
    "machType": 6,
    "b": "7FFF8DEC0000",
    "buildId": "552EF39E14D7363E90594565AC2F894E"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_asl.dylib",
    "machType": 6,
    "b": "7FFF8AFB5000",
    "buildId": "41F8E11F1BD03F1DBA3AAA1577ED98A9"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_blocks.dylib",
    "machType": 6,
    "b": "7FFF91908000",
    "buildId": "1244D9D5F6AA35BBB30786851C24B8E5"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_c.dylib",
    "machType": 6,
    "b": "7FFF9CDF0000",
    "buildId": "B552D565B7983B9BAE63F623B42A5F01"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_configuration.dylib",
    "machType": 6,
    "b": "7FFF8E876000",
    "buildId": "3DEB7DF9680437E1BC830166882FF0FF"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_coreservices.dylib",
    "machType": 6,
    "b": "7FFF8B8FE000",
    "buildId": "1B3F5AFCFFCD3ECB8B9A5538366FB20D"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_coretls.dylib",
    "machType": 6,
    "b": "7FFF9CDBC000",
    "buildId": "C90DAE384082381CA1852A6A8B677628"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_dnssd.dylib",
    "machType": 6,
    "b": "7FFF92CDD000",
    "buildId": "4D10E12B59B5386F82DA326F18028F0A"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_info.dylib",
    "machType": 6,
    "b": "7FFF98C85000",
    "buildId": "FAA9226D64DE3769A6D86CABA4B7FF4D"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_kernel.dylib",
    "machType": 6,
    "b": "7FFF98CDF000",
    "buildId": "78E54D59D2B03F549A4A0A68D671F253"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_m.dylib",
    "machType": 6,
    "b": "7FFF9F7D7000",
    "buildId": "08E1A4B264483DFEA58CACC7335BE7E4"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_malloc.dylib",
    "machType": 6,
    "b": "7FFF94E79000",
    "buildId": "5748E8B2F81C34C68B13456213127678"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_network.dylib",
    "machType": 6,
    "b": "7FFF9A5E8000",
    "buildId": "B52DAB7392DC3DA7B9F4B899D66445C1"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_networkextension.dylib",
    "machType": 6,
    "b": "7FFF97679000",
    "buildId": "66095DC7653938F295EE458F15F6D014"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_notify.dylib",
    "machType": 6,
    "b": "7FFF9942C000",
    "buildId": "D48BDE340F7E34CAA0FFC578E39987CC"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_platform.dylib",
    "machType": 6,
    "b": "7FFF8DEB7000",
    "buildId": "29A905EF67773C3382B06C3A88C4BA15"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_pthread.dylib",
    "machType": 6,
    "b": "7FFF98D4E000",
    "buildId": "3DD1EF4C1D1B3ABF8CC6B3B1CEEE9559"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_sandbox.dylib",
    "machType": 6,
    "b": "7FFF921C2000",
    "buildId": "150A9D3DF69E32F78C7B8E72CAAFF7E4"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_secinit.dylib",
    "machType": 6,
    "b": "7FFF95103000",
    "buildId": "32B1A8C6DC843F4FB8CE9A52B47C3E6B"
  },
  {
    "path": "\/usr\/lib\/system\/libsystem_trace.dylib",
    "machType": 6,
    "b": "7FFF993CB000",
    "buildId": "F00E92E4DBDA3749B5B30C3FBBABA1CB"
  },
  {
    "path": "\/usr\/lib\/system\/libunc.dylib",
    "machType": 6,
    "b": "7FFF9ECDF000",
    "buildId": "DDB1E947C77533B8B46163E5EB698F0E"
  },
  {
    "path": "\/usr\/lib\/system\/libunwind.dylib",
    "machType": 6,
    "b": "7FFF8B125000",
    "buildId": "F6EB48E54D12359AAB54C937FBBE9043"
  },
  {
    "path": "\/usr\/lib\/system\/libxpc.dylib",
    "machType": 6,
    "b": "7FFFA10B6000",
        "buildId": "54D1328E054E3DAA89E2375722F9D18F"
      },
      {
        "path": "\/usr\/lib\/libobjc.A.dylib",
        "machType": 6,
        "b": "7FFF9E13F000",
        "buildId": "7489D2D61EFD3414B18D2AECCCC90286"
      },
      {
        "path": "\/usr\/lib\/libauto.dylib",
        "machType": 6,
        "b": "7FFF9B6C1000",
        "buildId": "999E610F41FC32A3ADCA5EC049B65DFB"
      },
      {
        "path": "\/usr\/lib\/libc++abi.dylib",
        "machType": 6,
        "b": "7FFF8E88B000",
        "buildId": "DCCC81773D0935BC97842A04FEC4C71B"
      },
      {
        "path": "\/usr\/lib\/libDiagnosticMessagesClient.dylib",
        "machType": 6,
        "b": "7FFF96E9C000",
        "buildId": "4243B6B421E9355B9C5A95A216233B96"
      }
    ]
  }
}

[UPDATE]

Now the daemon won't even start anymore.

2016-06-24T10:05:32.986+0200 I -        [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2016-06-24T10:05:32.986+0200 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=9G,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),
2016-06-24T10:05:33.226+0200 E STORAGE  [initandlisten] WiredTiger (-31803) [1466755533:226568][6493:0x7fff75e89000], txn-recover: Recovery failed: WT_NOTFOUND: item not found
2016-06-24T10:05:33.248+0200 I -        [initandlisten] Assertion: 28595:-31803: WT_NOTFOUND: item not found
2016-06-24T10:05:33.249+0200 I STORAGE  [initandlisten] exception in initAndListen: 28595 -31803: WT_NOTFOUND: item not found, terminating
2016-06-24T10:05:33.249+0200 I CONTROL  [initandlisten] dbexit:  rc: 100

Answer 1

In your manual restart , it looks like you are running the mongodb with all defaults (the dbpath is /data/db, whereas the original ulimit error was logging /usr/local/var/mongodb).

Which path is supposed to be correct?

If you ran it manually with the wrong dbpath you would see an empty db.

Then if you didn't cleanly stop this process (e.g kill - 9), that could have corrupted this instance (hopefully on an incorrect dbpath ).

First, take a copy of the /usr/local/var/mongodb path.

Then try restarting the mongod process with the correct dbpath, user, permissions and ulimit settings (the earlier /etc/security/limits.conf suggestion was likely not useful on a Mac).

If none of this helps, then maybe try upgrade first (3.2.0 is quite old).