MongoDB upgrade from 3.6 to 4.0: How to fix the "collection does not have uuid in kvcatalog" exception?

Question

I run a MongoDB replica set in a containerization environment orchestrated by Rancher. After rolling out the latest update, MongoDB as been upgraded from 3.6.x to 4.0.x (latest).

No upgrade path has been followed and, as a result, when trying to bind the volume holding the database storage, the service fails to start, exiting with the following exception:

STORAGE  [initandlisten] exception in initAndListen: MustDowngrade: Collection does not have UUID in KVCatalog. Collection: admin.system.version, terminating

I get the same outcome even attempting to launch mongod with the --repair option. I have also tried to rollback the container to MongoDB 3.6.16, but the journal version is now incompatible.

We don't have snapshots of the volume, hence restoring the data is not an option (this is not a production environment). The only solution I've found online is not applicable, since it suggests to leverage mongodump and mongorestore, requiring a running database with the data storage attached.

I'm running out of ideas, any advice about how to fix this?

Answer 1

A bug report on this issue recommended following the full upgrade path and that worked for me. I was able to go back to my old version when I ran into this error, and then upgrade properly.

Specifically: upgrade just one major version at a time and check the prerequisites and upgrade steps carefully. The most important part of this is checking your compatibility with db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } ) and make sure it matches the current version before upgrading to the next version. Update it with command like db.adminCommand( { setFeatureCompatibilityVersion: "3.6" } )

Read the full docs at pages like https://docs.mongodb.com/manual/release-notes/3.6/#upgrade-procedures https://docs.mongodb.com/manual/release-notes/4.0/#upgrade-procedures etc