Redis – Failed opening .rdb for saving: Permission denied

Question

I am using redis version 3.0.6. The redis-server process is being run by the redis user.

Suddenly from 5 days after 24 hours redis began failing "opening .rdb for saving." It was working properly before this.

As you can see in the snippet from the logs below, Redis was behaving normally, and then started failing. Power-cycling the server later resolved the issue.

1427:M 24 May 01:09:05.102 * Background saving started by pid 2493    
2493:C 24 May 01:09:34.916 * DB saved on disk
2493:C 24 May 01:09:34.917 * RDB: 310 MB of memory used by copy-on-write
1427:M 24 May 01:09:34.950 * Background saving terminated with success
1427:M 24 May 01:14:35.026 * 10 changes in 300 seconds. Saving...
1427:M 24 May 01:14:35.036 * Background saving started by pid 2494
2494:C 24 May 01:15:04.329 * DB saved on disk
2494:C 24 May 01:15:04.330 * RDB: 298 MB of memory used by copy-on-write
1427:M 24 May 01:15:04.408 * Background saving terminated with success
1427:M 24 May 01:20:05.008 * 10 changes in 300 seconds. Saving...
1427:M 24 May 01:20:05.018 * Background saving started by pid 2499
2499:C 24 May 01:20:33.830 * DB saved on disk
2499:C 24 May 01:20:33.831 * RDB: 330 MB of memory used by copy-on-write
1427:M 24 May 01:20:33.843 * Background saving terminated with success
1427:M 24 May 01:23:46.966 # Failed opening .rdb for saving: Read-only file system
1427:M 24 May 01:25:34.029 * 10 changes in 300 seconds. Saving...
1427:M 24 May 01:25:34.038 * Background saving started by pid 2500
2500:C 24 May 01:25:34.038 # Failed opening .rdb for saving: Read-only file system
1427:M 24 May 01:25:34.139 # Background saving error
1427:M 24 May 01:25:40.059 * 10 changes in 300 seconds. Saving...
1427:M 24 May 01:25:40.064 * Background saving started by pid 2501
2501:C 24 May 01:25:40.064 # Failed opening .rdb for saving: Read-only file system
1427:M 24 May 01:25:40.165 # Background saving error
1427:M 24 May 01:25:46.080 * 10 changes in 300 seconds. Saving...
1427:M 24 May 01:25:46.085 * Background saving started by pid 2502
2502:C 24 May 01:25:46.085 # Failed opening .rdb for saving: Read-only file system
1427:M 24 May 01:25:46.186 # Background saving error
1427:M 24 May 01:25:52.100 * 10 changes in 300 seconds. Saving...
1427:M 24 May 01:25:52.105 * Background saving started by pid 2503
2503:C 24 May 01:25:52.105 # Failed opening .rdb for saving: Read-only file system
1427:M 24 May 01:25:52.206 # Background saving error

So, my question: how could this happen? Please give me proper solution for this.

Answer 1

The "Read-only file system" I think is the key here. It's possible the device it's trying to write to is mounted incorrectly but since it happened randomly, the system may have forced the filesystem into readonly mode. There's a number of conditions that can trigger the operating system to put the filesystem into a read-only mode. This can mean that the filesystem became corrupt or there was some other filesystem consistency issue. If you're hosting on a cloud provider and the disk is network-backed like EBS in AWS, this can be triggered by a temporary network issue. Sometimes the issues are momentary and either force remounting the partition (or power cycling the server) will fix the issue. Other times it's permanent, but since your server came back up just fine, that would appear to not be the case. But the true fix for this would lie in your hardware setup which wasn't detailed.

This answer is related albeit thin on the "why": Failed opening the RDB file ... Read-only file system