How/where to store temp files and logs for a cloud app?

Question

I am working on a Python/MySQL cloud app with a fairly complex architecture. Operating this system (currently) generates temporary files (plain text, YAML) and log files and I had intended to store them on the filesystem.

However, our prospective cloud operator only provides a temporary, non-persistent filesystem to apps. This means that the initial approach with storing the temporary and log files won't work.

There must be a standard approach to solving this problem which I am not aware of. I don't want to use object storage like S3 because it would extend the current stack and add complexity. But I have the possibility to install an additional, dedicated app (if there is anything made for this purpose) on a different server with the same provider. The only limitation is that it would have to be in PHP, Python, MySQL.

The generic question: What is the standard approach to storing files when no persistent filesystem is available?

And for my specific case: Is there any solution using Python and/or MySQL which is simple and quick to implement? Is this a usecase for Redis?

Answer 1

Redis is a bad choice for this problem. It keeps all data in memory, so it's expensive and not practical for long term keeping of log files.

MySql is OK, however MongoDB is a more flaxible and fast solution. Actually it's one of it's main use cases: https://docs.mongodb.com/ecosystem/use-cases/storing-log-data/

Answer 2

As you are asking two questions, let me also answer them one by one:

The generic question: What is the standard approach to storing files when no persistent filesystem is available?

If the contents of a file are not suitable for regular database storage, but you really want to store the file itself (like images, binaries, etc) persistently, Amazon S3 and similar services are usually your way to go. There are also free alternatives available like Riak, Aerospike or the more heavy Cassandra.

While all of these services are free (or have free versions available), they would require setup and ongoing maintenance. Also you are very unlikely to achieve the same levels of availability and scalability as hosted cloud services like S3 have. If you account for this, the savings by not using a cloud service like S3 are questionable at least. But as always, YMMV.

And for my specific case: Is there any solution using Python and/or MySQL which is simple and quick to implement? Is this a usecase for Redis?

Your question is a little bit contradictory. You mention your application generates temporary files, yet you don't want to lose them?

If your files are temporary, a solution like Redis or memcached would be perfectly suitable for the job, as long as you have the understanding that both are caches and data will be lost on restart, or (if enabling Redis snapshots) at least don't guarantee that writes are persisted to disk.

If you do not want to extend your stack, and Redis does not give you the levels of guarantees you are looking for, MySQL supports the storage of blobs, which can be used for storing files (binary data) in MySQL. This generally does not scale very well, but again, deciding whether this is acceptable for you highly depends on your situation. (See also this excellent answer regarding the pro's and con's of storing files in MySQL.)