CarrierWave save image to gridfs and upload in background s3

Question

Is there any way to save image to mongo's gridfs and after asynchronous upload to S3 in background?

Maybe it is possible to chain uploaders?

The problem in next: Multiple servers used, thus - saved to hard drive image and running background process can be on different servers.

Also 1. it should remove from gridfs when uploaded to s3 2. it should auto remove from s3 when correspond entity destroyed.

Thanks.

Answer 1

What does your deployment architecture look like? I'm a little confused by when you say "multiple servers"- do you mean multiple mongod instances? Also, it's a bit confusing when you specify your requirements. According to requirement 1, if you upload to S3, then the gridfs file should be removed. However, according to your requirements, it cannot exist in both S3 and Gridfs, so requirement 2 seems to be a contradiction to the first, ie, it shouldn't exist in gridfs in the first place. Are you preserving some files on both Gridfs and S3?

If you are running in a replica set or sharded cluster, you could create a tailable cursor on your gridfs collection (you can also do this on a single node, although it's not recommended). When you see an insert operation (will look like 'op':'i') you could execute a script or do something in your application to grab the file from gridfs and push the appropriate file to s3. Similarly, when you see a delete operation ('op':'d') you could summarily delete the file from s3.

The beauty of a tailable cursor is that it allows for asynchronous operations- you can have another process monitoring the oplog on a different server and performing the appropriate actions.