Design pattern for accumulate / flush messages

Question

currently we have a publish / consumer service where the consumer writes the messages received to AWS S3. We are currently writing more than 100.000.000 objects per month. However, we can group this messages based on some rules in order to save some money. These rules, can be something like:

• If we have received 10 messages of the User 1 -> group them, and write to S3.
• If we have received < 10 messages of the User 1 and the elapsed time since the last message is more than 5 seconds, flush to S3.
• If the "internal" queue, is bigger than N, start to flush

What we don't want is to eat our memory... Because of that, I am looking of what would be the best approach from design patterns perspective, taking into consideration that we are speaking about a high loaded system, so we don't have infinite memory resources.

Thanks!,

Answer 1

Well, based on your further explanation in comments, there are related algorithms called Leaky bucket and Token bucket. Their primary purpose is slightly different but you may consider using some modification - especially you may consider viewing the "leaking droplets out of the bucket" as the regular commit of all the messages of a single user in a bunch flush to S3.

So more or less modification like this (please read the description of the algorithms first):

1. You have a bucket per user (you may easily afford it, as you have only about 300 users)
2. Each bucket is filled by messages coming from each user
3. You regularly let each bucket leak (flush all the messages or just a limited bunch of messages)

I guess that it somehow follows what your original requirement might have been.