I've got a large queue of emails that are loaded in a MySQL table as events occur around my website. At anytime there maybe 1k-2k waiting to be sent at anytime.
I've been working on batching sets of emails and sending them out via one worker. Now I'm looking to add X workers to the system as load increases.
My question is about not allowing workers to grab the same email. I've heard the following options.
I see value in each of these but also a lot of reading / writing to the db. So what would you do?
Doing this via PHP and MySQL
Another quick and dirty option, assuming you have a primary key which it either random or structured, is letting each of N workers pick up only mails where key%N==i with i the worker id.
You could of course also take some other hash value which is sufficiently random. No need to lock, no extra coordination, just a tiny problem when one worker dies and you don't notice...
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With