Best way to test high-volume SMTP email sending code?

Question

I've written a component in a Windows service (C#) which is responsible for sending sometimes large volumes of emails. These emails will go to recipients on many domains – really, any domain. (Yes, the recipients want the email. No, I'm not spamming. Yes, I'm in complaince with CAN-SPAM. Yes, I'm aware sending email from code sucks.) Many of the emails are transactional (generated in response to user actions); some are bulk (mail-merges basically).

I do not want to rely on an external SMTP server. (Among other considerations, the thought of having to check a mailbox for bounce messages and trying to parse them gives me bad feelings.)

My design is fairly simple. Both the transactional and bulk messages are generated and inserted into a DB table. This table contains the email envelope and content, plus an attempt count and retry-after date.

The service runs a few worker threads which grab 20 rows at a time and loop through each. Using the Simple DNS Plus library, I grab the MX record(s) of the recipient's domain and then use System.Net.Mail.SmtpClient to synchronously send the email. If the call to Send() succeeds, I can dequeue the email. If it temporarily fails, I can increment the attempt count and set an appropriate retry-after date. If it permanently fails, I can dequeue and handle the failure.

Obviously, sending thousands of test emails to hundreds of different actual domains is a Very Bad Idea. However, I definitely need to stress-test my multi-threaded send code. I'm also not quite sure what the best way is to simulate the various failure modes of SMTP. Plus, I want to make sure I get past the various spam control methods (graylisting to name the most relevant to the network layer of things).

Even my small-scale testing difficulties are exacerbated by my recent discovery of my ISP blocking connections to port 25 on any server other than my ISP's SMTP server. (In production, this thing will of course be on a proper server where port 25 isn't blocked. That does not help me test from my dev machine.)

So, the two things I'm most curious about:

1. How should I go about testing my code?
2. What are the various ways that SmtpClient.Send() can fail? Six exceptions are listed; SmtpException and SmtpFailedRecipientsException seem to be the most relevant.

---

Update: Marc B's answer points out that I'm basically creating my own SMTP server. He makes the valid point that I'm reinventing the wheel, so here's my rationale for not using an 'actual' one (Postfix, etc) instead:

1. Emails have different send priorities (though this is unrelated to the envelope's X-Priority). Bulk email is low priority; transactional is high. (And any email or group of emails can be further configured to have an arbitrary priority.) I need to be able to suspend the sending of lower-priority emails so higher-priority emails can be delivered first. (To accomplish this, the worker threads simply pick up the highest priority items from the queue each time they get another 20.)

   If I've already submitted several thousand bulk items to an external SMTP server, I have no way of putting those on hold while the items I wish to submit now get sent. A cursory Google search shows Postfix doesn't really support priorities; Sendmail prioritizes on information in the envelope, which does not meet my needs.

2. I need to be able to display the progress of the send process of a blast (group of bulk emails) to my users. If I've simply handed all of my emails off to an external server, I have no idea how far along in actual delivery it is.

3. I'm hesitant to parse bounce messages because each MTA's bounce message is different. Sendmail's is different from Exchange's is different from [...]. Also, at what frequency do I check my bounce inbox? What if a bounce message itself isn't delivered?

4. I'm not too terribly concerned with a blast failing half-way through.

   If we're talking catastrophic failure (app-terminating unhandled exception, power failure, whatever): Since the worker threads dequeue each email from the database when it is successfully delivered, I can know who has received the blast and who hasn't. Further, when the service resets after a failure, it simply picks up where it left off in the queue.

   If we're talking local failure (a SmtpException, DNS failure, or so forth): I just log the failure, increment the email's attempt counter, and try again later. (Which is basically what the SMTP spec calls for.) After n attempts, I can permanently fail the message (dequeue it) and log the failure for examination later. This way, I can find weird edge cases that my code isn't handling – even if my code isn't 100% perfect the first time. (And let's be honest, it won't be.)

5. I'm hoping the roll-my-own route will ultimately allow me to get emails out faster than if I had to rely on an external SMTP server. I'd have to worry about rate-limiting if the server weren't under my control; even if it were, it's still a bottleneck. The multithreaded architecture I've gone with means I'm connecting to multiple remote servers in parallel, decreasing the total amount of time it takes to deliver n messages.

Answer 1

Assume you've got two servers available. One will be the sender, one will be the receiver. You can set up DNS (or even just hosts files) on both with a long series of fake domains. As far as the two servers are concerned, those domains are perfectly valid as the local DNS servers are authoritative for them, but are completely invalid as far as the rest of the net is concerned. Just make sure the resolver checks the hosts file before DNS.

Once that's done, you can have the sending server spam the receiving server to your heart's content, as have the receiver do various things to test your code's reactions. Greylisting, TCP delays, hard bounces, ICMP unreachables, ICMP hops exceeded, etc...

Of course, given you have to test all these conditions, you're basically creating your own SMTP server, so why not use an actual one to begin with? I'd guess the effort required to do some basic parseing of bounce messages will be far less than having to come up with code chunks to handle all the failure modes that postfix/sendmail/exim/etc... already handle perfectly well on their own.

And this is especially true when you consider your sending code has to be perfect from the get-go. If an email blast fails part-way through and only half the recipient list gets the message, you're in a far bigger hole than if a few hundred or a few thousand messages bounce. Or worse yet, fails in multiple different ways (some servers unreachable, some greylisting you for excessive traffic, etc...). Whereas bounces will happily sit in the incoming queue until you process them manually, or patch up your bounce parser to handle them.

Answer 2

After searching around, I ended up firing up Papercut on several extra machines I had laying around. I then populated my database with test addresses *@[test-machine-*.local].

While this did work well enough, I tested with 25 send threads and it looked like I was overwhelming the four computers running Papercut. Several hundred send attempts experienced TCP connection failures; those messages were properly requeued to be sent later (and ultimately did arrive). However, out of 25,000 test emails, about 500 simply disappeared – adding up the *.eml files in Papercut's folder on each test machine yielded only ~24,500.

Now I'm left wondering whether the missing emails are a due to a problem in my code, or if Papercut dropped messages which it reported in SMTP as 250 OK.