Sending High Volume of Emails with Coldfusion

Question

This question could probably be related to doing anything high volume, but in this case I am trying to send emails.

I have setup the sending process in a new thread so user doesn't wait, and have overridden the request timeout to an hour.

The problem is that once the process get's up to about 2000 emails sent (looped over the below code about 2000 times) the server runs out of memory, stops responding, and needs a reboot.

Reading other topics on this, CF should be able to handle this volume of emails fine.

One thing I have considered is changing all object calls to straight DB Queries and using the cfmail tag to (I guess) remove all objects from being created and building up on reach request (which I guess is what is happening), but I'm not sure if that would make a difference, and really want to avoid that approach if possible. Something else I considered was splitting it across 3 or 4 seperate threads, but again, not sure if that would solve the problem.

Has anyone faced this problem, and what did you find worked to allow processing to continue without the ram slowly filling up and killing the server?

thread name="sendBroadcastEmail" rc="#rc#" prc="#prc#" filters="#rc.filters#" email="#email#" emailSignature="#emailSignature#"{
    createObject( "java", "coldfusion.tagext.lang.SettingTag" ).setRequestTimeout(javaCast( "double", 3600 ));

        //get profiles that it will be sent to
        var sendToProfiles = profileService.getWithFilters(rc.filters);

        var mailService = getPlugin("MailService");
        var emailSent = false;
        var sentCount = 0;
        var failedCount = 0;

        //send the email (and log in profile events)
        if (listFind(attributes.rc.email.action,'send')){

            for ( i=1; i<=arrayLen(sendToProfiles);i++){
                var profile = sendToProfiles[i];
                try{

                    if (len(trim(profile.getPrimaryEmail()))){

                        var emailBody = profile.processDynamicPlaceholders(attributes.rc.email.body);
                        var emailBody = attributes.emailSignature.getHeader() & emailBody & attributes.emailSignature.getFooter();

                        var sendEmail = mailService.newMail(
                             from = attributes.emailSignature.getEmailAddress(),
                             //to = profile.getPrimaryEmail(),
                             to = Application.settings.testemail,
                             subject = attributes.rc.email.subject,
                             body = emailBody,
                             type="html");

                             sendEmail.addMailParam(disposition='attachment', file=attributes.email.getAttachmentWithPath());
                             mailService.send(sendEmail);

                        //log profile event
                        profile.saveEvent(eventType = 3,
                                        title="Broadcast Email: #attributes.rc.email.subject#", 
                                        description="Broadcast Email Sent: Subject: <br> #attributes.rc.email.subject#",
                                        sentContent=emailBody,
                                        ref2=1);
                    sentCount++;
                    }
                }
                catch (any exception){
                    //log profile event
                    profile.saveEvent(eventType = 3,
                                    title="FAILED Broadcast Email", 
                                    description="<br>Subject: #attributes.email.subject#<br>This email should have been sent to this profile, but the attempted send failed.  The likely cause is a malformed email address.",
                                    sentContent=emailBody,
                                    ref2=0);
                    failedCount++;
                }

            }   
            emailSent = true;

        }


        //persist email object
        if (listFind(attributes.rc.email.action,'save')){
            email.setTstamp(attributes.prc.now);
            email.setSent(emailSent);
            email.setStatsSent(sentCount);
            email.save();
        }

    }//end thread   

Answer 1

One approach would be to generate the emails in timed batches to spread the load evenly. The batch size and delay between batches can be adjusted to suit your environment.

    thread name="sendBroadcastEmail" rc="#rc#" prc="#prc#" filters="#rc.filters#" email="#email#" emailSignature="#emailSignature#"{
    createObject( "java", "coldfusion.tagext.lang.SettingTag" ).setRequestTimeout(javaCast( "double", 3600 ));

            // set thread to a lowish prority
            var currentThread = CreateObject( "java","java.lang.Thread" ).currentThread();
            var priority = currentThread.getPriority();
            currentThread.setPriority( 3 );

        //get profiles that it will be sent to
        var sendToProfiles = profileService.getWithFilters(rc.filters);

        var mailService = getPlugin("MailService");
        var emailSent = false;
        var sentCount = 0;
        var failedCount = 0;

        //send the email (and log in profile events)
        if (listFind(attributes.rc.email.action,'send')){

            var emailsPerBatch = 1000; // divide into batches, set size here
            var batchcount = Ceiling( ArrayLen( sendToProfiles ) / emailsPerBatch ); // number of batches
            var batchdelay = 120000; // set delay between batches (ms)
            // initialise first batch
            var firstitem = 1;
            var lastitem = emailsPerBatch;

            for( var batch=1; batch<=batchcount; batch++ ) {
                if( batch > 1 ){
                    // delay sending next batch and give way to other threads
                    currentThread.yield();
                    currentThread.sleep( batchdelay );
                }

            for ( var i=firstitem; i<=lastitem;i++ ){
                var profile = sendToProfiles[i];

                            // generate emails ...

            }


            // initialise next batch
            firstitem = lastitem++;
            lastitem += emailsPerBatch;
            if( lastitem > ArrayLen( sendToProfiles ) ) {
                // last batch
                lastitem = ArrayLen( sendToProfiles );
            }

            }
            emailSent = true;
        }

            currentThread.setPriority( priority ); // reset thread priority


    }//end thread