Sending newsletter using PHP SwiftMailer and SMTP

Question

I send daily a newsletter to about 20K recipients. After moving to other server I need about six times more time to this using the same PHP scripts – I tried many different settings but had no luck with it (please see this post). The email server seems to be properly configured (with regard to spam, relay and viruses), the server runs with Plesk 11.5.30.

Doing many experiments, changing PHP scripts and Postfix settings I was able to achieve the best performance with the following:

1. Sending emails from PHP using SwiftMailer.
2. Using SMTP (local account) as transport.
3. Dividing recipients list in four parts and starting separate PHP-CLI process for each one.

When I start only one process, PHP sends about 5 emails / second (which are about the same when I send emails using mail() function). Postfix has no problems with sending them "on the fly“ so there is almoust no queue.

When I start four processes, PHP sends about 10 emails / second, however postfix is no more able to send them „on the fly“ (it can send only about 5 emails / second) so the queue becomes bigger and bigger. When the PHP processes are over, I have about 4500 emails in my queue. And here comes the interesting part: when there are no more active PHP processes sending emails, Postfix „accelerates“ to sending about 20 emails / second. Would Postfix be able to send the emails at this rate while PHP is delivering new emails? there would be no queue and it would solve my problem.

I would appreciate any idea what could I do to improve the performance of my script and server sending the newsletter.

Here comes the Postfix configuration:

root@hostname:~# postconf -n 
alias_database = hash:/etc/aliases 
alias_maps = hash:/etc/aliases, hash:/var/spool/postfix/plesk/aliases 
append_dot_mydomain = no 
biff = no 
config_directory = /etc/postfix 
disable_vrfy_command = yes 
inet_interfaces = all 
inet_protocols = ipv4 
mailbox_size_limit = 0 
mailman_destination_recipient_limit = 1 
message_size_limit = 51200000 
mydestination = (none), localhost, localhost.localdomain, localhost 
mydomain = domain.com 
myhostname = myhostname.com 
mynetworks = 127.0.0.0/8 [::1]/128 my.ip.add.ress 
plesk_virtual_destination_recipient_limit = 1 
readme_directory = no 
recipient_delimiter = + 
relayhost = 
sender_dependent_default_transport_maps = hash:/var/spool/postfix/plesk/sdd_transport_maps 
smtp_connect_timeout = 10s 
smtp_helo_timeout = 100s 
smtp_send_xforward_command = yes 
smtp_tls_security_level = may 
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache 
smtp_use_tls = no 
smtpd_authorized_xforward_hosts = 127.0.0.0/8 [::1]/128 
smtpd_banner = $myhostname ESMTP $mail_name (Debian/GNU) 
smtpd_client_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_rbl_client zen.spamhaus.org 
smtpd_proxy_timeout = 3600s 
smtpd_recipient_restrictions = permit_mynetworks, check_client_access pcre:/var/spool/postfix/plesk/no_relay.re, permit_sasl_authenticated, reject_unauth_destination 
smtpd_sasl_auth_enable = yes 
smtpd_sender_restrictions = check_sender_access hash:/var/spool/postfix/plesk/blacklists, permit_sasl_authenticated, check_client_access pcre:/var/spool/postfix/plesk/non_auth.re 
smtpd_timeout = 3600s 
smtpd_tls_cert_file = /etc/postfix/postfix_default.pem 
smtpd_tls_key_file = $smtpd_tls_cert_file 
smtpd_tls_security_level = may 
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache 
smtpd_use_tls = yes 
transport_maps = , hash:/var/spool/postfix/plesk/transport 
virtual_alias_maps = $virtual_maps, hash:/var/spool/postfix/plesk/virtual 
virtual_gid_maps = static:31 
virtual_mailbox_base = /var/qmail/mailnames 
virtual_mailbox_domains = $virtual_mailbox_maps, hash:/var/spool/postfix/plesk/virtual_domains 
virtual_mailbox_limit = 0 
virtual_mailbox_maps = , hash:/var/spool/postfix/plesk/vmailbox 
virtual_transport = plesk_virtual 
virtual_uid_maps = static:110

Answer 1

Since you write about your PHP process are sending about 5 or 20 mails per seconds, I'll suppose you are already measuring in detail your PHP code, so if this delay is in the PHP code you should be already discovered.

Reading the comments I also suppose your systems are quite big, so you have enough ram and disk.

In my experience a very common problem that could cause such kinds of delay are the DNS resolutions, well, I mean a wrong configuration of DNS, that leads in a delay all around. But this hypothesis is all to be proved.

On the other hand, I even remember that postfix have a throttling configuration in order to avoid server abuses. http://www.postfix.org/TUNING_README.html

At last, if you don't find anything in the postfix configuration, what I can suggest you is to try to debug the postfix process during its execution, with the utility strace.

It is a quite useful utility, I used it often when I have to debug non common problems. In short: strace lists all system calls done by the process. There are a lot of options available in strace, but I usually use only:

strace -fp <pid of main process>

During the debug you'll see all the system calls executed by postfix main process and all its children. If you add the -c option then you'll see the time spent for every system call.