PHP / MYSQL connection failures under heavy load through mysql.sock

Question

I've done quite a bit of reading before asking this, so let me preface by saying I am not running out of connections, or memory, or cpu, and from what I can tell, I am not running out of file descriptors either.

Here's what PHP throws at me when MySQL is under heavy load:

Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (11 "Resource temporarily unavailable")

This happens randomly under load - but the more I push, the more frequently php throws this at me. While this is happening I can always connect locally through the console and from PHP through 127.0.0.1 instead of "localhost" which uses the faster unix socket.

Here's a few system variables to weed out the usual problems:

cat /proc/sys/fs/file-max = 4895952
lsof | wc -l = 215778 (during "outages")

Highest usage of available connections: 26% (261/1000)

InnoDB buffer pool / data size: 10.0G/3.7G (plenty o room)

• soft nofile 999999
• hard nofile 999999

I am actually running MariaDB (Server version: 10.0.17-MariaDB MariaDB Server)

These results are generated both under normal load, and by running mysqlslap during off hours, so, slow queries are not an issue - just high connections.

Any advice? I can report additional settings/data if necessary - mysqltuner.pl says everything is a-ok

and again, the revealing thing here is that connecting via IP works just fine and is fast during these outages - I just can't figure out why.

Edit: here is my my.ini (some values may seem a bit high from my recent troubleshooting changes, and please keep in mind that there are no errors in the MySQL logs, system logs, or dmesg)

socket=/var/lib/mysql/mysql.sock
skip-external-locking
skip-name-resolve
table_open_cache=8092
thread_cache_size=16
back_log=3000
max_connect_errors=10000
interactive_timeout=3600
wait_timeout=600                                                                                            
max_connections=1000
max_allowed_packet=16M
tmp_table_size=64M
max_heap_table_size=64M
sort_buffer_size=1M
read_buffer_size=1M
read_rnd_buffer_size=8M
join_buffer_size=1M
innodb_log_file_size=256M
innodb_log_buffer_size=8M
innodb_buffer_pool_size=10G

[mysql.server]
user=mysql

[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
open-files-limit=65535

Answer 1

Most likely it is due to net.core.somaxconn What is the value of /proc/sys/net/core/somaxconn

net.core.somaxconn 

# The maximum number of "backlogged sockets".  Default is 128.

Connections in the queue which are not yet connected. Any thing above that queue will be rejected. I suspect this in your case. Try increasing it according to your load.

as root user run

echo 1024 > /proc/sys/net/core/somaxconn