HHVM fastcgi + Nginx performance fluctuations

Question

Currently we started using HHVM in a production envirionment and so far almost all results are pretty inpressive. Our overall transaction rate is greatly improved compared to PHP-FPM with APC. Almost all requests are under 500ms however every couple of requests (5 to 10 or so) results in a request time of 2 to even 5 seconds.

The page requested does not seem to make any difference and also requesting the same page over and over again will trigger this behavior within a couple of requests.

We are running HHVM in server mode with the following command line options:

/usr/bin/hhvm --mode server -vServer.Type=fastcgi -vServer.FileSocket=/usr/local/php55/sockets/admin.sock -vPidFile=/var/run/hhvm/admin.pid -vEval.Jit=true -vServer.ThreadCount=24 -vServer.APC.EnableApc=true

We are running nginx for the webserver with these relevant configurations (I am sorry if i forgot something importand here).

fastcgi_buffer_size 128k;
fastcgi_buffers 256 16k;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;
fastcgi_read_timeout 240;
fastcgi_intercept_errors on;

The server has 128GB of memory and 24 cores (hyperthreading so actually 12).

We did a fair bit of searching on https://github.com/facebook/hhvm/wiki/Runtime-options however most options aren't explained very well so I have no idea what they do and testing them in a production environment is a bit scary.

Had anybody here had a similar problem or could maybe point me in a direction with some of the HHVM options I would be very grateful.

The HHVM version used is from http://www.hop5.in/yum/el6/

HipHop VM 3.0.1 (rel)
Compiler: 
Repo schema: e69de29bb2d1d6434b8b29ae775ad8c2e48c5391

And the the /etc/hhvm/config.hdf

Log {
  Level = Warning
  AlwaysLogUnhandledExceptions = true
  RuntimeErrorReportingLevel = 8191
}

MySQL {
  TypedResults = false
}

We are using supervisord to start HHVM so here the supervisor config as wel:

[program:hhvm]
stopasgroup=true
killasgroup=true
command=/usr/bin/hhvm --mode server -vServer.Type=fastcgi -vServer.FileSocket=/usr/local/php55/sockets/admin.sock -vPidFile=/var/run/hhvm/admin.pid -vEval.Jit=true -vServer.ThreadCount=24 -vServer.APC.EnableApc=true 
user=admin
stdout_logfile=/var/log/hhvm/admin.log
stderr_logfile=/var/log/hhvm/admin.error.log
directory=/home/admin
umask=000

There is a php.ini in /etc/hhvm/php.ini but the content is empty. Also the pages tried all do some database connectivity but this is usually very minimal. For a complete picture also the my.cnf. The mysql version used is percona

[mysql]

# CLIENT #
port                           = 3306
socket                         = /var/lib/mysql/mysql.sock

[mysqld]

# GENERAL #
user                           = mysql
default-storage-engine         = InnoDB
socket                         = /var/lib/mysql/mysql.sock
pid-file                       = /var/lib/mysql/mysql.pid

[mysqld]

# MyISAM #
key-buffer-size                = 32M
myisam-recover                 = FORCE,BACKUP

# SAFETY #
max-allowed-packet             = 128M
max-connect-errors             = 1000000

# DATA STORAGE #
datadir                        = /var/lib/mysql/

# BINARY LOGGING #
log-bin                        = /var/lib/mysql/mysql-bin
expire-logs-days               = 14
sync-binlog                    = 1

# CACHES AND LIMITS #
tmp-table-size                 = 128M
max-heap-table-size            = 256M
query-cache-size               = 10G
max-connections                = 1000
thread-cache-size              = 100
open-files-limit               = 65535
table-definition-cache         = 4096
table-open-cache               = 4000
join-buffer-size               = 1M

# INNODB #
innodb-flush-method            = O_DIRECT
innodb-log-files-in-group      = 2
innodb-log-file-size           = 512M
innodb-flush-log-at-trx-commit = 1
innodb-file-per-table          = 1
innodb-buffer-pool-size        = 73G

# LOGGING #
log-error                      = /var/lib/mysql/mysql-error.log
log-queries-not-using-indexes  = 1
slow-query-log                 = 1
slow-query-log-file            = /var/lib/mysql/mysql-slow.log

Mysql version:

innodb_version  5.6.17-65.0
protocol_version    10
slave_type_conversions  
version 5.6.17-65.0-56-log
version_comment Percona Server (GPL), Release 65.0, Revision 587
version_compile_machine x86_64
version_compile_os  Linux

Answer 1

So we managed to figure it out, it was a combined problem of HHVM and the application we are using.

HHVM produces quite a bit of perf-****.map files and there where also heaps of sess_ files. In total there where over 8 million files in the /tmp directory. After removing those our problems went away and almost all our requests are significantly improved in performance. Also the session files shouldn't have been there in the first place but this was a problem which is already fixed.

Now we disabled the generation of the perf- files by adding -vEval.PerfPidMap to our startup arguments.