I use playframework 2.0 in production and I see a rapidly growing number of files open by the corresponding java process. I changed the default allowed number of open files for a process from 1024 to 4096, but this just delays a problem and after some time the number of open files becomes so big, that the server slows down and sometimes even a java.net.SocketException
"too many open files" appears in a log.
I use ubuntu server 12.04, sun jdk 1.7.0_09.
lsof
command demonstrates that almost all of open files are sockets created for user connections, so I make a conclusion that the Netty server that is used by playframework keeps alive a lot of open connections. As far as I understand I should change the keep-alive behaviour of the Netty server, for example, to set a low keep-alive timeout. How can I configure this in playframework 2.0? Or may be the problem is in something else? I can provide any configuration if needed.
UPD: here is a bit of output of lsof -aPn -p 12251
. I censored some parts of ips with ***
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 12251 root 324u IPv6 279502 0t0 TCP 10.127.0.***:80->***.***.65.137:51506 (ESTABLISHED)
java 12251 root 325u IPv6 286749 0t0 TCP 10.127.0.***:80->***.***.18.121:1624 (ESTABLISHED)
java 12251 root 327u IPv6 287220 0t0 TCP 10.127.0.***:80->***.***.126.210:49244 (ESTABLISHED)
java 12251 root 330u IPv6 279289 0t0 TCP 10.127.0.***:80->***.***.65.155:12444 (ESTABLISHED)
java 12251 root 331u IPv6 285609 0t0 TCP 10.127.0.***:80->***.***.33.233:2552 (ESTABLISHED)
java 12251 root 332u IPv6 285610 0t0 TCP 10.127.0.***:80->***.***.33.233:2554 (ESTABLISHED)
java 12251 root 333u IPv6 287236 0t0 TCP 10.127.0.***:80->***.***.90.20:16040 (ESTABLISHED)
java 12251 root 334u IPv6 284047 0t0 TCP 10.127.0.***:80->***.***.195.2:1175 (ESTABLISHED)
java 12251 root 335u IPv6 279357 0t0 TCP 10.127.0.***:80->***.***.65.137:51273 (ESTABLISHED)
java 12251 root 336u IPv6 279988 0t0 TCP 10.127.0.***:80->***.***.65.137:51287 (ESTABLISHED)
Keep-Alive Timeout The time (in seconds) before idle keep-alive connections are closed. Set this value in the Admin Console in the Timeout field on the configuration's Performance tab ⇒ HTTP tab, under Keep Alive Settings. The default is 30 seconds, meaning the connection times out if idle for more than 30 seconds.
The keep alive timeout on the Message Processor allows a single TCP connection to send and receive multiple HTTP requests/responses from/to the backend server, instead of opening a new connection for every request/response pair.
If you're using Play without any HTTP server you can manipulate the response headers before sending the Result
(Scala version) , otherwise you need to check settings of the front-end server.
response().setHeader(CONNECTION, "Keep-Alive");
response().setHeader("Keep-Alive", "timeout=3, max=10");
return ok(index.render());
Note, that these values are sample and I don't know if they will fit your needs.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With