Java UrlConnection triggering "Connection reset" exceptions under high load. Why?

Question

I'm using Java to stream files from Amazon S3, on Linux (Ubuntu 10) 64-bit servers.

I'm using a separate thread for each file, and each file opens an HttpURLConnection which downloads and processes each file concurrently.

Everything works beautifully until I reach a certain number of streams (usually around 2-300 concurrent streams). At irregular points after this, several (say 10) of the threads will start experiencing java.net.IOException: Connection reset errors simultaneously.

I am throttling the download speed, and am way below the 250mbit/s limit of an m1.large instance. There is also insignificant load on all other server aspects (e.g. CPU, load average and memory usage are all fine).

What could be causing this, or how could I track it down?

Answer 1

not trivial to guess what may happen but this is a couple of hints , may be some may apply into your context:

• can you check your shell (linux bash /zsh or any other) to see if you raise up the standard limits restricting the number of file descriptors (but sockets too), man ulimit with bash shell
• did you close the streams explicitly in your Java code ? not closing streams may induce such clever problems
• try to google for Linux TCP kernel tuning to try to see if your ubuntu server has a well suited stack for such load context...

HTH Jerome

Answer 2

They might have spillover problem at VIPs because of number of con-current connections reached the limit. You may decrease the size and see...