Connection reset consuming REST service (scala / spray)

Question

I have a problem sending concurrently requests to rest service; The messages in the client (Apache JMeter) is "Connection reset" for some requests, depending of the requests number, by example, I send 100 requests and the response of server is 100% successful, but if I send 500 requests, the 30% of the responses are error.

java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:196)
    at java.net.SocketInputStream.read(SocketInputStream.java:122)
    at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166)
    at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90)
    at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:61)
    at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254)
    at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289)
    at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252)
    at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191)
    at org.apache.jmeter.protocol.http.sampler.MeasuringConnectionManager$MeasuredConnection.receiveResponseHeader(MeasuringConnectionManager.java:201)
    at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127)
    at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
    at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.executeRequest(HTTPHC4Impl.java:517)
    at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:331)
    at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:74)
    at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1146)
    at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1135)
    at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:434)
    at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:261)
    at java.lang.Thread.run(Thread.java:745)

I modified the "application.conf", the content is the next:

spray.can {
  server {
    server-header = spray-can/${spray.version}
    ssl-encryption = off
    pipelining-limit = 16 
    idle-timeout = 60 s 
    request-timeout = 30 s
    timeout-timeout = 2 s
    timeout-handler = ""
    reaping-cycle = 250 ms
    stats-support = on
    remote-address-header = off
    raw-request-uri-header = off
    transparent-head-requests = on
    chunkless-streaming = off
    verbose-error-messages = on
    request-chunk-aggregation-limit = 1m
    response-header-size-hint = 512
    bind-timeout = infinite
    unbind-timeout = 1s
    registration-timeout = 1s
    default-host-header = ""
    automatic-back-pressure-handling = on
    back-pressure {
      noack-rate = 10
      reading-low-watermark = infinite
    }
    parsing = ${spray.can.parsing}
  }
  client {
    user-agent-header = spray-can/${spray.version}
    idle-timeout = 60 s
    request-timeout = 40 s 
    reaping-cycle = 250 ms
    response-chunk-aggregation-limit = 1m
    chunkless-streaming = off
    request-header-size-hint = 256 
    max-encryption-chunk-size = 1m
    connecting-timeout = 30s
    proxy {
      http = default
      https = default
    }
    ssl-tracing = off
    parsing = ${spray.can.parsing}
  }
  host-connector {
    max-connections = 80 
    max-retries = 8 
    max-redirects = 0
    pipelining = enabled
    idle-timeout = 30 s
    client = ${spray.can.client}
  }
}

The settings of JVM are:

-Xms1024M
-Xmx2048M 
-Xss1M 
-XX:MaxPermSize=1024m

IMPORTANT: Because bussines logic, is necesary that the server support concurrently transactions; 500 individual connections (transactions) in less that 5 seconds.

Answer 1

Your timeout settings look fine, and handling 500 requests per second is definitely not a problem.

Most likely your requests take too long to process, i.e. more than request-timeout + timeout-timeout = 32 seconds. You need to check your architecture and see where and why it spends so much time. This would be quite unusual for regular web services, where most of the requests complete in millisecond range. In case you have some heavy processing that you have to do that takes longer than the timeout you can reply with 202 Accepted and do processing on the background. You can return a URI where client can check the status of the request, or use callback to the client or any other mechanism to communicate that the request is done.

Remember not to block in the route itself otherwise you would be effectively blocking all other requests and you could be getting timeout errors. See this answer for instance: Use a Dispatcher with Spray HttpService. To implement non-blocking request processing see this: How does spray.routing.HttpService dispatch requests?.

Some ideas for troubleshooting: 1) measure how much time it takes to process single request, and see how it scales - do you have resource contention? 2) check that your network and client does not cause timeout - their timeout should be higher than server's.