Spark Job Server: "The server was not able to produce a timely response to your request"

Question

I'm using Spark Job Server to run a Spark job and it works perfectly. But when I try to execute a big job (needs more than 40 sec) I get this error:

The server was not able to produce a timely response to your request.

Is there some configuration required in order to wait for the server answer? What should I do?

Thank you

Answer 1

There are several different timeouts that you can hit with sync job submissions. Yours is coming from Spray-can server. You can configure it through your conf file:

spray.can.server {
      idle-timeout = <set desired timeout>
      request-timeout = <set desired timeout>
}

You can set both values to infinite to disable these timeouts at all.

In general, there are at least 3 common different timeouts that I've observed and that can return your request before job completion:

1. Akka-based ask timeout. Will return a JSON response with "ERROR" status, if sync job was not completed before X sec. Default in SJS is 10sec but you can overwrite it by passing timeout=Y argument to your POST /jobs request.
2. Spray-can server timeouts: idle-timeout, request-timeout. Will return default Spray response, making it harder to catch. They default to 60 and 40 secs respectively.
3. Yet another timeout may come from your client REST library... Some of them configure their defaults to disable any timeout on this layer, some of them may take into account the server's timeouts and configure their own behavior according to that... For example, C++ REST SDK library kicks in WinHTTP timeout of 30 secs once you disable timeout #2 above, with #2 in place it will wait for 40 sec...

Answer 2

In your Rest call, place(sync=false) at the end of the URL. Similar to http://server:8090/jobs?classPath=.... &sync=false at the end. It will start the job on the server and provide you a JobId.

The JobId can then be used to get the results:

For example: http://server:8090/jobs/b3b46a27-f711-469w-be09-4942006896b5

If that job is not finished, it will indicate the status as RUNNING. If it is finished, it will give you a status of FINISHED and the results.