Jenkins slave cannot connect with master: Incorrect acknowledgement sequence

Question

After update of our Jenkins master installation to its latest LTS version 2.46.3 one of its slaves (Windows 7 machine, 32-bit) cannot connect with the master.

The error we're getting is:

java -jar slave.jar -jnlpUrl https://<jenkins-name>/computer/<node-name>/slave-agent.jnlp -secret <secret-value>
Jun 22, 2017 1:19:05 PM hudson.remoting.jnlp.Main createEngine
INFO: Setting up slave: node-name
Jun 22, 2017 1:19:05 PM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Jun 22, 2017 1:19:05 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [https://<jenkins-name>/]
Jun 22, 2017 1:19:05 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP3-connect, JNLP-connect, CLI2-connect, Ping, CLI-connect, JNLP4-connect, JNLP2-c
onnect]
Jun 22, 2017 1:19:05 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
  Agent address: <jenkins-name>
  Agent port:    <jenkins-port>
  Identity:      <id:en:ti:ty>
Jun 22, 2017 1:19:05 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Jun 22, 2017 1:19:05 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to <jenkins-name>:9150
Jun 22, 2017 1:19:05 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Jun 22, 2017 1:19:05 PM org.jenkinsci.remoting.protocol.impl.AckFilterLayer abort
WARNING: [JNLP4-connect connection to <our-proxy>/10.253.0.11:81] Incorrect acknowledgement sequence, expected 0x0003414333 got 0x4854545044
Jun 22, 2017 1:19:05 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Protocol JNLP4-connect encountered an unexpected exception
java.util.concurrent.ExecutionException: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowled
gement sent
        at org.jenkinsci.remoting.util.SettableFuture.get(SettableFuture.java:223)
        at hudson.remoting.Engine.innerRun(Engine.java:385)
        at hudson.remoting.Engine.run(Engine.java:287)
Caused by: org.jenkinsci.remoting.protocol.impl.ConnectionRefusalException: Connection closed before acknowledgement sent
        at org.jenkinsci.remoting.protocol.impl.AckFilterLayer.onRecvClosed(AckFilterLayer.java:280)
        at org.jenkinsci.remoting.protocol.FilterLayer.abort(FilterLayer.java:164)
        at org.jenkinsci.remoting.protocol.impl.AckFilterLayer.abort(AckFilterLayer.java:130)
        at org.jenkinsci.remoting.protocol.impl.AckFilterLayer.onRecv(AckFilterLayer.java:258)
        at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:669)
        at org.jenkinsci.remoting.protocol.NetworkLayer.onRead(NetworkLayer.java:136)
        at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access$2200(BIONetworkLayer.java:48)

We spent a lot of time trying to fix the problem. Unfortunately without success.

Do you have an idea what could have caused the problem and how can it be solved?

Answer 1

Is the Jenkins master instance running behind a load balancer? I had the same issue when my instance was running behind an Application Load Balancer in AWS.

If so, then the acknowledgement sequence could get modified because of differing protocols in the Load balancer. JNLP requires TCP connection on port 50000 by default.

If your setup is on AWS, you could try creating a private hosted zone in Route53 with an Alias record for your Jenkins instance's private IP address.

For e.g: jenkins.example.com -> your Jenkins instance's private IP

Then, in Jenkins UI -> Manage Jenkins -> Configure System -> Manage nodes and clouds -> Configure clouds -> (under advanced settings)

Tunnel connection through : jenkins.example.com:50000

This avoids your slave agents to have to go through the load balancer to connect to the Jenkins Master.