Jenkins Plugin Manager fails to install/update a plugin because of a read timeout

Question

Our team recently had some problems with Jenkins Plugin Manager installing / updating plugins. After we fixed some problems with our firewall and proxy rules (we are behind a corporate firewall and a proxy), we still had issues downloading some of the plugin files.

Some plugin files were downloaded and installed successfully, others failed with the following error:

Failed to install git-client
java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
    at java.net.SocketInputStream.read(SocketInputStream.java:171)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
    at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
    at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1593)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
    at sun.net.www.protocol.http.HttpURLConnection.getHeaderField(HttpURLConnection.java:3057)
    at java.net.URLConnection.getHeaderFieldLong(URLConnection.java:629)
    at java.net.URLConnection.getContentLengthLong(URLConnection.java:501)
    at java.net.URLConnection.getContentLength(URLConnection.java:485)
    at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1161)
Caused: java.net.SocketTimeoutException: Read timed out
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1950)
    at sun.net.www.protocol.http.HttpURLConnection$10.run(HttpURLConnection.java:1945)
    at java.security.AccessController.doPrivileged(Native Method)
    at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1944)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1514)
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1498)
    at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1177)
Caused: java.io.IOException: Failed to load http://updates.jenkins-ci.org/download/plugins/git-client/3.0.0/git-client.hpi to /var/lib/jenkins/plugins/git-client.jpi.tmp
    at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1184)
Caused: java.io.IOException: Failed to download from http://updates.jenkins-ci.org/download/plugins/git-client/3.0.0/git-client.hpi (redirected to: http://ftp-nyc.osuosl.org/pub/jenkins/plugins/git-client/3.0.0/git-client.hpi)
    at hudson.model.UpdateCenter$UpdateCenterConfiguration.download(UpdateCenter.java:1218)
    at hudson.model.UpdateCenter$DownloadJob._run(UpdateCenter.java:1766)
    at hudson.model.UpdateCenter$InstallationJob._run(UpdateCenter.java:2037)
    at hudson.model.UpdateCenter$DownloadJob.run(UpdateCenter.java:1740)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at hudson.remoting.AtmostOneThreadExecutor$Worker.run(AtmostOneThreadExecutor.java:112)
    at java.lang.Thread.run(Thread.java:748)

To diagnose potential network problems, I tried downloading git-client.hpi directly from the machine hosting the Jenkins server via curl, which was successful:

[root@servername ~]# curl -vk -x http://proxyserver:7689 http://ftp-nyc.osuosl.org/pub/jenkins/plugins/git-client/3.0.0/git-client.hpi -o /tmp/git-client.hpi
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to proxy proxyserver port 6789 (#0)
*   Trying 10.100.50.100...
* Connected to proxyserver (10.100.50.100) port 6789(#0)
> GET http://ftp-nyc.osuosl.org/pub/jenkins/plugins/git-client/3.0.0/git-client.hpi HTTP/1.1
> User-Agent: curl/7.29.0
> Host: ftp-nyc.osuosl.org
> Accept: */*
> Proxy-Connection: Keep-Alive
>
  0     0    0     0    0     0      0      0 --:--:--  0:01:01 --:--:--     0< HTTP/1.1 200 OK
< Date: Mon, 11 Nov 2019 08:18:55 GMT
< Server: Apache
< Last-Modified: Sat, 02 Nov 2019 12:03:46 GMT
< ETag: "7dd268-5965bdfed67ce"
< Accept-Ranges: bytes
< Content-Type: application/vnd.hp-hpid
< Age: 6915
< Content-Length: 8245864
< Via: 1.1 proxyserver
<
  0 8052k    0     0    0     0      0      0 --:--:--  0:01:01 --:--:--     0{ [data not shown]
100 8052k  100 8052k    0     0   130k      0  0:01:01  0:01:01 --:--:-- 2132k
* Connection #0 to host proxyserver left intact

Downloading via curl was successful, although I noticed that after successful connection, the download process was stalled at 0% for about a minute. Then it suddenly started, and finished in seconds.

EDIT: this is not an insfrastructure problem, and has nothing to do with INFRA-2393, nor any other problems related to update site infrastructure:

• This read timeout was encountered first in October 2019, and could be reproduced in the middle of December 2019. Infrastucture problems tend to get solved a lot faster than that.

• I also managed to reproduce the problem at about 2020-01-02 10:30 CET, with /plugins/subversion/2.13.0/subversion.hpi using a 60 sec read timeout setting, long after INFRA-2393 got resolved. I also managed to fix the problem by raising the read timeout setting as described below.

• The read timeout only happens within Jenkins. Using the same download link in a browser or in a curl command downloads the affected files. This is because Jenkins has a default read timeout of 60 seconds, while browsers / curl do not.

• While diagnosing this earlier, we rearranged our network policy and made Jenkins reach the update sites directly (without using a proxy). In such a context, all the problem downloads worked. This indicates the stalling download is not related to an update site bug, but to our own proxy.

---

Also, stop asking me to file a report on Jenkins Jira. The very reason I posted this question and my answer is that I got frustrated with trying to find a solution using Jenkins Jira. The material that I found related to my problem there roughly fell into two categories:

• Infrastructure issues dating back several years, obviously fixed years ago. In other words: noise.

• Genuine Jenkins issues posted with similar symptoms, with dozens / hundreds of comments posted, mostly boiling down to Jenkins staff stating that the problem is on the user's side, and that people should fix their own corporate firewall / proxy / etc. Not a word related to timeout values whatsoever. In other words: a lot more noise.

Again: this is not a problem related to Jenkins update sites, nor is it a Jenkins software bug. If it's related to anything, then it's the lack of documentation. Exactly the thing that SO is meant to cover IMHO.

Answer 1

The important thing to notice from the above is that this is not a connection timeout but a read timeout. The connection to the download mirror is estabilished, but the client does not receive a single byte for about a minute. (We hypothesised that this happens because our proxy analyses the contents of git-client.hpi before forwarding the bytes to the client that requested the download.)

The curl download is successful because there's no read timeout set in the command. Jenkins obviously uses a 60 second default socket read timeout, and so after there's no traffic for 60 seconds, it considers the connection dead and fails with a SocketException.

I did not find any official documentation on this, but looking at Jenkins source code confirms the theory: the UpdateCenter.PLUGIN_DOWNLOAD_READ_TIMEOUT constant is initialized from a property value, and defaults to 60 in its absence.

The other thing I did not find is a comprehensive documentation of Jenkins' properties (key names, possible values, description, etc.).

The exact property key to be set is: hudson.model.UpdateCenter.pluginDownloadReadTimeoutSeconds

AFAIK Jenkins does not have any *.properties file, but accepts property settings on startup as JVM parameters.

I solved this problem by finding the jenkins options file (if you use RHEL, and installed Jenkins from package repository, the exact path is: /etc/sysconfig/jenkins), and appending -Dhudson.model.UpdateCenter.pluginDownloadReadTimeoutSeconds=120 to the value of the JENKINS_JAVA_OPTIONS setting.

UPDATE: I found the official Jenkins documentation about Jenkins system properties, which also has a few words about the hudson.model.UpdateCenter.pluginDownloadReadTimeoutSeconds property.

UPDATE (2022-07-08):

Beginning with Jenkins 2.332.1 and Jenkins 2.335, the Linux package installers use systemd to manage services. The RPM and deb package installers migrate configuration settings from System V init to systemd overrides.

This means that configuration files formerly used (/etc/sysconfig/jenkins or /etc/default/jenkins, depending on distribution) are now outdated and systemd unit files should be used to set system properties. See Jenkins documentation.