cURL randomly throws "curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL" after some failed requests

Question

I have a PHP script that quickly sends a bunch of requests to the Apple API (APNs). Sometimes 10k requests are sent totally fine (just for the record, it takes ~30sec). However, when the API returns some non-200 codes, establishing new connections to this API throws the following error:

curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to api.push.apple.com:443

With the debug mode enabled I get strange results:

When everything is fine:

* Found bundle for host api.push.apple.com: 0x55eba8b11660 [serially]
* Found bundle for host api.push.apple.com: 0x55eba8b11660 [serially]
*   Trying 17.188.140.151...
* TCP_NODELAY set
* Hostname 'api.push.apple.com' was found in DNS cache
*   Trying 17.188.140.151...
...
...

(full output is here).

After the issue happened:

* Found bundle for host api.push.apple.com: 0x561d83bb1f00 [serially]
* Server doesn't support multiplex (yet)
* Connection #0 is still name resolving, can't reuse
* Found bundle for host api.push.apple.com: 0x561d83bb1f00 [serially]
* Server doesn't support multiplex (yet)
* Connection #0 is still name resolving, can't reuse
* Connection #1 is still name resolving, can't reuse
*   Trying 17.188.156.30:443...
* TCP_NODELAY set
* Hostname 'api.push.apple.com' was found in DNS cache
*   Trying 17.188.156.30:443...
...
...

(full output is here).

So the IP is changed and something about multiplexing.

After it happens, sending requests to this API using the cli curl also stops working:

$ curl -v -I https://api.push.apple.com
*   Trying 17.188.156.30:443...
* TCP_NODELAY set
* Connected to api.push.apple.com (17.188.156.30) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to api.push.apple.com:443 
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to api.push.apple.com:443 

(full output is here).

The error may disappear after a few minutes, may not. The code runs in a docker container (php:7.3-cli-alpine), restarting the container usually resets the issue until some future requests get non-200 codes. Performing restarts isn't an option.

Presumably, curl somehow stores opened connections and tries to re-use them, but for some reason something is broken inside the curl and it doesn't allow curl to correctly re-use the connections.

While curl stops working, openssl works fine:

$ openssl s_client -connect api.push.apple.com:443
CONNECTED(00000003)
write:errno=0
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 320 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

Software versions:

$ curl --version
curl 7.67.0 (x86_64-alpine-linux-musl) libcurl/7.67.0 OpenSSL/1.1.1d zlib/1.2.11 nghttp2/1.40.0
Release-Date: 2019-11-06
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS HTTP2 HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL TLS-SRP UnixSockets

$ openssl version
OpenSSL 1.1.1d  10 Sep 2019

Looks like a curl issue, but how to fix it?

Any help will be appreciated

Answer 1

Just in case it's helpful to you or someone else, the "one error breaks things for a long time" nature of this (and the re-use of Curl connections) reminded me a bit of the Curl bug: https://github.com/curl/curl/issues/3966

Not sure whether the fix for that is in your Curl version though - from the date it might well be.