Azure DevOps Git Repo Error - RPC failed; curl 56 Failure

Question

I have an Azure DevOps project with a couple of Git repos. When I do a git clone from a Command Prompt on the "bad" repo I get the following error message:

Cloning into 'myBadRepo'...
remote: Azure Repos
remote: Found 176 objects to send. (0 ms)
error: RPC failed; curl 56 Failure when receiving data from the peer
Receiving objects: 100% (176/176), 10.22 MiB | 25.46 MiB/s, done.
Resolving deltas: 100% (48/48), done.

Searching Google for curl 56 Failure mostly says that this is an issue with a firewall or proxy. I am on a corporate network, using Cisco AnyConnect as a VPN, but turning that off makes no difference. My machine is also using zScaler for "Internet Security" which I know is messing with the certificates, but I cannot disable it. But I doubt that the firewall, proxy or certs are main the issue, because I can successfully clone a different Git repo from the same Azure DevOps organization and project:

Cloning into 'myWorkingRepo'...
remote: Azure Repos
remote: Found 107 objects to send. (3 ms)
Receiving objects: 100% (107/107), 859.57 KiB | 29.64 MiB/s, done.
Resolving deltas: 100% (35/35), done.

I have tried doing a "shallow clone" as suggested at https://stackoverflow.com/a/66207817/11057678 and increasing the http.postBuffer as suggested at https://stackoverflow.com/a/68097529/11057678 but neither makes any differece.

I turned on verbose mode and tracing, as suggested at https://confluence.atlassian.com/stashkb/git-clone-fails-error-rpc-failed-result-56-http-code-200-693897332.html

set GIT_TRACE_PACKET=1
set GIT_TRACE=1
set GIT_CURL_VERBOSE=1

I can see that both logs have repeated warnings of Info: schannel: failed to decrypt data, need more data which is concerning, but the "working" repo is cloning successfully, while the "bad" repo has a lot more of these warnings, and after the last set of 647 (!) failed to decrypt messages it shows a closed connection and then the curl 56 Failure:

Info: schannel: server closed abruptly (missing close_notify)
Info: Closing connection 0
channel: shutting down SSL/TLS connection with dev.azure.com port 443
error: RPC failed; curl 56 Failure when receiving data from the peer

Excerpts from the "bad" log is below (the full log is too long to post). Both logs contain the same authorization-related warning and errors, which are not however preventing a clone of the "good" repo. One error in particular is the URL-encoded

The user 'aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa' is not authorized to access this resource

Where is this user aaaaaaa coming from? I suspect that if I can configure Git and/or the Azure DevOps repo in a way to eliminate these authorization-related warnings and errors, my problems with the "bad" repo will be solved. Hopefully someone can take a look at them and give me some advice or direction, thanks!

"Bad" repo log excerpts:

git clone https://[email protected]/myOrganization/myDevOpsProject/_git/myBadRepo

05:13:49.157466 exec-cmd.c:237          trace: resolved executable dir: C:/Program Files/Git/mingw64/bin
05:13:49.158466 git.c:459               trace: built-in: git clone https://[email protected]/myOrganization/myDevOpsProject/_git/myBadRepo
Cloning into 'myBadRepo'...
05:13:49.180609 run-command.c:654       trace: run_command: git remote-https origin https://[email protected]/myOrganization/myDevOpsProject/_git/myBadRepo
05:13:49.194684 exec-cmd.c:237          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
05:13:49.195683 git.c:748               trace: exec: git-remote-https origin https://[email protected]/myOrganization/myDevOpsProject/_git/myBadRepo
05:13:49.196685 run-command.c:654       trace: run_command: git-remote-https origin https://[email protected]/myOrganization/myDevOpsProject/_git/myBadRepo
05:13:49.212111 exec-cmd.c:237          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
05:13:49.220376 http.c:689              == Info: Couldn't find host dev.azure.com in the (nil) file; using defaults
05:13:49.225816 http.c:689              == Info:   Trying 13.107.42.20:443...
05:13:49.244824 http.c:689              == Info: Connected to dev.azure.com (13.107.42.20) port 443 (#0)
05:13:49.245003 http.c:689              == Info: schannel: disabled automatic use of client certificate
05:13:49.313976 http.c:636              => Send header, 0000000240 bytes (0x000000f0)
05:13:49.313976 http.c:648              => Send header: GET /myOrganization/myDevOpsProject/_git/myBadRepo/info/refs?service=git-upload-pack HTTP/1.1
05:13:49.313976 http.c:648              => Send header: Host: dev.azure.com
05:13:49.313976 http.c:648              => Send header: User-Agent: git/2.36.0.windows.1
05:13:49.313976 http.c:648              => Send header: Accept: */*
05:13:49.313976 http.c:648              => Send header: Accept-Encoding: deflate, gzip, br, zstd
05:13:49.313976 http.c:648              => Send header: Pragma: no-cache
05:13:49.313976 http.c:648              => Send header: Git-Protocol: version=2
05:13:49.313976 http.c:648              => Send header:
05:13:49.571301 http.c:689              == Info: schannel: failed to decrypt data, need more data
... 8 lines removed ...
05:13:49.672466 http.c:689              == Info: schannel: failed to decrypt data, need more data
05:13:49.672466 http.c:689              == Info: Mark bundle as not supporting multiuse
05:13:49.672466 http.c:636              <= Recv header, 0000000027 bytes (0x0000001b)
05:13:49.672466 http.c:648              <= Recv header: HTTP/1.1 401 Unauthorized
[....]
05:13:49.672466 http.c:648              <= Recv header: WWW-Authenticate: Bearer authorization_uri=https://login.microsoftonline.com/cc808ba9-3e5f-4f13-a70b-5b65bf454995
05:13:49.672466 http.c:636              <= Recv header, 0000000071 bytes (0x00000047)
05:13:49.672466 http.c:648              <= Recv header: WWW-Authenticate: Basic realm="https://tfsprodcus6.visualstudio.com/"
05:13:49.672466 http.c:636              <= Recv header, 0000000033 bytes (0x00000021)
[...]
05:13:49.672466 http.c:648              <= Recv header: X-FRAME-OPTIONS: SAMEORIGIN
05:13:49.672466 http.c:636              <= Recv header, 0000000059 bytes (0x0000003b)
05:13:49.672466 http.c:648              <= Recv header: X-TFS-FedAuthRealm: https://tfsprodcus6.visualstudio.com/
05:13:49.672466 http.c:636              <= Recv header, 0000000059 bytes (0x0000003b)
05:13:49.672466 http.c:648              <= Recv header: X-TFS-FedAuthIssuer: https://dev.azure.com/myOrganization/
05:13:49.672466 http.c:636              <= Recv header, 0000000073 bytes (0x00000049)
05:13:49.672466 http.c:648              <= Recv header: X-VSS-AuthorizationEndpoint: https://vssps.dev.azure.com/myOrganization/
05:13:49.672466 http.c:636              <= Recv header, 0000000060 bytes (0x0000003c)
05:13:49.672466 http.c:648              <= Recv header: X-VSS-ResourceTenant: cc808ba9-3e5f-4f13-a70b-5b65bf454995
05:13:49.672466 http.c:636              <= Recv header, 0000000710 bytes (0x000002c6)
05:13:49.672466 http.c:648              <= Recv header: X-TFS-SoapException: %3C%3Fxml%20version%3D%221.0%22%20encoding%3D%22utf-8%22%3F%3E%3Csoap%3AEnvelope%20xmlns%3Asoap%3D%22http%3A%2F%2Fwww.w3.org%2F2003%2F05%2Fsoap-envelope%22%3E%3Csoap%3ABody%3E%3Csoap%3AFault%3E%3Csoap%3ACode%3E%3Csoap%3AValue%3Esoap%3AReceiver%3C%2Fsoap%3AValue%3E%3Csoap%3ASubcode%3E%3Csoap%3AValue%3EUnauthorizedRequestException%3C%2Fsoap%3AValue%3E%3C%2Fsoap%3ASubcode%3E%3C%2Fsoap%3ACode%3E%3Csoap%3AReason%3E%3Csoap%3AText%20xml%3Alang%3D%22en%22%3ETF400813%3A%20The%20user%20%27aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa%27%20is%20not%20authorized%20to%20access%20this%20resource.%3C%2Fsoap%3AText%3E%3C%2Fsoap%3AReason%3E%3C%2Fsoap%3AFault%3E%3C%2Fsoap%3ABody%3E%3C%2Fsoap%3AEnvelope%3E
05:13:49.673499 http.c:636              <= Recv header, 0000000148 bytes (0x00000094)
05:13:49.673499 http.c:648              <= Recv header: X-TFS-ServiceError: TF400813%3A%20The%20user%20%27aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa%27%20is%20not%20authorized%20to%20access%20this%20resource.
05:13:49.673499 http.c:636              <= Recv header, 0000000079 bytes (0x0000004f)
05:13:49.673499 http.c:648              <= Recv header: X-VSS-S2STargetService: 00000002-0000-8888-8000-000000000000/visualstudio.com
05:13:49.673499 http.c:636              <= Recv header, 0000000540 bytes (0x0000021c)
05:13:49.673499 http.c:648              <= Recv header: X-TFS-FedAuthRedirect: https://spsprodcus4.vssps.visualstudio.com/_signin?realm=dev.azure.com&reply_to=https%3A%2F%2Fdev.azure.com%2FmyOrganization%2FmyDevOpsProject%2F_git%2FmyBadRepo%2Finfo%2Frefs%3Fservice%3Dgit-upload-pack&redirect=1&hid=73b36635-d080-433b-9d64-87b14cd2b062&context=eyJodCI6MiwiaGlkIjoiOWE0OWZlYjMtY2FkYS00YWRmLWFkMjItZWZiZWViYWY0NDdhIiwicXMiOnt9LCJyciI6IiIsInZoIjoiIiwiY3YiOiIiLCJjcyI6IiJ90#ctx=eyJTaWduSW5Db29raWVEb21haW5zIjpbImh0dHBzOi8vbG9naW4ubWljcm9zb2Z0b25saW5lLmNvbSIsImh0dHBzOi8vbG9naW4ubWljcm9zb2Z0b25saW5lLmNvbSJdfQ2

Answer 1

Following the suggestion and link from @Philippe, adding the zScaler certificate to my Git configuration solved my problem. After reading through the posts, the exact steps I took were:

• From the Windows State Menu, run the “Manage Computer Certificates” control panel app
• Go to Trusted Root Certification Authorities -> Certificates -> Zscaler Root CA, right-click the certificate to Open it
• On the Details tab, click “Copy to File”
• The Certificate Export Wizard will open, click Next
• Select “Base-64 encoded X.509 (.cer)” and click Next
• Save the file as “C:\Users<username>\AppData\Roaming\ZscalerRootCA.cer”, click Finish, and exit the control panel app
• Open the Command prompt and configure Git to use the certificate, by running the following commands:

- git config --global http.sslbackend openssl
- git config --global http.sslcainfo C:\Users\<username>\AppData\Roaming\ZscalerRootCA.cer

Thanks!

Answer 2

TL;DR: It's definitely a Zscaler issue. The only solution (provided by Zscaler) is to install the Zscaler certificate and configure git to use it.

Download and store certificate in a good place, then configure git to use it by running powershell commands:

gc .\NameOfZscalerRootCertificate.crt| ac $(git config --get http.sslcainfo)
git config --global http.sslBackend openssl

I had exactly the same issue (the bigger is the repo, the higher is the probability to have the curl error but not always) and found the explanation in this answer / alternative link

Copy pasting message here (because Zscaler break link once and I had to find it in Internet Archive, which is very slow to load):

Hello All,

Just some points to note with GIT, curl and Zscaler.

Git uses the underlying curl library for downloading. Due to enforcement of the RFC Zscaler is not compatible with the GNU ssl library and in some cases the Schannel library. It is >recommended to always use the openssl library instead.

On Linux e.g. Ubuntu:

You may see an error like:

error: RPC failed; curl 56 GnuTLS recv error (-110): The TLS connection was non-properly terminated.

A good way to rebuild curl is here: https://github.com/paul-nelson-baker/git-openssl-shellscript/blob/master/compile-git-with-openssl.sh 38

For windows

You may see an error like: error: RPC failed; curl 56 Failure when receiving data from the peer

Sometimes the repository will download, other times such as on larger repositories you may get an issue.

When installing git or curl simply choose the openssl version and import your certificate into the cabundle as per the other guides here.

You may in some cases need to configure git with the global: git config --global http.sslBackend openssl

If you debug git you will see (missing close_notify) in the logs.