PHP stream wrappers and Windows Certificate Store with Proxy

Question

Setup/Environment:

In our PHP application, we sometimes need to make HTTPS requests from PHP to other servers. The setup in question is as follows:

• We are using PHP stream wrappers for doing the HTTP requests (using Guzzle HTTP). We are doing this because stream wrappers support using the Windows Certficiate Store for certificate verification.
• The server runs on Windows.
• We use a proxy on for the HTTPS requests.
• The firewalls are configured to allow
  1. Access to the servers we are doing our requests to.
  2. Access to all certificate revocation lists relevant for the certificates used.

Our problem:

Sometimes, out of the blue, our HTTPS requests fail, with certificate validation errors. This problem persists, until someone opens a remote desktop session to the server and requests the very same URL we are trying to query in the servers Internet Explorer. After that, our PHP application can do its requests as it should.

Question:

What is the problem here? And what can we do to analyse this further?

Answer 1

If that were a Guzzle problem, it would happen every time.

However, do try to issue the same HTTPS call using cURL to both verify this is the case, and see if by any chance the cURL request also temporarily clears the issue, just as Internet Explorer does.

But this rather looks like a caching problem - the PHP server request is not able to properly access (priming certificates) the Certificate Store, it is only able to use its services after someone else has gained access, and only as long as the cache does not expire. To be sure this is the case, simply issue calls periodically and mark the time elapsed between user logging in and using IE, and Guzzle calls starting to fail. If I am right, that time will always be the same.

It could be a permission problem (I think it probably is, but what permissions to give to what, that I'm at a loss to guess at). Maybe calls aren't allowed unless fresh CRLs for that URL are available, and PHP doesn't get them). This situation could also either be fixed temporarily by running a IE connection attempt to the same URL from a PowerShell script launched by PHP in case of error, or (more likely, and hopefully) attempting to run said script will elicit some more informative error message.

update

I have looked into how PHP on Windows handles TLS through Guzzle, and nothing obvious came out. But I found an interesting page about TLS/SSL quirks.

More interestingly, I also found out several references on how PHP ends up using Schannel for TLS connections, and how Windows and specifically Internet Explorer have a, let us say, cavalier attitude about interoperability. So I would suggest you try activating the Schannel log on Windows and see whether anything comes out of it.

Additionally, on the linked page there is a reference to a client cache being used, and the related page ends up here ("ClientCacheTime").

Answer 2

Its not an application problem.

I am 99% sure this is routing problem and in some circumstances packets are dropped in the router. I would look at the network, change the environment or, if possible, do some network sniffing or monitoring.

If You have a decent network infrastructure, You can do SNPM traps for request count and timeout data collecting (from routers and switches) and ingest it in Elastic APM. This would give You quite detailed time-series analysis.