Python HTTPS against Azure service management API fails on Windows

Question

I've recently extended a Python API for the Windows Azure storage APIs (PyAzure) to include support for the service management APIs. See https://github.com/bmb/pyazure.

I'm using a HTTPSClientAuthHandler like the one suggested in using pyOpenSSL to create urllib custom opener. On Linux, with various versions of Python 2.6 and 2.7 this works well. However, Windows is another story. All requests against the Azure management host address fail with:

[Errno 10054] An existing connection was forcibly closed by the remote host

Which I think, is the socket errno 10054 "Connection reset by peer", in drag.

This doesn't appear to be a problem in my API code (unless the client cert authentication method I'm using is bogus somehow), but something lower-level. I can reproduce the issue without urllib2 or httplib by simply setting up an SSL socket and sending the same HTTP request down the pipe as urllib2 would, e.g. to list the valid Azure data centre locations:

>>> import socket, ssl, sys
>>> sys.version
'2.7.1 (r271:86832, Nov 27 2010, 17:19:03) [MSC v.1500 64 bit (AMD64)]'

>>> s = ssl.wrap_socket(socket.socket(), certfile='c:\\users\\blair\\research\\clouds\\azure\\BlairBethwaiteAzure1.pfx.pem')
>>> s.connect(('management.core.windows.net',443))
>>> s.send("GET /SUBSCRIPTION_ID/locations HTTP/1.1\r\nAccept-Encoding: identity\r\nX-Ms-Version: 2011-10-01\r\nHost: management.core.windows.net\r\nConnection: close\r\nUser-Agent: Python-urllib/2.6\r\n\r\n")
202

>>> s.read()
Traceback (most recent call last):
c:\Users\blair\research\clouds\azure\pyazure\<ipython-input-63-3306c981d8a7>
in <module>()
----> 1 s.read()

C:\Python27\lib\ssl.pyc in read(self, len)
   136
   137         try:
--> 138             return self._sslobj.read(len)
   139         except SSLError, x:
   140             if x.args[0] == SSL_ERROR_EOF and self.suppress_ragged_eofs:


error: [Errno 10054] An existing connection was forcibly closed by the remote host

Replace SUBSCRIPTION_ID above, with your Azure subscription ID. The exception is raised ~45s after calling SSLSocket.read. The cert is a properly formatted PEM file including both the private key and certificate, it was converted from the pfx (in Ubuntu 10.04) using:

openssl pkcs12 -in pfxfile -out pemfile -nodes

I don't think it matters here, but I also tried unix2dos-ing the PEM file, to no avail though. I get the same behaviour even when I don't provide any cert, but doing that on Linux results in a proper API error from the server:

'HTTP/1.1 403 Forbidden\r\nContent-Length: 0\r\nServer: Microsoft-HTTPAPI/2.0\r\nDate: Thu, 01 Dec 2011 13:59:29 GMT\r\nConnection: close\r\n\r\n'

This has been independently verified by another person using Windows 7 (same as me). It's not a client-side firewall issue - the same code works in a NAT-ed Linux VM running on the same host.

I'm stumped. Would really appreciate any help folks here might be able to provide...

Update: This appears to be related to the underlying SSL implementation in Python. CPython 2.7.1 has the error behaviour as shown above, but I've since tested and had success with ActiveState Python (both 2.7 and 2.6), e.g.:

>>> import sys, socket, ssl
>>> sys.version
'2.7.1 (r271:86832, Feb  7 2011, 11:30:38) [MSC v.1500 32 bit (Intel)]'
>>> s = ssl.wrap_socket(socket.socket(), certfile='\\\\VBOXSVR\\azure\\BlairBethwaiteAzure1.pfx.pem')
>>> s.connect(('management.core.windows.net',443))
>>> s.send('GET /SUBSCRIPTION_ID/locations HTTP/1.1\r\nAccept-Encoding: identity\r\nX-Ms-Version: 2011-10-01\r\nHost: management.core.windows.net\r\nUser-Agent: Python-urllib/2.6\r\n\r\n')
183

>>> s.read(4096)
'HTTP/1.1 200 OK\r\nContent-Length: 908\r\nContent-Type: application/xml; charset=utf-8\r\nServer: Microsoft-HTTPAPI/2.0\r\nx-ms-request-id: 08ca048cda6b445da6b3a8f3e4890197\r\nDate: Fri, 02 Dec 2011 03:02:14 GMT\r\n\r\n<Locations xmlns="http://schemas.microsoft.com/windowsazure" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><Location><Name>Anywhere US</Name><DisplayName>Anywhere US</DisplayName></Location><Location><Name>South Central US</Name><DisplayName>South Central US</DisplayName></Location><Location><Name>North Central US</Name><DisplayName>North Central US</DisplayName></Location><Location><Name>Anywhere Europe</Name><DisplayName>Anywhere Europe</DisplayName></Location><Location><Name>North Europe</Name><DisplayName>North Europe</DisplayName></Location><Location><Name>West Europe</Name><DisplayName>West Europe</DisplayName></Location><Location><Name>Anywhere Asia</Name><DisplayName>Anywhere Asia</DisplayName></Location><Location><Name>Southeast Asia</Name><DisplayName>Southeast Asia</DisplayName></Location><Location><Name>East Asia</Name><DisplayName>East Asia</DisplayName></Location></Locations>'

And as expected my API works too:

ActivePython 2.6.7.20 (ActiveState Software Inc.) based on
Python 2.6.7 (r267:88850, Jun 27 2011, 13:20:48) [MSC v.1500 64 bit (AMD64)] on
win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from pyazure import pyazure
>>> pa = pyazure.PyAzure(subscription_id=SUBSCRIPTION_ID, management_cert_path='c:\\users\\blair\\research\\clouds\\azure\\BlairBethwaiteAzure1.pfx.pem')
>>> list(pa.wasm.list_locations())
['Anywhere US', 'South Central US', 'North Central US', 'Anywhere Europe', 'North Europe', 'West Europe', 'Anywhere Asia', 'Southeast Asia', 'East Asia']

The Lib\ssl.py files in CPython2.7 and ActivePython2.7 are identical, so I guess this must be due to some difference in the underlying C libs, perhaps a bug in CPython. Any gurus out there?

Answer 1

I haven't been able to pin down a definitive explanation for this, but after a bit of trial and error I'm confident of where the issue lays...

Short answer: it's the ssl implementation in the http://www.python.org/ Windows bundle. Use ActiveState Python instead.

Long Answer: The Windows CPython distributions available from http://www.python.org/download/ bundle quite an old version of OpenSSL (0.9.8l), compared with the ActiveState Python distributions which are based on CPython but (amongst other things) provide regular updates to 3rd party inclusions such as OpenSSL (currently 0.9.8r).

I downloaded Windows binaries of OpenSSL and tested via the openssl s_client interface, e.g.:

openssl s_client -connect management.core.windows.net:443 -cert /home/blair/nimrod-dev/BlairBethwaiteAzure1.pfx.pem

The current version works, as expected. Unfortunately it seems difficult to get one's hands on old OpenSSL binaries for Windows, perhaps not surprising given it is a security library... But anyway, I built 0.9.8l from source under Ubuntu 10.04 and found that it hangs after sending a HTTP request down the pipe, presumably the server silently dropped the connection for some reason:

blair@venus-vm:~/Downloads/openssl-0.9.8l/apps$ ./openssl s_client -connect management.core.windows.net:443 -cert ./BlairAzure.pem 
CONNECTED(00000003)
depth=2 /CN=Microsoft Internet Authority
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/CN=management.core.windows.net
   i:/DC=com/DC=microsoft/DC=corp/DC=redmond/CN=Microsoft Secure Server Authority
 1 s:/DC=com/DC=microsoft/DC=corp/DC=redmond/CN=Microsoft Secure Server Authority
   i:/CN=Microsoft Internet Authority
 2 s:/CN=Microsoft Internet Authority
   i:/C=US/O=GTE Corporation/OU=GTE CyberTrust Solutions, Inc./CN=GTE CyberTrust Global Root
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIGhDCCBWygAwIBAgIKFnL3ogAIAAIjlDANBgkqhkiG9w0BAQUFADCBizETMBEG
CgmSJomT8ixkARkWA2NvbTEZMBcGCgmSJomT8ixkARkWCW1pY3Jvc29mdDEUMBIG
CgmSJomT8ixkARkWBGNvcnAxFzAVBgoJkiaJk/IsZAEZFgdyZWRtb25kMSowKAYD
VQQDEyFNaWNyb3NvZnQgU2VjdXJlIFNlcnZlciBBdXRob3JpdHkwHhcNMTEwNjE2
MDg0MjI3WhcNMTMwNjE1MDg0MjI3WjAmMSQwIgYDVQQDExttYW5hZ2VtZW50LmNv
cmUud2luZG93cy5uZXQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCS
Z9PTUqQLh5keX/IRJ6JxaQkVBIy/iyoCIx2Y0zy5F5tll8CRydGzFDjXMLWEG425
EuuRDQrBgQnmVtlZ2t42QfIRBSvfJheZVh8k27g/tH5wpchZ47gxxatUKsVJ84P8
Me2S0RP9xtk3P14dVqTDJIhUC3k8JYdBiTTtk64EB5Dbq8sxEtjVb/68XDgTZKek
te/vqWSW/KcduKEjsfjOwNSM9UbYrFOTbelac+mf/L+CluAJpYAlIhOMUP2afy5e
tYIg6zK04pDNjPjizpfN3xrGX7NRY16kaafrdqJQixfmEVlMDN8FsXLWDweXWfMM
XRh4vuAVb6tA9wBkPDhZAgMBAAGjggNMMIIDSDALBgNVHQ8EBAMCBLAwHQYDVR0l
BBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMHgGCSqGSIb3DQEJDwRrMGkwDgYIKoZI
hvcNAwICAgCAMA4GCCqGSIb3DQMEAgIAgDALBglghkgBZQMEASowCwYJYIZIAWUD
BAEtMAsGCWCGSAFlAwQBAjALBglghkgBZQMEAQUwBwYFKw4DAgcwCgYIKoZIhvcN
AwcwHQYDVR0OBBYEFEaKqx6Auvu3fvHS6KqQl8KXoOoAMB8GA1UdIwQYMBaAFAhC
49tOEWbztQjFQNtVfDNGEYM4MIIBCgYDVR0fBIIBATCB/jCB+6CB+KCB9YZYaHR0
cDovL21zY3JsLm1pY3Jvc29mdC5jb20vcGtpL21zY29ycC9jcmwvTWljcm9zb2Z0
JTIwU2VjdXJlJTIwU2VydmVyJTIwQXV0aG9yaXR5KDgpLmNybIZWaHR0cDovL2Ny
bC5taWNyb3NvZnQuY29tL3BraS9tc2NvcnAvY3JsL01pY3Jvc29mdCUyMFNlY3Vy
ZSUyMFNlcnZlciUyMEF1dGhvcml0eSg4KS5jcmyGQWh0dHA6Ly9jb3JwcGtpL2Ny
bC9NaWNyb3NvZnQlMjBTZWN1cmUlMjBTZXJ2ZXIlMjBBdXRob3JpdHkoOCkuY3Js
MIG/BggrBgEFBQcBAQSBsjCBrzBeBggrBgEFBQcwAoZSaHR0cDovL3d3dy5taWNy
b3NvZnQuY29tL3BraS9tc2NvcnAvTWljcm9zb2Z0JTIwU2VjdXJlJTIwU2VydmVy
JTIwQXV0aG9yaXR5KDgpLmNydDBNBggrBgEFBQcwAoZBaHR0cDovL2NvcnBwa2kv
YWlhL01pY3Jvc29mdCUyMFNlY3VyZSUyMFNlcnZlciUyMEF1dGhvcml0eSg4KS5j
cnQwPwYJKwYBBAGCNxUHBDIwMAYoKwYBBAGCNxUIg8+JTa3yAoWhnwyC+sp9geH7
dIFPg8LthQiOqdKFYwIBZAIBCjAnBgkrBgEEAYI3FQoEGjAYMAoGCCsGAQUFBwMC
MAoGCCsGAQUFBwMBMCYGA1UdEQQfMB2CG21hbmFnZW1lbnQuY29yZS53aW5kb3dz
Lm5ldDANBgkqhkiG9w0BAQUFAAOCAQEAsqHBR/JxRnGQMTXxJzCau49dDgeum1JH
heA38lzsoUaRELHxxrQZskjSqc0HrI7cnJPSipWQseDDwKtLwXzukCdZNk84u7xo
uHa7/dmxo1m+z353HSvEr85ZE2mzwF6qmwGMmvvVzIJ94M8fcN55yoF64vQsAWFF
k2QJC9ccb8eDoTs5NX4ntpz02xf8eEBQ5yKZySfi3+oFJEUnLmXcvHTTMl/1N/NI
fWiKIZ9PDTBlPxL5kNJ/aDGIgiqCi7Vm7KfjvWSFhopUPtVeeItgW9wMLEkuQsw6
sViSbU50CMPWTJAslLZgCju6cxszgpLl19xrgNteHRw2HouwTTsJnA==
-----END CERTIFICATE-----
subject=/CN=management.core.windows.net
issuer=/DC=com/DC=microsoft/DC=corp/DC=redmond/CN=Microsoft Secure Server Authority
---
No client certificate CA names sent
---
SSL handshake has read 4691 bytes and written 450 bytes
---
New, TLSv1/SSLv3, Cipher is AES128-SHA
Server public key is 2048 bit
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES128-SHA
    Session-ID: <SNIP>
    Session-ID-ctx: 
    Master-Key: <SNIP>
    Key-Arg   : None
    Start Time: 1324443511
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---
GET /<SUBSCRIPTION_ID>/locations HTTP/1.1
Accept-Encoding: identity
X-Ms-Version: 2011-10-01
Host: management.core.windows.net
Connection: close

Under newer and even slightly older (e.g., Ubuntu10.04's 0.9.8e) OpenSSLs the server responds to the request with the expected:

<Locations xmlns="http://schemas.microsoft.com/windowsazure" xmlns:i="http://www.w3.org/2001/XMLSchema-instance"><Location><Name>Anywhere US</Name><DisplayName>Anywhere US</DisplayName></Location><Location><Name>South Central US</Name><DisplayName>South Central US</DisplayName></Location><Location><Name>Anywhere Europe</Name><DisplayName>Anywhere Europe</DisplayName></Location><Location><Name>West Europe</Name><DisplayName>West Europe</DisplayName></Location><Location><Name>Anywhere Asia</Name><DisplayName>Anywhere Asia</DisplayName></Location><Location><Name>Southeast Asia</Name><DisplayName>Southeast Asia</DisplayName></Location><Location><Name>East Asia</Name><DisplayName>East Asia</DisplayName></Location><Location><Name>North Central US</Name><DisplayName>North Central US</DisplayName></Location><Location><Name>North Europe</Name><DisplayName>North Europe</DisplayName></Location></Locations>

But with OpenSSL 0.9.8l I get nothing.

Answer 2

The following works as expected using IronPython 2.7.1 on Windows 7 and CPython 2.6.6 on OS X 10.6.8:

import socket, ssl, sys

sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(('management.core.windows.net',443)) 

s = ssl.wrap_socket(sock, certfile=sys.argv[1])
s.send('GET /SUBSCRIPTION_ID/locations HTTP/1.1\r\nAccept-Encoding: identity\r\nX-Ms-Version: 2011-10-01\r\nHost: management.core.windows.net\r\nUser-Agent: Python-urllib/2.6\r\n\r\n')

print(s.read(4096))

[NOTE: I'm passing MYKEYFILENAME.pem as a command-line parameter.]

Happy Azure hacking!