How to trace overhead added by Application Request Routing?

Question

We have a Delphi SOAP service which needs to be SSL-enabled. I opted to use an IIS ARR reverse proxy to do SSL offloading for ease of configuration (compared to OpenSSL and manual certificate + passphrase management). ARR works, but it adds an insane amount of overhead... Response time went from under 2 seconds to 19 seconds for 18 service requests (about 60Kb compressed total).

I added timestamp logging to client & server for when messages are both sent and received. It shows about 1 second added to each request routing through ARR between sending from the client and receipt by the service. The response is routed back very quickly, only the request routing via ARR is slow (see image below).

How can I trace the source of overhead? Is ARR not suited to this use case? I tried tweaking and disabling most settings, including caching. I tried different hosts with clean IIS setups, including a production Windows Server 2012. SSL itself is not the overhead, just having an ARR HTTP reverse proxy causes the delay.

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <rewrite>
            <rules>
                <rule name="ReverseProxyInboundRule1" stopProcessing="true">
                    <match url="(.*)" />
                    <action type="Rewrite" url="http://localhost:8987/{R:1}" />
                </rule>
            </rules>
        </rewrite>
    </system.webServer>
</configuration>

Request & response samples from Fiddler:

• Raw HTTP Request
• Raw HTTP Response

Answer 1

We have the same problem. I found the root, it's in System.Net.Sockets.Socket.DoConnect The problem is related to IPv6:

https://social.msdn.microsoft.com/Forums/vstudio/en-US/203b6230-e4c0-477c-9a0a-0c21a7ad1615/strange-onesecond-delay-with-tcpconnections-to-localhost?forum=clr

http://msdn.microsoft.com/en-us/library/115ytk56.aspx

"If IPv6 is enabled and the TcpClient(String, Int32) method is called to connect to a host that resolves to both IPv6 and IPv4 addresses, the connection to the IPv6 address will be attempted first before the IPv4 address. This may have the effect of delaying the time to establish the connection if the host is not listening on the IPv6 address."

To resolve it for loopback requests you need to disable IPv6 on a machine, see p.4-5-6: https://stackoverflow.com/a/12403731