Using SwiftNIO and SwiftNIOHTTP2 as an HTTP2 client

Question

I'm currently working on a simple HTTP2 client in Swift using SwiftNIO and the SwiftNIOHTTP2 beta. My implementation looks like this:

let group = MultiThreadedEventLoopGroup(numberOfThreads: 1)
let bootstrap = ClientBootstrap(group: group)
    .channelOption(ChannelOptions.socket(SocketOptionLevel(SOL_SOCKET), SO_REUSEADDR), value: 1)
    .channelInitializer { channel in
        channel.pipeline.add(handler: HTTP2Parser(mode: .client)).then {
            let multiplexer = HTTP2StreamMultiplexer { (channel, streamID) -> EventLoopFuture<Void> in
                return channel.pipeline.add(handler: HTTP2ToHTTP1ClientCodec(streamID: streamID, httpProtocol: .https))
            }
            return channel.pipeline.add(handler: multiplexer)
        }
}

defer {
    try! group.syncShutdownGracefully()
}

let url = URL(string: "https://strnmn.me")!

_ = try bootstrap.connect(host: url.host!, port: url.port ?? 443)
    .wait()

Unfortunately the connection always fails with an error:

nghttp2 error: Remote peer returned unexpected data while we expected SETTINGS frame. Perhaps, peer does not support HTTP/2 properly.

However, connecting and issuing a simple request using nghttp2 from the command line works fine.

$ nghttp -vn https://strnmn.me
[  0.048] Connected
The negotiated protocol: h2
[  0.110] recv SETTINGS frame <length=18, flags=0x00, stream_id=0>
          (niv=3)
          [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):128]
          [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65536]
          [SETTINGS_MAX_FRAME_SIZE(0x05):16777215]
[  0.110] recv WINDOW_UPDATE frame <length=4, flags=0x00, stream_id=0>
          (window_size_increment=2147418112)
[  0.110] send SETTINGS frame <length=12, flags=0x00, stream_id=0>
          (niv=2)
          [SETTINGS_MAX_CONCURRENT_STREAMS(0x03):100]
          [SETTINGS_INITIAL_WINDOW_SIZE(0x04):65535]
[  0.110] send SETTINGS frame <length=0, flags=0x01, stream_id=0>
          ; ACK
          (niv=0)
[  0.110] send PRIORITY frame <length=5, flags=0x00, stream_id=3>
          (dep_stream_id=0, weight=201, exclusive=0)
[  0.110] send PRIORITY frame <length=5, flags=0x00, stream_id=5>
          (dep_stream_id=0, weight=101, exclusive=0)
[  0.110] send PRIORITY frame <length=5, flags=0x00, stream_id=7>
          (dep_stream_id=0, weight=1, exclusive=0)
[  0.110] send PRIORITY frame <length=5, flags=0x00, stream_id=9>
          (dep_stream_id=7, weight=1, exclusive=0)
[  0.110] send PRIORITY frame <length=5, flags=0x00, stream_id=11>
          (dep_stream_id=3, weight=1, exclusive=0)
[  0.111] send HEADERS frame <length=35, flags=0x25, stream_id=13>
          ; END_STREAM | END_HEADERS | PRIORITY
          (padlen=0, dep_stream_id=11, weight=16, exclusive=0)
          ; Open new stream
          :method: GET
          :path: /
          :scheme: https
          :authority: strnmn.me
          accept: */*
          accept-encoding: gzip, deflate
          user-agent: nghttp2/1.34.0
[  0.141] recv SETTINGS frame <length=0, flags=0x01, stream_id=0>
          ; ACK
          (niv=0)
[  0.141] recv (stream_id=13) :status: 200
[  0.141] recv (stream_id=13) server: nginx
[  0.141] recv (stream_id=13) date: Sat, 24 Nov 2018 16:29:13 GMT
[  0.141] recv (stream_id=13) content-type: text/html
[  0.141] recv (stream_id=13) last-modified: Sat, 01 Jul 2017 20:23:11 GMT
[  0.141] recv (stream_id=13) vary: Accept-Encoding
[  0.141] recv (stream_id=13) etag: W/"595804af-8a"
[  0.141] recv (stream_id=13) expires: Sat, 24 Nov 2018 16:39:13 GMT
[  0.141] recv (stream_id=13) cache-control: max-age=600
[  0.141] recv (stream_id=13) x-frame-options: SAMEORIGIN
[  0.141] recv (stream_id=13) content-encoding: gzip
[  0.141] recv HEADERS frame <length=185, flags=0x04, stream_id=13>
          ; END_HEADERS
          (padlen=0)
          ; First response header
[  0.142] recv DATA frame <length=114, flags=0x01, stream_id=13>
          ; END_STREAM
[  0.142] send GOAWAY frame <length=8, flags=0x00, stream_id=0>
          (last_stream_id=0, error_code=NO_ERROR(0x00), opaque_data(0)=[])

How can I establish a session and issue a GET request using SwiftNIOHTTP2?

Answer 1

That's a very good question! Let's first analyse why this is more complicated than sending a HTTP/1.x request. Broadly speaking these issues fall into two categories:

1. NIO at the moment makes it more complicated than necessary, therefore much of what I'll write further down can be unintuitive at times. I'm one of the NIO core team and even I had to dig through quite a bit of code to get this fully working, mostly because we still don't have doc generation for swift-nio-ssl and swift-nio-http2 on http://docs.swiftnio.io .
2. HTTP/2 is just much more complicated than HTTP/1 and NIO is more a toolbox that can be used to build HTTP clients so we need to use a bunch of tools together to get it all working.

I'll focus on the necessary complexity (2) here and will file bugs/fixes for (1). Let's check what tools we need from the NIO toolbox to get this working:

1. TLS. No real-world HTTP/2 server will allow you to speak HTTP/2 over plaintext
2. ALPN. HTTP/1 and HTTP/2 share the same port (usually 443) so we need to tell the server that we want to speak HTTP/2 because for backwards compatibility the default remains HTTP/1. We can do this using a mechanism called ALPN (Application-layer Protocol Negotiation), the other option would be to perform a HTTP/1 upgrade to HTTP2 but that's both more complicated and less performant so let's not do this here
3. some HTTP/2 tools: a) open a new HTTP/2 b) HTTP/2 to HTTP/1 message translation c) HTTP/2 multiplexing

The code in your question contains the most important bits, namely 3b and 3c of the above list. But we need to add 1, 2 and 3a so let's do this :)

Let's start with 2) ALPN:

let tlsConfig = TLSConfiguration.forClient(applicationProtocols: ["h2"])
let sslContext = try SSLContext(configuration: tlsConfig)

This is an SSL configuration with the "h2" ALPN protocol identifier there which will tell the server that we want to speak HTTP/2 as documented in the HTTP/2 spec.

Ok, let's add TLS with the sslContext set up before:

let sslHandler = try! OpenSSLClientHandler(context: sslContext, serverHostname: hostname)

It's also important that we tell the OpenSSLClientHandler the server's hostname so it can validate the certificate properly.

Lastly we need to do 3a (creating a new HTTP/2 stream to issue our request on) which can be easily done using a ChannelHandler:

/// Creates a new HTTP/2 stream when our channel is active and adds the `SendAGETRequestHandler` so a request is sent.
final class CreateRequestStreamHandler: ChannelInboundHandler {
    typealias InboundIn = Never

    private let multiplexer: HTTP2StreamMultiplexer
    private let responseReceivedPromise: EventLoopPromise<[HTTPClientResponsePart]>

    init(multiplexer: HTTP2StreamMultiplexer, responseReceivedPromise: EventLoopPromise<[HTTPClientResponsePart]>) {
        self.multiplexer = multiplexer
        self.responseReceivedPromise = responseReceivedPromise
    }

    func channelActive(ctx: ChannelHandlerContext) {
        func requestStreamInitializer(channel: Channel, streamID: HTTP2StreamID) -> EventLoopFuture<Void> {
            return channel.pipeline.addHandlers([HTTP2ToHTTP1ClientCodec(streamID: streamID, httpProtocol: .https),
                                                 SendAGETRequestHandler(responseReceivedPromise: self.responseReceivedPromise)],
                                                first: false)
        }

        // this is the most important line: When the channel is active we add the `HTTP2ToHTTP1ClientCodec` to deal in HTTP/1 messages as well as the `SendAGETRequestHandler` which will send a request.
        self.multiplexer.createStreamChannel(promise: nil, requestStreamInitializer)
    }
}

Okay, that's the scaffolding done. The SendAGETRequestHandler is the last part which is a handler that will be added as soon as the new HTTP/2 stream that we have opened before has been opened successfully. To see the full response, I also implemented accumulating all bits of the response together into a promise:

/// Fires off a GET request when our stream is active and collects all response parts into a promise.
///
/// - warning: This will read the whole response into memory and delivers it into a promise.
final class SendAGETRequestHandler: ChannelInboundHandler {
    typealias InboundIn = HTTPClientResponsePart
    typealias OutboundOut = HTTPClientRequestPart

    private let responseReceivedPromise: EventLoopPromise<[HTTPClientResponsePart]>
    private var responsePartAccumulator: [HTTPClientResponsePart] = []

    init(responseReceivedPromise: EventLoopPromise<[HTTPClientResponsePart]>) {
        self.responseReceivedPromise = responseReceivedPromise
    }

    func channelActive(ctx: ChannelHandlerContext) {
        assert(ctx.channel.parent!.isActive)
        var reqHead = HTTPRequestHead(version: .init(major: 2, minor: 0), method: .GET, uri: "/")
        reqHead.headers.add(name: "Host", value: hostname)
        ctx.write(self.wrapOutboundOut(.head(reqHead)), promise: nil)
        ctx.writeAndFlush(self.wrapOutboundOut(.end(nil)), promise: nil)
    }

    func channelRead(ctx: ChannelHandlerContext, data: NIOAny) {
        let resPart = self.unwrapInboundIn(data)
        self.responsePartAccumulator.append(resPart)
        if case .end = resPart {
            self.responseReceivedPromise.succeed(result: self.responsePartAccumulator)
        }
    }
}

To finish it up, let's set up the client's channel pipeline:

let bootstrap = ClientBootstrap(group: group)
    .channelInitializer { channel in
        let myEventLoop = channel.eventLoop
        let sslHandler = try! OpenSSLClientHandler(context: sslContext, serverHostname: hostname)
        let http2Parser = HTTP2Parser(mode: .client)
        let http2Multiplexer = HTTP2StreamMultiplexer { (channel, streamID) -> EventLoopFuture<Void> in
            return myEventLoop.newSucceededFuture(result: ())
        }
        return channel.pipeline.addHandlers([sslHandler,
                                             http2Parser,
                                             http2Multiplexer,
                                             CreateRequestStreamHandler(multiplexer: http2Multiplexer,
                                                                        responseReceivedPromise: responseReceivedPromise),
                                             CollectErrorsAndCloseStreamHandler(responseReceivedPromise: responseReceivedPromise)],
                                            first: false).map {

        }
}

To see a fully working example, I put something together a PR for swift-nio-examples/http2-client.

Oh, and the reason that NIO was claiming that the other end isn't speaking HTTP/2 properly was the lack of TLS. There was no OpenSSLHandler so NIO was speaking plaintext HTTP/2 to a remote end which was speaking TLS and then the two peers don't understand each other :).