How to write a http proxy using netty

Question

I want to write a simple program using netty to proxy http request send by browser. I think it can be divided into 3 steps

1. get request send by browser
2. send it to the website
3. receive data from website and send it back to the browser.

Question:

1. How to translate url into host and port when I using Bootstrap.connect(host, port);
2. When I using HttpServerResponseHandler.connect and ChannelHandlerContext.writeAndFlush(httpMessage); to send data to website, how can I get the response data from the website and send it back to the browser?

It's my first day studying netty, so please try to answer as easy as possible. Thank you very much.

public class Server {
    public static void main(String[] args) throws InterruptedException {
        final int port = 8888;

        // copy from https://github.com/netty/netty/wiki/User-guide-for-4.x
        EventLoopGroup bossGroup = new NioEventLoopGroup();
        EventLoopGroup workerGroup = new NioEventLoopGroup();
        try {
            ServerBootstrap b = new ServerBootstrap();
            b.group(bossGroup, workerGroup)
                    .channel(NioServerSocketChannel.class)
                    .childHandler(new ChannelInitializer<SocketChannel>() {
                        @Override
                        public void initChannel(SocketChannel ch) throws Exception {
                            ch.pipeline().addLast(new HttpRequestDecoder(), new HttpServerRequestHandler());
                        }
                    })
                    .option(ChannelOption.SO_BACKLOG, 128)
                    .childOption(ChannelOption.SO_KEEPALIVE, true);

            // Bind and start to accept incoming connections.
            ChannelFuture f = b.bind(port).sync();

            // Wait until the server socket is closed.
            // In this example, this does not happen, but you can do that to gracefully
            // shut down your server.
            f.channel().closeFuture().sync();
        } finally {
            workerGroup.shutdownGracefully();
            bossGroup.shutdownGracefully();
        }
    }
} 

public class HttpServerRequestHandler extends ChannelInboundHandlerAdapter {

    @Override
    public void channelRead(ChannelHandlerContext ctx, Object msg) {
        // step 1 get data from browser
        if (msg instanceof LastHttpContent) {
            ctx.close();
            return;
        }
        DefaultHttpRequest httpMessage = (DefaultHttpRequest) msg;
        System.out.println("浏览器请求====================");
        System.out.println(msg);
        System.out.println();
        doWork(ctx, httpMessage);
    }

    @Override
    public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
        cause.printStackTrace();
        ctx.close();
    }

    private void doWork(ChannelHandlerContext ctx, final DefaultHttpRequest msg) {
        // step 2 send data to website
        // translate url into host and port
        String host = msg.uri();
        int port = 80;
        if (host.startsWith("https://")) {
            host = host.replaceFirst("https://", "");
            port = 443;
        } else if (host.startsWith("http://")) {
            host = host.replaceFirst("http://", "");
            port = 80;
        }
        if (host.contains(":443")) {
            host = host.replace(":443", "");
            port = 443;
        }

        EventLoopGroup workerGroup = new NioEventLoopGroup();
        try {
            Bootstrap b = new Bootstrap();
            b.group(workerGroup);
            b.channel(NioSocketChannel.class);
            //b.option(ChannelOption.AUTO_READ, true);
            b.handler(new ChannelInitializer<SocketChannel>() {
                @Override
                public void initChannel(SocketChannel ch) throws Exception {
                    ch.pipeline().addLast(new HttpServerResponseHandler(msg), new HttpRequestEncoder());
                }
            });

            // question 1
            ChannelFuture f = b.connect(host, port).sync();
            //ChannelFuture f = b.connect("www.baidu.com", 443).sync();
            f.channel().closeFuture().sync();
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            workerGroup.shutdownGracefully();
        }
    }
}

public class HttpServerResponseHandler extends ChannelOutboundHandlerAdapter {

    private Object httpMessage;

    public HttpServerResponseHandler(Object o) {
        this.httpMessage = o;
    }


    @Override
    public void write(ChannelHandlerContext ctx, Object msg, ChannelPromise promise) {
        System.out.println("网页请求结果=========================");
        System.out.println(httpMessage);
        System.out.println();
    }

    @Override
    public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) { // (4)
        cause.printStackTrace();
        ctx.close();
    }

    @Override
    public void connect(ChannelHandlerContext ctx, SocketAddress remoteAddress,
                        SocketAddress localAddress, ChannelPromise promise) throws Exception {
        System.out.println("connect !!!!!!!!!!!");
        // question 2
        ctx.writeAndFlush(httpMessage);
    }
}

Answer 1

Coincidentally, I've also been working on a Netty proxy server for the purposes of learning. I've a fully working code that you can find on my GitHub, but I'll answer your questions here. Netty also has an official proxy server example here but unlike my code, they don't have unit tests.

(FYI, My code is in Kotlin).

Core Idea:

When creating a proxy server, you need a server to accept client requests, as well as a client for the remote than you are proxying. You created the server, but not the client. It is best to reuse the EventLoop created by the server rather than creating a new one for the client. Each event loop runs on a dedicated thread, so creating more would produce additional threads, necessitating context switching when exchanging data between the accepted Channel and the client Channel.

How to translate url into host and port

To keep things simple, I've used a HttpObjectAggregator that aggregates an HttpMessage and its following HttpContents into a single FullHttpRequest or FullHttpResponse (depending on if it used to handle requests or responses). Setting the URL is trivial: just call FullHttpRequest.setUri.

To get the host and port, call Channel.remoteAddress() on the client channel and cast the resulting SocketAddress to an InetSocketAddress, from which you can get the host and port. Don't forget to similarly reset the Host header if present.

how can I get the response data

After establishing a client channel (that you're missing), you need to make a request on that channel. The client channel has a handler with a reference to the original server channel. Once the handler receives a response, it writes it to the server channel.