Buffer ownership in Netty 4: How is buffer life-cycle managed?

Question

I've been trying to write an HTTP client to fetch multiple feeds concurrently (up to 1k), also as an exercise to learn Netty 4.

My question is, if there is a good explanation somewhere how the new ByteBuf infrastructure works? Who "owns" them, how are they shared (are they?) ? Does every ChannelHandler in a ChannelPipeline has it's own ByteBuf?

Here is an example that left me puzzled:

I added an instance of the following class to a HTTP client pipeline:

public class MyFilter extends MessageToMessageDecoder<HttpObject> {

    @Override
    protected Object decode(ChannelHandlerContext ctx, HttpObject msg) throws Exception {
        // do some work, but leave msg unchanged
        BufUtil.retain(msg); // Why do I need to call BufUtil.retain(msg) ???
        return msg;
}

If I don't call BufUtil.retain on msg, it seems to get GCd and I get all sorts of spurious errors.

Answer 1

HttpContent extends ReferenceCounted to keep track of the life cycle of the buffer it holds. When a ReferenceCounted is instantiated, it starts its life with refCnt of 1. If you call retain() on it, refCnt is increased. refCnt is decreased on release(), and the underlying resource (ByteBuf in this case) is destroyed once refCnt becomes 0.

Usually, a handler does not need to keep a reference to the message it finished handling, because the message is usually thrown away or transformed into something else once handled. Therefore, release() method must be called on the message once your handler is done with it. This is often error-prone and will lead to resource leak very easily.

To avoid the leak that is very hard to track down, extend SimpleChannelInboundHandler which calls release() automatically on the messages once they are handled.

For more information about reference counting in Netty, please read this wiki page. It also gives you the detailed information about how to troubleshooting the buffer leaks by making use of Netty's buffer leak detection mechanism.