golang grpc transport.newBufWriter and bufio.NewReaderSize not releasing memory

Question

I have a simple grpc server in golang which does CRUD operations on an object. However, when I run it the memory never goes down even after requests stop. pprof of heap show has the following result:

> flat  flat%   sum%        cum   cum%
>   932.39MB 62.45% 62.45%   932.39MB 62.45%  google.golang.org/grpc/internal/transport.newBufWriter
>   463.13MB 31.02% 93.46%   463.13MB 31.02%  bufio.NewReaderSize
>    13.50MB   0.9% 94.37%    13.50MB   0.9%  runtime.malg
>       13MB  0.87% 95.24%  1420.52MB 95.14%  google.golang.org/grpc/internal/transport.newHTTP2Server
>       11MB  0.74% 95.98%    12.10MB  0.81%  time.NewTimer
>     8.50MB  0.57% 96.54%     8.50MB  0.57%  golang.org/x/net/http2/hpack.(*headerFieldTable).addEntry
>     5.50MB  0.37% 96.91%    17.60MB  1.18%  google.golang.org/grpc/internal/transport.(*http2Server).keepalive
>     3.50MB  0.23% 97.15%     7.50MB   0.5%  google.golang.org/grpc/internal/transport.newLoopyWriter
>     1.50MB   0.1% 97.25%    12.50MB  0.84%  google.golang.org/grpc.(*Server).serveStreams
>          0     0% 97.25%       10MB  0.67%  golang.org/x/net/http2.(*Framer).ReadFrame

Can anyone guide me on how to go about fixing this memory issue? The server runs with default options and I have even enabled debug.FreeOSMemory() function to release memory.

Answer 1

Most likely you need to close ClientConn, or reuse it. I encountered the same issue and the problem was creating new ClientConn for each RPC call without closing these connection later.

Great explanation here pooling-grpc-connections

Related question GRPC Connection Management in Golang