How to change the buffer limit in Google's protobuf?

Question

I'm getting this warning and an error afterwards when I try to parse a large message. I know than 64MB which is the default limit. I am using message.ParseFromIstream now. Does any one know to get access to CodedInputStream object to call the SetTotalBytesLimit function? or any other way to solve this problem?

Reading dangerously large protocol message. If the message turns out to be larger than 67108864 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.

Answer 1

The correct fix: You should try to limit the sizes of your protobuf messages. Please see: https://developers.google.com/protocol-buffers/docs/techniques#streaming

The quick and dirty (read not recommended) approach: In the file coded_stream.h of the protobuf library source, change the values of kDefaultTotalBytesLimit and kDefaultTotalBytesWarningThreshold, recompile, and reinstall.

Answer 2

Here's a comment from the code (google/protobuf/io/coded_stream.h) that sets the message limit for those who's wondering what is the security reason they are talking about. In my case I cannot modify how my application work so I have to change this limit.

This thread is quite old, but recently deep learning has got attention and the library Caffe used Protobuf so maybe more people will stumbled upon this. I have to do neural network stuff with Caffe, and the whole network took so much memory even with smallest batch size.

  // Total Bytes Limit -----------------------------------------------
  // To prevent malicious users from sending excessively large messages
  // and causing integer overflows or memory exhaustion, CodedInputStream
  // imposes a hard limit on the total number of bytes it will read.

  // Sets the maximum number of bytes that this CodedInputStream will read
  // before refusing to continue.  To prevent integer overflows in the
  // protocol buffers implementation, as well as to prevent servers from
  // allocating enormous amounts of memory to hold parsed messages, the
  // maximum message length should be limited to the shortest length that
  // will not harm usability.  The theoretical shortest message that could
  // cause integer overflows is 512MB.  The default limit is 64MB.  Apps
  // should set shorter limits if possible.  If warning_threshold is not -1,
  // a warning will be printed to stderr after warning_threshold bytes are
  // read.  For backwards compatibility all negative values get squashed to -1,
  // as other negative values might have special internal meanings.
  // An error will always be printed to stderr if the limit is reached.
  //
  // This is unrelated to PushLimit()/PopLimit().
  //
  // Hint:  If you are reading this because your program is printing a
  //   warning about dangerously large protocol messages, you may be
  //   confused about what to do next.  The best option is to change your
  //   design such that excessively large messages are not necessary.
  //   For example, try to design file formats to consist of many small
  //   messages rather than a single large one.  If this is infeasible,
  //   you will need to increase the limit.  Chances are, though, that
  //   your code never constructs a CodedInputStream on which the limit
  //   can be set.  You probably parse messages by calling things like
  //   Message::ParseFromString().  In this case, you will need to change
  //   your code to instead construct some sort of ZeroCopyInputStream
  //   (e.g. an ArrayInputStream), construct a CodedInputStream around
  //   that, then call Message::ParseFromCodedStream() instead.  Then
  //   you can adjust the limit.  Yes, it's more work, but you're doing
  //   something unusual.

Answer 3

Just reading the documentation of the function that the error already told you about, would've answered that question:

Hint: If you are reading this because your program is printing a warning about dangerously large protocol messages, you may be confused about what to do next. The best option is to change your design such that excessively large messages are not necessary. For example, try to design file formats to consist of many small messages rather than a single large one. If this is infeasible, you will need to increase the limit. Chances are, though, that your code never constructs a CodedInputStream on which the limit can be set. You probably parse messages by calling things like Message::ParseFromString(). In this case, you will need to change your code to instead construct some sort of ZeroCopyInputStream (e.g. an ArrayInputStream), construct a CodedInputStream around that, then call Message::ParseFromCodedStream() instead. Then you can adjust the limit. Yes, it's more work, but you're doing something unusual.

Source

Also it's probably a really good idea to follow the first part of the advice and redesign the application.