Cassandra resets connection under heavy insert workload

Question

I have a cassandra cluster of 8 nodes, cassandra 1.0.8.

I'm trying to do lots of small inserts in a loop with batch_mutate(). After some time (~200K inserts) server resets connection with following exception:

org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:147)
at org.apache.thrift.transport.TFramedTransport.flush(TFramedTransport.java:157)
at org.apache.cassandra.thrift.Cassandra$Client.send_batch_mutate(Cassandra.java:998)
at org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:986)
at org.scale7.cassandra.pelops.Mutator$1.execute(Mutator.java:46)
at org.scale7.cassandra.pelops.Mutator$1.execute(Mutator.java:42)
at org.scale7.cassandra.pelops.Operand.tryOperation(Operand.java:56)
at org.scale7.cassandra.pelops.Mutator.execute(Mutator.java:51)
    ...
    Caused by: java.net.SocketException: Connection reset
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:145)
... 25 more

Otherwise than it, the cluster works fine. Server logs are clean.

What might be causing this problem?

Thanks!

Answer 1

I've found the cause: batch mutation size was exceeding the frame size of TFramedTransport. There might two possible solutions: either to increase "thrift_framed_transport_size_in_mb" configuration property in cassandra.yaml, or to use smaller batch size.