Menu
I have a Java client that pushes (INSERT) records in batch to Cassandra cluster. The elements in the batch all have the same row key, so they all will be placed in the same node. Also I don't need the transaction to be atomic so I've been using unlogged batch.
The number of INSERT commands in each batch depends on different factors, but can be anything between 5 to 50000. First I just put as many commands as I had in one batch and submitted it. This threw com.datastax.driver.core.exceptions.InvalidQueryException: Batch too large. Then I used a cap of 1000 INSERT per batch, and then down to 300. I noticed I'm just randomly guessing without knowing exactly where this limit comes from, which can cause trouble down the road.
My question is, what is this limit? Can I modify it? How can I know how many elements can be placed in a batch? When my batch is "full"?
269
The batch statement combines multiple data modification language statements (such as INSERT, UPDATE, and DELETE) to achieve atomicity and isolation when targeting a single partition or only atomicity when targeting multiple partitions.
There is a batch insert operation in Cassandra. You can batch together inserts, even in different column families, to make insertion more efficient. In Hector, you can use HFactory. createMutator then use the add methods on the returned Mutator to add operations to your batch.
I would recommend not increasing the cap, and just splitting into multiple requests. Putting everything in a giant single request will negatively impact the coordinator significantly. Having everything in one partition can improve the throughput in some sized batches by reducing some latency, but batches are never meant to be used to improve performance. So trying to optimize to get maximum throughput by using different batch sizes will depend largely on use case/schema/nodes and will require specific testing, since there's generally a cliff on the size where it starts to degrade.
There is a
# Fail any batch exceeding this value. 50kb (10x warn threshold) by default.
batch_size_fail_threshold_in_kb: 50
option in your cassandra.yaml to increase it, but be sure to test to make sure your actually helping and not hurting your throughput.
52
Looking at the Cassandra logs you'll be able to spot things like:
ERROR 19:54:13 Batch for [matches] is of size 103.072KiB, exceeding specified threshold of 50.000KiB by 53.072KiB. (see batch_size_fail_threshold_in_kb)
25
I fixed this issue by changing the CHUNKSIZE to a lower value (for exemple 1) https://docs.datastax.com/en/cql/3.1/cql/cql_reference/copy_r.html
COPY mytable FROM 'mybackup' WITH CHUNKSIZE = 1;
The operation is much slower but at least it work now
4
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
