Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Java 8 Stream batch processing [duplicate]

I have a Stream of MyObject that I want batch persist into DB (not one by one but let's say 1000 at once). So I want to do a transformation, something like

Stream<MyObject> ---> Stream<List<MyObject>>

where each List has some fixed size batchSize. Is there a way to do that with standard Java 8 Stream API?

like image 652
Andrii Karaivanskyi Avatar asked Jan 27 '26 17:01

Andrii Karaivanskyi


1 Answers

Edit: the original solution below does not work, since java stream does not allow calling skip or limit more than once on a same stream. I ended up simple processing like

    final AtomicInteger counter = new AtomicInteger();

    List<T> entityBatch = new ArrayList<>();

    entityStream.forEach(entity -> {
        if (counter.intValue() = batchSize) {
            processBatch(entityBatch);

            entityBatch.clear();
            counter.set(0);
        }

        entityBatch.add(entity);
        counter.incrementAndGet();
    });

    if (!entityBatch.isEmpty()) {
        processBatch(entityBatch);
    }

Original solution: It looks like I found the way to do that:

<T> Stream<List<T>> batchStream(Stream<T> stream, int batchSize) {
    return Stream.iterate(stream, s -> s.skip(batchSize)).map(s -> s.limit(batchSize).collect(toList()));
}
like image 164
Andrii Karaivanskyi Avatar answered Jan 30 '26 10:01

Andrii Karaivanskyi



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!