Efficient boolean reductions `any`, `all` for PySpark RDD?

Question

PySpark supports common reductions like sum, min, count, ... Does it support boolean reductions like all and any?

I can always fold over or_ and and_ but this seems inefficient.

Answer 1

this is very late, but all on a set of boolean values z is the same as min(z) == True and any is the same as max(z) == True

Answer 2

No the underlying Scala API doesn't have it so the Python one definitely won't. I don't think they will add it either as it's very easy to define in terms of filter.

Yes using fold would be inefficient because it won't parallelelize. Do something like .filter(!condition).take(1).isEmpty to mean .forall(condition) and .filter(condition).take(1).nonEmpty to mean .exists(condition)

(General suggestion: the underlying Scala API is generally more flexible than Python API, suggest you move to it - it also makes debugging much easier as you have less layers to dig through. Scala means Scalable Language - it's much better for scalable applications and more robust than dynamically typed languages)