How to re-partition pyspark dataframe?

Question

data.rdd.getNumPartitions() # output 2456

Then I do
data.rdd.repartition(3000) But
data.rdd.getNumPartitions() # output is still 2456

How to change number of partitions. One approach can be first convert DF into rdd,repartition it and then convert rdd back to DF. But this takes a lot of time. Also does increasing number of partitions make operations more distributed and so more fast? Thanks

Answer 1

You can check the number of partitions:

data.rdd.partitions.size

To change the number of partitions:

newDF = data.repartition(3000)

You can check the number of partitions:

newDF.rdd.partitions.size

Beware of data shuffle when repartitionning and this is expensive. Take a look at coalesce if needed.

Answer 2

print df.rdd.getNumPartitions()
# 1


df.repartition(5)
print df.rdd.getNumPartitions()
# 1


df = df.repartition(5)
print df.rdd.getNumPartitions()
# 5

see Spark: The definitive Guide chapter 5- Basic Structure Operations
ISBN-13: 978-1491912218
ISBN-10: 1491912219