polars dataFrame.sort() - large memory requirements

Question

When I call the DataFrame.sort() from the Python-polars library, the RAM jumps to more then double of its original values. This is of course a problem when dealing with large datasets (we are talking tens to hundereds of GB). Is there any workaround that (even in cost of performance) cost less RAM?

Thank you for any hints.

Answer 1

According to the Polars documentation, you can use the Streaming API not to run out your RAM. As shown by @Dean MacGregor, you can associate the Streaming with the polars.LazyFrame.sort

DataFrame.lazy().sort().collect(streaming=True)

LazyFrame is simply a DataFrame that utilizes this lazy evaluation. It is a technique to delay the evaluation of an expression until it’s actually needed.