How to exchange Polars-DataFrame between Rust and Python

Question

I want to write a Python extension using Rust with Ctypes or Pyo3 to get better performance than native Python. But how to exchange data such as Polars DataFrame or ndarray type between Rust and Python?

Answer 1

The easiest way to have Rust and Python work together with Polars is via the official pyo3-polars crate. You make your crate using PyO3 as usual, and when you want to pass a Series, DataFrame or LazyFrame, you pass pyo3_polars::PySeries, pyo3_polars::PyDataFrame and pyo3_polars::PyLazyFrame respectively. They all have one public field with the Rust Polars type which makes it very easy to convert between them and the Polars type, but they are also convertible from/to Python Polars objects with PyO3.

If you want to extend Polars using user-defined functions, make sure to read about expression plugins. They are the most efficient way to do that.

ndarray is not easily convertible to Python objects since it was not designed with Python interoperability in mind, but you can use the numpy crate. It gives you access to Numpy arrays (creation and convertion), and can show them (with zero copy!) as ndarray views.