Export Data from Postgres Database to Parquet file

Question

I want to install an extension to export data from my PostgreSQL Database into a Parquet file. The Database is running in a Docker container an I have no clue how to do it.

I tried some solutions but got stuck at some point eventually.

https://github.com/adjust/parquet_fdw/blob/master/README.md

https://lib.rs/crates/pg2parquet

I don't have the experience to know whats best and if there is a better option.

Answer 1

The extension with the foreign data wrapper that you mention (parquet_fdw) will allow your postgres server to read from parquet files, but currently it does not solve your problem of writting to parquet files.

On the contrary, since your server is running on a container and if you only want to get the postgres tables as parquet files, the pg2parquet utility can do exactly what you want:

1. Install the pg2parquet tool (cargo method seems simple) in a server in from which you can access (you can use yum install cargo or apt-get install cargo as superuser first if not available depending if your linux server is rpm or deb based):

cargo install [email protected]

2. Use it directly to query your DB table and export it to a parquet file (make sure you know your connection details given below as $PG variables):

~/.cargo/bin/pg2parquet export --host $PGHOST --dbname $PGDATABASE --user $PGUSER --password $PGPASSWORD --table $PGTABLE --output-file $PGTABLE.parquet