SQLAlchemy "excluded" PostgreSQL namespace in INSERT ... ON CONFLICT

Question

I can't find a way to execute PostgreSQL INSERT .. ON UPDATE through SQLAlchemy. Is there a way to do it with multiple rows, performing the operation on the whole data at once?

I try to upsert with values from a pandas dataframe:

for insert_values in df.to_dict(orient='records'):
    insert_statement = sqlalchemy.dialects.postgresql.insert(orders_to_channels).values(insert_values)
    upsert_statement = insert_statement.on_conflict_do_update(
        constraint='orders_to_channels_pkey',
    set_=insert_values
    conn.execute(upsert)

This works on a row basis and as every row is handled separately - it works terribly slow (20 minutes for 7000 rows). Is there a way to perform this operation as a single SQL statement?

I am looking for some kind of an opportunity to pass parameters like {'column_name':'excluded .column_name'} to the update part of the statement, where "excluded" won't be parsed as a part of the string value, but rather as a SQL literal. Is there a way to do this?

Answer 1

Use the special alias excluded of the postgresql.dml.Insert object:

insert_statement = sqlalchemy.dialects.postgresql.insert(orders_to_channels)
upsert_statement = insert_statement.on_conflict_do_update(
    constraint='orders_to_channels_pkey',
    set_={ 'column_name': insert_statement.excluded.column_name }
)
insert_values = df.to_dict(orient='records')
conn.execute(upsert_statement, insert_values)

Note that psycopg2's executemany() is essentially equivalent to execute() in a loop, so you might not see as big a performance upgrade as expected. You could try using the "multiple values" syntax:

insert_values = df.to_dict(orient='records')
insert_statement = sqlalchemy.dialects.postgresql.insert(orders_to_channels).values(insert_values)
...

But, that might not be any faster.