Source of the ' unexpected keyword argument 'fetch' ' error in pandas to_sql?

Question

I am trying to upload a dataframe to a Heroku postgreSQL server, which I have done successfully several time before.

here is my code, where for_db is the name of my Pandas dataframe:

from sqlalchemy import create_engine 

engine = create_engine('postgresql://wgam{rest of url}',
                       echo=False)
  
# attach the data frame to the sql server 
for_db.to_sql('phil_nlp', 
               con = engine,
              if_exists='replace') 

At first, it was not able to connect because the server URL Heroku gave me had only 'postgres' at the beginning, but I understand it has to be changed to 'postgresql' to work properly and have gotten past that initial error.

Now I am getting a new error.

/usr/local/lib/python3.7/dist-packages/sqlalchemy/dialects/postgresql/psycopg2.py in do_executemany(self, cursor, statement, parameters, context)
    899                 template=executemany_values,
    900                 fetch=bool(context.compiled.returning),
--> 901                 **kwargs
    902             )
    903 

TypeError: execute_values() got an unexpected keyword argument 'fetch'

I'm not understanding why this would come up. Obviously I never specified such a keyword argument. I've done a lot of searching without any good results. Anyone know why it would now throw this error in code that was working just last week?

Answer 1

I ran into the same issue running the DataFrame.to_sql method. Adding method='multi' does get it working and is a good workaround.

Investigating it a bit a further it turned out to be an issue with the versions of sqlalchemy and psycopg2 that I had installed. These github issues here and here led me to the following.

The fetch parameter was added on psycopg2 version 2.8. I had version 2.7 and sqlalchemy 1.4.15

Installing a newer version fixed the problem without the need to add the method='multi' parameter.

pip install psycopg2-binary==2.8.6

Hope this helps anyone else finding this issue

Answer 2

Was able to fix this by adding a 'multi' as the method parameter:

for_db.to_sql('phil_nlp', 
                   con = engine,
                  if_exists='replace',
                  index=False,
                  method='multi') 

Still not sure what caused the error, but I guess that's the problem fixed :)