How to write pandas dataframe to oracle database using to_sql?

Question

I'm a new oracle learner. I'm trying to write a pandas dataframe into an oracle table. After I have made research online, I found the code itself is very simple, but I don't know why my code doesn't work.

I have read the pandas dataframe from my local file:

import cx_Oracle
import pandas as pd
import os

dir_path = os.path.dirname(os.path.realpath("__file__"))
df = pd.read_csv(dir_path+"/sample.csv")

Now print df, the dataframe df shold be like this:

   DATE            YEAR     MONTH      SOURCE      DESTINATION
0  11/1/2017 1:00  2017     1          AL          CO  
1  11/2/2017 1:00  2017     5          GA          ID  
2  11/3/2017 1:00  2017     12         GA          MO    

Then I create connection with the database by using cx_Oracle, it works. Next I try to write the dataframe df into the table TEST. This table TEST is an empty table which already exist in oracle database, it has columns including DATE, YEAR, MONTH, SOURCE, DESTINATION in oracle. All the datatype matches the df sample data. My code is as follows:

conn_str = u'account/password@host:1521/server'
conn = cx_Oracle.connect(conn_str)

# Write records stored in a DataFrame to a oracle database
df.to_sql('TEST', conn, if_exists='replace') # the error shows here

conn.close()

It shows error:

DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': ORA-01036: illegal variable name/number

How to solve the problem? Thank you very much for your time!

Answer 1

I am able to load an Oracle table using the following code:

import pandas as pd
import os

creds = {}
creds['tns_admin'] = 'Wallet_Path'
creds['sid'] = 'dev_low'
creds['user'] = 'username'
creds['password'] = pwd

os.environ['TNS_ADMIN'] = creds['tns_admin']


uri = 'oracle+cx_oracle://' + creds['user'] + ':' + creds['password'] + '@' + creds['sid']
df = pd.read_csv("test.csv")
df.to_sql('test', uri, schema='PRD', if_exists='replace')

Instead of connection we need to build and pass an URI.

Note: New Oracle databases (Autonomous) requires a wallet, so we need to set wallet path in TNS_ADMIN environment variable.

Also, I didn't have to import cx_Oracle, I did double check that

To ensure, I am not getting fooled, I dropped the table and did commit

And I executed above code, it created new table with data.

Answer 2

I've seen similar questions on SO - it happens when you try to write to Oracle DB using connection object created by cx_Oracle.

Try to create connection using SQL Alchemy:

import cx_Oracle
from sqlalchemy import types, create_engine

conn = create_engine('oracle+cx_oracle://scott:tiger@host:1521/?service_name=hr')

df.to_sql('TEST', conn, if_exists='replace')

Answer 3

After referring to this solution, I was able to get this done using the following steps.

from sqlalchemy.engine import create_engine

DIALECT = 'oracle'
SQL_DRIVER = 'cx_oracle'
USERNAME = 'your_username' 
PASSWORD = 'your_password'
HOST = 'subdomain.domain.tld' 
PORT = 1521 
SERVICE = 'your_oracle_service_name'
ENGINE_PATH_WIN_AUTH = DIALECT + '+' + SQL_DRIVER + '://' + USERNAME + ':' + PASSWORD +'@' + HOST + ':' + str(PORT) + '/?service_name=' + SERVICE

engine = create_engine(ENGINE_PATH_WIN_AUTH)

After successfully creating the SQLAlchemy engine you can pass it to pandas to_sql() function.

df.to_sql('name_of_sql_table',engine,schema='your_schema')