Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

to_sql pandas method changes the scheme of sqlite tables

Tags:

python

pandas

When I write Pandas DataFrame to my SQLite database using to_sql method it changes the .schema of my table even if I use if_exists='append'. For example after execution

with sqlite3.connect('my_db.sqlite') as cnx:
    df.to_sql('Resolved', cnx, if_exists='append')

original .schema:

CREATE TABLE `Resolved` (
`Name` TEXT NOT NULL COLLATE NOCASE,
`Count` INTEGER NOT NULL,
`Obs_Date` TEXT NOT NULL,
`Bessel_year` REAL NOT NULL,
`Filter` TEXT NOT NULL,
`Comments` TEXT COLLATE NOCASE
);

changes to:

CREATE TABLE Resolved (
                  [Name] TEXT,
  [Count] INTEGER,
  [Obs_Date] TEXT,
  [Bessel_year] REAL,
  [Filter] TEXT,
  [Comments] TEXT

                  );

How to save the original scheme of my table? I use pandas 0.14.0, Python 2.7.5

like image 700
drastega Avatar asked Jun 05 '14 15:06

drastega


People also ask

What is schema in to_sql?

The schema parameter in to_sql is confusing as the word "schema" means something different from the general meaning of "table definitions". In some SQL flavors, notably postgresql, a schema is effectively a namespace for a set of tables. For example, you might have two schemas, one called test and one called prod .

How does DF to_sql work?

DataFrame - to_sql() function. The to_sql() function is used to write records stored in a DataFrame to a SQL database. Name of SQL table. Using SQLAlchemy makes it possible to use any DB supported by that library.

Does pandas work with SQLite?

sqlite3 provides a SQL-like interface to read, query, and write SQL databases from Python. sqlite3 can be used with Pandas to read SQL data to the familiar Pandas DataFrame. Pandas and sqlite3 can also be used to transfer between the CSV and SQL formats.


1 Answers

Starting from 0.14 (what you are using), the sql functions are refactored to use sqlalchemy to improve the functionality`. See the whatsnew and docs on this.
The raw sqlite3 connection is still supported as a fallback (but that is the only sql flavor that is supported without sqlalchemy).

Using sqlalchemy should solve the issue. For this you can just create a sqlalchemy engine instead of the direct sqlite connection cnx:

engine = sqlalchemy.create_engine('sqlite:///my_db.sqlite')
df.to_sql('Resolved', engine, if_exists='append')

But I filed an issue for the case with the sqlite cnx fallback option: https://github.com/pydata/pandas/issues/7355

like image 178
joris Avatar answered Sep 21 '22 04:09

joris