I am trying to use 'pandas.read_sql_query' to copy data from MS SQL Server into a pandas DataFrame. I need to do multiple joins in my SQL query. The tables being joined are on the same server but in different databases. The query I am passing to pandas works fine inside MS SQL Server Management Studio. In a Jupyter Notebook I tried to query data like so (to make things readable the query itself is simplified to just 2 joins and generic names are used):
import pandas as pd
import sqlalchemy as sql
import pyodbc
server = '100.10.10.10'
driver = 'SQL+Server+Native+Client+11.0'
myQuery = '''SELECT first.Field1, second.Field2
FROM db1.schema.Table1 AS first
JOIN db2.schema.Table2 AS second
ON first.Id = second.FirstId
'''
engine = sql.create_engine('mssql+pyodbc://{}?driver={}'.format(server, driver))
df = pd.read_sql_query(myQuery, engine)
This does not work and returns an error:
DBAPIError: (pyodbc.Error) ('IM010', '[IM010] [Microsoft][��������� ��������� ODBC] ������� ������� ��� ��������� ������ (0) (SQLDriverConnect)')
It seems that the problem is in the engine which does not include information about the database, because everything works fine with the next kind of code, where I include database in the engine:
myQuery = 'select Field1 from schema.Table1'
db = 'db1'
engine = sql.create_engine('mssql+pyodbc://{}/{}?driver={}'.format(server, db, driver))
df = pd.read_sql_query(myQuery, engine)
but breaks like the code with joins above if I don't include database in the engine, but add it to the query like so:
myQuery = 'select Field1 from db1.schema.Table1'
engine = sql.create_engine('mssql+pyodbc://{}?driver={}'.format(server,
driver))
df = pd.read_sql_query(myQuery, engine)
So how should I specify the pandas.read_sql_query 'sql' and 'con' parameters in this case when I need to join tables from different databases but the same server?
P.S. I only have read access to this server I am connecting to. I can not create new tables or views or anything like that.
Update: The MS SQL Server version is 2008 R2.
Update 2: I am using Python 3.6 and Windows 10.
PyODBC allows you connecting to and using an ODBC database using the standard DB API 2.0. SQL Alchemy is a toolkit that resides one level higher than that and provides a variety of features: Object-relational mapping (ORM) Query constructions.
We can write any conventional SQL query inside the text function enclosed by “”. Now, passing this SQL query to execute the function the engine object which we created while connecting the database will interpret this and convert this query to SQLAlchemy compatible format and results in the result.
So I have found a workaround: use pymssql instead of pyodbc (both in the import statement and in the engine). It lets you build your joins using database names and without specifying them in the engine. And there is no need to specify a driver in this case.
There might be a problem if you are using Python 3.6 which is not supported by pymssql oficially yet, but you can find unofficial wheels for your Python 3.6 here. It works as is supposed to with my queries.
Here is the original code with joins, rebuilt to work with pymssql:
import pandas as pd
import sqlalchemy as sql
import pymssql
server = '100.10.10.10'
myQuery = '''SELECT first.Field1, second.Field2
FROM db1.schema.Table1 AS first
JOIN db2.schema.Table2 AS second
ON first.Id = second.FirstId'''
engine = sql.create_engine('mssql+pymssql://{}'.format(server))
df = pd.read_sql_query(myQuery, engine)
As for the unofficial wheels, you need to download the file for Python 3.6 from the link I gave above, then cd to the download folder and run pip install wheels
where 'wheels' is the name of the wheels file.
UPDATE:
Actually, it is possible to use pyodbc too. I am not sure if this should work for any SQL Server setup, but everything worked for me after I had set 'master' as my database in the engine. The resulting code would look like this:
import pandas as pd
import sqlalchemy as sql
import pyodbc
server = '100.10.10.10'
driver = 'SQL+Server'
db = 'master'
myQuery = '''SELECT first.Field1, second.Field2
FROM db1.schema.Table1 AS first
JOIN db2.schema.Table2 AS second
ON first.Id = second.FirstId'''
engine = sql.create_engine('mssql+pyodbc://{}/{}?driver={}'.format(server, db, driver))
df = pd.read_sql_query(myQuery, engine)
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With