correlation between columns python blaze

Question

Got a simple question about how-to use python blaze module for analysis. So, i'm trying to do this code:

from blaze import SQL,Table
from sqlalchemy import create_engine
from scipy.stats import pearsonr
sql_path=r'/path/to/my/database.db'
e=create_engine('sqlite:///%s'%sql_path)
blz_sql=SQL(e,'analysis_dataframe')
blz_frame=Table(blz_sql)
blz_cols=blz_frame.columns
corr=pearsonr(blz_frame[blz_cols[0]],blz_frame[blz_cols[10]])
print(corr)

And here i got this error:

TypeError: len() of unsized object

After reading some blaze docs, i found that the problem is about converting blaze column to some structure like this:

import pandas as pd
from blaze import into
df=into(pd.DataFrame,blz_frame[blz_cols[0]]

But this conversion makes iterative calculation of pearsonr on list of columns slower. So, how can i simply convert blaze column into np.array to use calculations (like pearsonr or statsmodels.api.Logit(blz_frame.y,blz_frame[[train_cols]]) on it?) If it makes sense,i'm using Anaconda for Python 3.4, my version of blaze:

import blaze
print(blaze.__version__)
#returns 0.6.3

Answer 1

Modules like scipy.stats often expect a numpy array or pandas DataFrame explicitly. Their logic is baked into these data structures.

Blaze can help you do numpy or pandas like things on foreign datasets (like your sqlite database) but are unable to reach into libraries like scipy.stats and change their code.

I see the following solutions:

1. Suck all of the data from sqlite into an ndarray/DataFrame (as you're doing here) (this is slow)
2. Improve scipy.stats so that it doesn't assume particular data structures. (this would require changing a mature codebase)
3. Write some basic statistics on a more general interface that includes Blaze

In the case of Pearson Correlation it would be quite simple to redefine the algorithm in a more general way (#3). Perhaps a Blaze-stats or just general stats module would be appropriate here.

Generally speaking, Blaze does not provide the promise that the existing scientific python code will work on foreign data structures. That is a lofty goal.