Speed up Pandas on Multi-core machine

Question

I have a pandas data frame that fits comfortably in memory. I do serval maps on the data frame, but each map is time-consuming due to the complexity of the call-back functions passed to map. I own a AWS C4 instance, which is 8-core and 16GB-RAM. I ran the python script on the machine and found that more than 80% of CPU time is idle. So, I think (correct me if I am not right) the python script is single-threaded and only consume 1 core. Is there a way to speed up pandas on multi-core machine? Here is the snippet of the two time-consuming maps

 tfidf_features = df.apply(lambda r: compute_tfidf_features(r.q1_tfidf_bow, r.q2_tfidf_bow), axis=1)
 bin_features = df.apply(lambda r: compute_bin_features(r.q1_bin_bow, r.q2_bin_bow), axis=1)

Here is the compute_tfidf_features function

def compute_tfidf_features(sparse1, sparse2):
    nparray1 = sparse1.toarray()[0]
    nparray2 = sparse2.toarray()[0]

    features = pd.Series({
    'bow_tfidf_sum1': np.sum(sparse1),
    'bow_tfidf_sum2': np.sum(sparse2),
    'bow_tfidf_mean1': np.mean(sparse1),
    'bow_tfidf_mean2': np.mean(sparse2),
    'bow_tfidf_cosine': cosine(nparray1, nparray2),
    'bow_tfidf_jaccard': real_jaccard(nparray1, nparray2),
    'bow_tfidf_sym_kl_divergence': sym_kl_div(nparray1, nparray2),
    'bow_tfidf_pearson': pearsonr(nparray1, nparray2)[0]
    })

    return features

I am aware of a python library called dask, but it says that it’s not intended for a data frame that can comfortably fit in memory.

Answer 1

Pandas does not support this. Dask arrays are mostly API compatible with Pandas and support parallel execution for apply.

You might also consider some bleeding edge solutions such as this new tool