crab: why is crab's UserBasedRecommender so slow?

Question

the benchmark for crab is (http://www.slideshare.net/marcelcaraciolo/crab-a-python-framework-for-building-recommender-systems page-37)

Benchmarks Pure Python w/ Python w/ Scipy Dataset dicts and NumpyMovieLens 100k 15.32 s 9.56 s http://www.grouplens.org/node/73 Old Crab New Crab

however, in my case, I need to take more than 30 minutes to do it, I don't know the reason

my code is

model = MatrixPreferenceDataModel(recommend_data.data)
similarity = UserSimilarity(model, pearson_correlation)
recommender = UserBasedRecommender(model, similarity, with_preference=True)
recommender.recommend("6")

my data is NumpyMovieLens 100k, which contains 100,000 ratings from 1000 users on 1700 movies.

Answer 1

When I last looked at Crab a little over a year ago, it was using dense arrays were it should be using sparse matrices (e.g. scipy.sparse). That made it horribly slow as well as a memory hog. I just had a quick glance and it seems Crab hasn't been updated for almost a year, so the situation is probably still the same.