Using python generators in scikit-learn [closed]

Question

I was wondering whether and how it is possible to use a python generator as data input to scikit-learn classifier's .fit() functions? Due to huge amounts of data, this seems to make sense to me.

In particular I am about to implement a random forest approach.

Regards K

Answer 1

The answer is "no". To do out of core learning with random forests, you should

1. Split your data into reasonably-sized batches (restricted by the amount of RAM you have; bigger is better);
2. train separate random forests;

3. append all the underlying trees together in the estimators_ member of one of the trees (untested):

   for i in xrange(1, len(forests)):
       forests[0].estimators_.extend(forests[i].estimators_)`
   

(Yes, this is hacky, but no solution to this problem has been found yet. Note that with very large datasets, it might pay to just sample a number training examples that fits in the RAM of a big machine instead of training on all of it. Another option is to switch to linear models with SGD, those implement a partial_fit method, but obviously they're limited in the kind of functions they can learn.)