Menu
We have trained an Extra Tree model for some regression task. Our model consists of 3 extra trees, each having 200 trees of depth 30. On top of the 3 extra trees, we use a ridge regression.
We trained our model for several hours and pickled the trained model (the entire class object), for later use. However, the size of saved trained model is too big, about 140 GB!
Is there a way to reduce the size of the saved model? Are there any configuration in pickle that could be helpful, or any alternative for pickle?
382
Reduce Size of SVM Classifier Reduce the size of a full support vector machine (SVM) classifier by removing the training data. Full SVM classifiers (that is, ClassificationSVM classifiers) hold the training data. To improve efficiency, use a smaller classifier. Load the ionosphere data set.
Therefore, the epoch when the validation error starts to increase is precisely when the model is overfitting to the training set and does not generalize new data correctly. This is when we need to stop our training.
The number one problem facing Machine Learning is the lack of good data. While enhancing algorithms often consumes most of the time of developers in AI, data quality is essential for the algorithms to function as intended.
So increasing the amount of data can only make overfitting worse if you mistakenly also increase the complexity of your model. Otherwise, the performance on the test set should improve or remain the same, but not get significantly worse.
You can try using joblib with compression parameter.
from sklearn.externals import joblib
joblib.dump(your_algo, 'pickle_file_name.pkl', compress=3)
compress - from 0 to 9. Higher value means more compression, but also slower read and write times. Using a value of 3 is often a good compromise.
You can use python standard compression modules zlib, gzip, bz2, lzma and xz. To use that you can just specify the format with specific extension
Example:
joblib.dump(obj, 'your_filename.pkl.z') # zlib
More information, see the link.
103
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
