I train a Random Forest with pySpark. I want to have a csv with the results, per dot in the grid. My code is:
estimator = RandomForestRegressor()
evaluator = RegressionEvaluator()
paramGrid = ParamGridBuilder().addGrid(estimator.numTrees, [2,3])\
.addGrid(estimator.maxDepth, [2,3])\
.addGrid(estimator.impurity, ['variance'])\
.addGrid(estimator.featureSubsetStrategy, ['sqrt'])\
.build()
pipeline = Pipeline(stages=[estimator])
crossval = CrossValidator(estimator=pipeline,
estimatorParamMaps=paramGrid,
evaluator=evaluator,
numFolds=3)
cvModel = crossval.fit(result)
So I want a csv:
numTrees | maxDepth | impurityMeasure
2 2 0.001
2 3 0.00023
Etc
What is the best way to do this?
You'll have to combine different bits of data:
Estimator
ParamMaps
extracted using getEstimatorParamMaps
method.avgMetrics
parameter.First get names and values of all parameters declared in the map:
params = [{p.name: v for p, v in m.items()} for m in cvModel.getEstimatorParamMaps()]
Thane zip
with metrics and convert to a data frame
import pandas as pd
pd.DataFrame.from_dict([
{cvModel.getEvaluator().getMetricName(): metric, **ps}
for ps, metric in zip(params, cvModel.avgMetrics)
])
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With