Can I extract significane values for Logistic Regression coefficients in pyspark

Question

Is there a way to get the significance level of each coefficient we receive after we fit a logistic regression model on training data?

I was trying to find out a way and could not figure out myself.

I think I may get the significance level of each feature if I run chi sq test but first of all not sure if I can run the test on all features together and secondly I have numeric data value so if it will give me right result or not that remains a question as well.

Right now I am running the modeling part using statsmodel and scikit learn but certainly want to know how can I get these results from pySparl ML or MLLib itself

If anyone can shed some light, it will be helpful

Answer 1

I use only mllib, I think that when you train a model you can use toPMML method to export your model un PMML format (xml file), then you can parse the xml file to get features weights, here an example

https://spark.apache.org/docs/2.0.2/mllib-pmml-model-export.html

Hope that will help