Error: AttributeError: 'DataFrame' object has no attribute '_jdf'

Question

I want to perform k-fold cross validation using pyspark to finetune the parameters and I'm using pyspark.ml. I am getting Attribute Error.

AttributeError: 'DataFrame' object has no attribute '_jdf'

I have tried initially using pyspark.mllib but was not able to succeed in performing k-fold cross validation

import pandas as pd
from pyspark import SparkConf, SparkContext
from pyspark.ml.classification import DecisionTreeClassifier

data=pd.read_csv("file:///SparkCourse/wdbc.csv", header=None)
type(data)
print(data)

conf = SparkConf().setMaster("local").setAppName("SparkDecisionTree")
sc = SparkContext(conf = conf)

# Create initial Decision Tree Model
dt = DecisionTreeClassifier(labelCol="label", featuresCol="features", 
maxDepth=3)

# Train model with Training Data
dtModel = dt.fit(data)

# I expect the model to be trained but I'm getting the following error 
AttributeError: 'DataFrame' object has no attribute '_jdf'

Note: I'm able to print the data. Error is in dtModel

Answer 1

Convert Panadas to Spark

from pyspark.sql import SQLContext
sc = SparkContext.getOrCreate()
sqlContext = SQLContext(sc)

spark_dff = sqlContext.createDataFrame(panada_df)