Pandas Dataframe to RDD

Question

Can I convert a Pandas DataFrame to RDD?

if isinstance(data2, pd.DataFrame):
    print 'is Dataframe'
else:
    print 'is NOT Dataframe'

is DataFrame

Here is the output when trying to use .rdd

dataRDD = data2.rdd
print dataRDD

AttributeError                            Traceback (most recent call last)
<ipython-input-56-7a9188b07317> in <module>()
----> 1 dataRDD = data2.rdd
      2 print dataRDD

/usr/lib64/python2.7/site-packages/pandas/core/generic.pyc in __getattr__(self, name)
   2148                 return self[name]
   2149             raise AttributeError("'%s' object has no attribute '%s'" %
-> 2150                                  (type(self).__name__, name))
   2151 
   2152     def __setattr__(self, name, value):

AttributeError: 'DataFrame' object has no attribute 'rdd'

I would like to use Pandas Dataframe and not sqlContext to build as I'm not sure if all the functions in Pandas DF are available in Spark. If this is not possible, is there anyone that can provide an example of using Spark DF

Answer 1

Can I convert a Pandas Dataframe to RDD?

Well, yes you can do it. Pandas Data Frames

pdDF = pd.DataFrame([("foo", 1), ("bar", 2)], columns=("k", "v"))
print pdDF

##      k  v
## 0  foo  1
## 1  bar  2

can be converted to Spark Data Frames

spDF = sqlContext.createDataFrame(pdDF)
spDF.show()

## +---+-+
## |  k|v|
## +---+-+
## |foo|1|
## |bar|2|
## +---+-+

and after that you can easily access underlying RDD

spDF.rdd.first()

## Row(k=u'foo', v=1)

Still, I think you have a wrong idea here. Pandas Data Frame is a local data structure. It is stored and processed locally on the driver. There is no data distribution or parallel processing and it doesn't use RDDs (hence no rdd attribute). Unlike Spark DataFrame it provides random access capabilities.

Spark DataFrame is distributed data structures using RDDs behind the scenes. It can be accessed using either raw SQL (sqlContext.sql) or SQL like API (df.where(col("foo") == "bar").groupBy(col("bar")).agg(sum(col("foobar")))). There is no random access and it is immutable (no equivalent of Pandas inplace). Every transformation returns new DataFrame.

If this is not possible, is there anyone that can provide an example of using Spark DF

Not really. It is far to broad topic for SO. Spark has a really good documentation and Databricks provides some additional resources. For starters you check these:

• Introducing DataFrames in Spark for Large Scale Data Science
• Spark SQL and DataFrame Guide