Python function such as max() doesn't work in pyspark application

Question

Python function max(3,6) works under pyspark shell. But if it is put in an application and submit, it will throw an error: TypeError: _() takes exactly 1 argument (2 given)

Answer 1

It looks like you have an import conflict in your application most likely due to wildcard import from pyspark.sql.functions:

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 1.6.1
      /_/

Using Python version 2.7.10 (default, Oct 19 2015 18:04:42)
SparkContext available as sc, HiveContext available as sqlContext.

In [1]: max(1, 2)
Out[1]: 2

In [2]: from pyspark.sql.functions import max

In [3]: max(1, 2)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-bb133f5d83e9> in <module>()
----> 1 max(1, 2)

TypeError: _() takes exactly 1 argument (2 given)

Unless you work in a relatively limited it is best to either perfix:

from pyspark.sql import functions as sqlf

max(1, 2)
## 2

sqlf.max("foo")
## Column<max(foo)>

or alias:

from pyspark.sql.functions import max as max_

max(1, 2)
## 2

max_("foo")
## Column<max(foo)>