how does pyspark broadcast variables work

Question

I know it utilizes pickle and ship things across nodes and keep in memory and so on. what i'm confused is why the syntax on using it in pyspark work.

def main():
    sc = SparkContext()
    someValue = rand()
    V = sc.broadcast(someValue)
    A = sc.parallelize().map(worker)

def worker(element):
    element *= V.value

why the above code doesn't get a "V" not defined complaint? I searched the broadcast-related source code in pyspark but didn't get any clue.

Answer 1

I believe your problem is simply a Python scope issue. If you try the following non-Spark Python code, it'll similarly error with "'V' is not defined":

def runner(func):
    func()

def main():
    V = 22
    A = runner(worker)

def worker():
    print V

if __name__ == '__main__':
    main()

One fix is you can move worker() to be inside of main() (or alternatively, make V a global variable):

def main():
    sc = SparkContext()
    someValue = rand()
    V = sc.broadcast(someValue)
    def worker(element):
        element *= V.value
    A = sc.parallelize().map(worker)