Menu
Within an interactive pyspark session you can import python files via sc.addPyFile('file_location'). If you need to make changes to that file and save them, is there any way to "re-broadcast" the updated file without having to shut down your spark session and start a new one?
Simply adding the file again doesn't work. I'm not sure if renaming the file works, but I don't want to do that anyways.
As far as I can tell from the spark documentation there is only a method to add a pyfile, not update one. I'm hoping that I missed something!
Thanks
446
Go to the Spark Installation directory from the command line and type bin/pyspark and press enter, this launches pyspark shell and gives you a prompt to interact with Spark in Python language. If you have set the Spark in a PATH then just enter pyspark in command line or terminal (mac users).
In addition, PySpark fully supports interactive use—simply run ./bin/pyspark to launch an interactive shell.
Using Virtualenv In the upcoming Apache Spark 3.1, PySpark users can use virtualenv to manage Python dependencies in their clusters by using venv-pack in a similar way as conda-pack. In the case of Apache Spark 3.0 and lower versions, it can be used only with YARN.
I don't think it's feasible during an interactive session. You will have to restart your session to use the modified module.
116
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
