Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Spark: How to delete a specific variable from spark-shell memory namespace?

I am using spark-shell and Scala. Is it possible to delete a variable from the shell's memory namespace?

For example, if I create variables like this:

val foo = 1
var bar = 2

Would it be possible to delete foo and bar from memory?

I know that in Python (using del) and R (using rm), you can delete specific variables from the namespace (aka environment or workspace).

like image 222
stackoverflowuser2010 Avatar asked Apr 18 '17 18:04

stackoverflowuser2010


1 Answers

Unfortunately, you cannot delete a specific variable in Scala REPL.[1] What you can do is assigning a new value to override an existing variable. Scala REPL also provides a command :reset to remove all variables.

[1] Investigating memory leaks in the Spark Shell: https://gist.github.com/dragos/77b048c2baba93d36cd8

like image 90
zsxwing Avatar answered Oct 28 '22 20:10

zsxwing