When debugging Spark/Scala code with IntelliJ, using e.g. df.select($"mycol")
does not work in the evaluate expression window, while df.select(col("mycol"))
works fine (but needs code change):
It says :
Error during generated code invocation: com.intellij.debugger.engine.evaluation.EvaluateException: Error evaluating method : 'invoke': Method threw 'java.lang.NoSuchFieldError' exception.: Error evaluating method : 'invoke': Method threw 'java.lang.NoSuchFieldError' exception.
Strangely, it seems to work sometimes, especially if the $
is already part of an existing expression in the code which I mark to evaluate. If I write arbitrary expressions (code-fragments), it fails consistently
EDIT: even repeating import spark.implicts._
in the code-fragment window does not help
Evaluate a complex expression in the editorClick Run | Debugging Actions | Quick Evaluate Expression Ctrl+Alt+F8 . Alternatively, hold Alt and click the selection.
Open the Evaluate dialog in one of the following ways: Press Shift+F9 . Select Evaluate Expression from the context menu. Choose Run | Debugging Actions | Evaluate Expression from the main menu.
Run the program in debug modeClick the Run icon in the gutter, then select Modify Run Configuration. Enter arguments in the Program arguments field. Click the Run button near the main method. From the menu, select Debug.
Try this workaround:
import spark.implicits._
$""
df.select($"diff").show()
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With