How to show the spark progress bar in Jupyter notebook (using pyspark)

I want to see the progress bar in the Jupyter notebook. The progress bar is only shown in the terminal and does not pass over to the notebook.

But this is not the case when you use Scala in the jupyter notebook (you can see the progress bar)

There is this post asking the same thing.

My jupyter version is 4.3.0

By the way, the progress bar is on since I can see it in the terminal.

How do I read spark progress bar?

Towards the end, as the last few tasks execute, B will start decreasing until it reaches 0, at which point A should equal C, the stage is done, and spark moves to the next stage. C will stay constant during the whole time, remember it is the total number of tasks in the stage and never changes.

There are a couple of Jupyter extensions that allow you to do this.

Have a look at mozilla/jupyter-spark
- https://github.com/mozilla/jupyter-spark
and at krishnan-r/sparkmonitor
- https://github.com/krishnan-r/sparkmonitor

Both links above have installation instructions and examples how they work.