pyspark show dataframe as table with horizontal scroll in ipython notebook

Question

a pyspark.sql.DataFrame displays messy with DataFrame.show() - lines wrap instead of a scroll.

but displays with pandas.DataFrame.head

I tried these options

import IPython
IPython.auto_scroll_threshold = 9999

from IPython.core.interactiveshell import InteractiveShell
InteractiveShell.ast_node_interactivity = "all"
from IPython.display import display

but no luck. Although the scroll works when used within Atom editor with jupyter plugin:

Answer 1

this is a workaround

spark_df.limit(5).toPandas().head() 

although, I do not know the computational burden of this query. I am thinking limit() is not expensive. corrections welcome.

Answer 2

Just add (and execute)

from IPython.core.display import HTML
display(HTML("<style>pre { white-space: pre !important; }</style>"))

And you'll get the df.show() with the scrollbar

Answer 3

I'm not sure if anyone's still facing the issue. But it could be resolved by tweaking some website settings using developer tools.

WHen you do

Open developer setting (F12). and then inspect element (ctrl+shift+c) and click on the output. and uncheck whitespace attribute (see snapshot below)

You just need to do this setting once. (unless you refresh the page)

This will show you the exact data natively as is. No need to convert to pandas.

Answer 4

Just edit the css file and you are good to go.

1. Open the jupyter notebook ../site-packages/notebook/static/style/style.min.css file.

2. Search for white-space: pre-wrap;, and remove it.

3. Save the file and restart jupyter-notebook.

Problem fixed. :)

Answer 5

Adding to the answers given above by @karan-singla and @vijay-jangir, a handy one-liner to comment out the white-space: pre-wrap styling can be done like so:

$ awk -i inplace '/pre-wrap/ {$0="/*"$0"*/"}1' $(dirname `python -c "import notebook as nb;print(nb.__file__)"`)/static/style/style.min.css

This translates as; use awk to update inplace lines that contain pre-wrap to be surrounded by */ -- */ i.e. comment out, on the file found in styles.css found in your working Python environment.

This, in theory, can then be used as an alias if one uses multiple environments, say with Anaconda.

REFs:

• https://stackoverflow.com/a/24884616/4521950

• Save modifications in place with awk