Make jupyter nbconvert write output to notebook after each cell finishes instead of waiting till the end of execution

Question

I am using jupyter nbconvert to execute some notebooks, as follows:

jupyter nbconvert --ExecutePreprocessor.timeout=-1 --to notebook --inplace --execute myNotebook.ipynb

The command works fine, however, I am unable to see the progress while the file is executing. It only writes to the notebook after all the notebook finishes. Is it possible to make it write to file after each cell finishes, or ideally write to notebook the same way as when executed from the browser interactively ? This is important when notebook takes days to finish.

Thanks,

Answer 1

I got to the same problem and solved it using papermill. It allows you to run Jupyter Notebooks from the command line, passing parameters to them, and it saves the results after each cell, which is good to see the progress:

pip install papermill
papermill notebook.ipynb output.ipynb