Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Run some tasks in IPython Notebook in detached mode

Is it possible to run some long tasks in IPython Notebook, close browser and then after some time open it again and reveal results of these tasks?

Say, task like this:

def f():
    import time
    time.sleep(100)
    with open("result.txt", "w") as fh:
        fh.write("Done.")

If I run task normally, and close browser before completion, as I open it back, I see no 'result.txt'.

If I run it using %px magic or parallel execution - again no result if I close browser before completion.

Any extensions or hacks available? Or am I missing something?

UPDATE 1:

Although there is background jobs control support in IPython, background jobs become stale after I disconnect browser. The only thing I could come up with is issuing %connect_info before closing browser, and then connect from screen terminal using

ipython console --existing <ID>.json

and run my jobs from there.

UPDATE 2:

Even more helpful hack is combination of ipython console and job control. I.e. I open console and attach to the same session while starting background job in browser and then I'm free to close it until job is finished.

UPDATE 3:

it seems to work since version iPython 1.0dev without any hacks. you just run what you want, close browser and it still runs.

like image 285
Anaderi Avatar asked Apr 23 '13 10:04

Anaderi


People also ask

What does %% do in Jupyter Notebook?

Both ! and % allow you to run shell commands from a Jupyter notebook. % is provided by the IPython kernel and allows you to run "magic commands", many of which include well-known shell commands. ! , provided by Jupyter, allows shell commands to be run within cells.

How do you run a Jupyter Notebook in the background No keep one terminal for it?

You can put the process into the background by using jupyter notebook --no-browser & disown . You can close the terminal afterwards and the process will still be running.

What is %% capture in Python?

Capturing Output With %%capture IPython has a cell magic, %%capture , which captures the stdout/stderr of a cell. With this magic you can discard these streams or store them in a variable.


2 Answers

I normally use the linux command screen, which opens a different shell that can run in the background.

For example, when you first enter your shell you type 'screen'. It then gives you a new terminal. Start ipython there with the above command. Once it is running, on your keyboard press CTRL+A CTRL+D. The program then starts running in the background and you can close the terminal. If you want to get back to it, type 'screen -r' in the command line.

like image 166
Stephan Avatar answered Nov 03 '22 00:11

Stephan


I don't know if this is new in IPython 1.0, but in my IPython notebook, if I call your function f(), and close the browser, the kernel is running in the background and after a while (I changed to time.sleep(10)) I see the results.txt file generated. I don't think the kernel, which you usually start in a console, stops when you close the browser window. Let me know if I am mistaken.

like image 23
joon Avatar answered Nov 03 '22 01:11

joon