Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python- Multiprocessing Daemon

I'm creating a multiprocess, which creates a csv file. When I run the code with d.daemon = False it works fine, ie it creates a file in the same folder. But when compiled and run with d.daemon = True, it does not, ie does not creates a file. Why's so?

My Code

I've a seed list of URLs from which I need to scrape the data.

for url in config.SEED_LIST:
    # starting a new process for each category.
    d = multiprocessing.Process(target=workers.scrape, args=())
    d.daemon = True
    d.start()


def scrape():
    import time
    time.sleep(5)
    # The above part of code takes some time to scrape a webpage, applying
    # some logic, which takes some time to execute, hence I've added a time
    # sleep of 5 secs. But when run with daemon = True, the file is not
    # created. Else it works fine.

    data = [[1, 2, 3, 4], [2224, 34, 34, 34, 34]]
    with open('1.csv', "wb") as f:
        writer = csv.writer(f)
        writer.writerows(data)
like image 305
PythonEnthusiast Avatar asked Dec 15 '14 23:12

PythonEnthusiast


People also ask

What is multiprocessing daemon?

Daemon processes in Python Python multiprocessing module allows us to have daemon processes through its daemonic option. Daemon processes or the processes that are running in the background follow similar concept as the daemon threads. To execute the process in the background, we need to set the daemonic flag to true.

What is a daemon in Python?

daemon-This property that is set on a python thread object makes a thread daemonic. A daemon thread does not block the main thread from exiting and continues to run in the background. In the below example, the print statements from the daemon thread will not printed to the console as the main thread exits.

How do I stop multiprocessing in Python?

A process can be killed by calling the Process. kill() function. The call will only terminate the target process, not child processes. The method is called on the multiprocessing.

How do I know if multiprocessing is working?

We can check if a process is alive via the multiprocessing. Process. is_alive() method.


2 Answers

According to multiprocess daemon documentation by setting d.daemon=True when your script ends its job will kill all subprocess. That occurs before they can start to write so no output will be produced.

like image 66
Michele d'Amico Avatar answered Sep 19 '22 21:09

Michele d'Amico


d.daemon = True means that the subprocess is automatically terminated after the parent process ends to prevent orphan processes. join() is helpful by simply adding d.join() after d.start(), so that the parent process does not end before the child process; instead, the parent process will wait until the child process ends.

like image 33
xxllxx666 Avatar answered Sep 20 '22 21:09

xxllxx666