Python subprocess in .exe

Question

I'm creating a python script that will copy files and folder over the network. it's cross-platform so I make an .exe file using cx_freeze

I used Popen method of the subprocess module

if I run .py file it is running as expected but when i create .exe subprocess is not created in the system

I've gone through all documentation of subprocess module but I didn't find any solution

everything else (I am using Tkinter that also works fine) is working in the .exe accept subprocess.

any idea how can I call subprocess in .exe.file ??

This file is calling another .py file

def start_scheduler_action(self, scheduler_id, scheduler_name, list_index):
       scheduler_detail=db.get_scheduler_detail_using_id(scheduler_id)
        for detail in scheduler_detail:
            source_path=detail[2]
        if not os.path.exists(source_path):
            showerror("Invalid Path","Please select valid path", parent=self.new_frame)
            return

        self.forms.new_scheduler.start_scheduler_button.destroy()

        #Create stop scheduler button
        if getattr(self.forms.new_scheduler, "stop_scheduler_button", None)==None:

            self.forms.new_scheduler.stop_scheduler_button = tk.Button(self.new_frame, text='Stop scheduler', width=10, command=lambda:self.stop_scheduler_action(scheduler_id, scheduler_name, list_index))
            self.forms.new_scheduler.stop_scheduler_button.grid(row=11, column=1, sticky=E, pady=10, padx=1)

        scheduler_id=str(scheduler_id)

        # Get python paths
        if sys.platform == "win32":
            proc = subprocess.Popen(['where', "python"], env=None, stdout=subprocess.PIPE)

        else:
            proc = subprocess.Popen(['which', "python"], env=None,stdout=subprocess.PIPE)

        out, err = proc.communicate()

        if err or not out:
            showerror("", "Python not found", parent=self.new_frame)

        else:

            try:
                paths = out.split(os.pathsep)

                # Create python path
                python_path = (paths[len(paths) - 1]).split('\n')[0]

                cmd = os.path.realpath('scheduler.py')
                #cmd='scheduler.py'

                if sys.platform == "win32":
                    python_path=python_path.splitlines()

                else:
                    python_path=python_path

                # Run the scheduler file using scheduler id

                proc = subprocess.Popen([python_path, cmd, scheduler_id], env=None, stdout=subprocess.PIPE)


                message="Started the scheduler : %s" %(scheduler_name)
                showinfo("", message, parent=self.new_frame)

                #Add process id to scheduler table
                process_id=proc.pid
                #showinfo("pid", process_id, parent=self.new_frame)
                def get_process_id(name):
                    child = subprocess.Popen(['pgrep', '-f', name], stdout=subprocess.PIPE, shell=False)
                    response = child.communicate()[0]
                    return [int(pid) for pid in response.split()]

                print(get_process_id(scheduler_name))

                # Add the process id in database
                self.db.add_process_id(scheduler_id, process_id)

                # Add the is_running status in database
                self.db.add_status(scheduler_id)

            except Exception as e:

                showerror("", e)

And this file is called:

def scheduler_copy():

    date= strftime("%m-%d-%Y %H %M %S", localtime())
    logFile = scheduler_name + "_"+scheduler_id+"_"+ date+".log"
    #file_obj=open(logFile, 'w')

    # Call __init__ method of xcopy file 
    xcopy=XCopy(connection_ip, username , password, client_name, server_name, domain_name)
    check=xcopy.connect()

    # Cretae a log file for scheduler
    file_obj=open(logFile, 'w')

    if check is False:

        file_obj.write("Problem in connection..Please check connection..!!")
        return

    scheduler_next_run=schedule.next_run()
    scheduler_next_run="Next run at: " +str(scheduler_next_run)

    # If checkbox_value selected copy all the file to new directory
    if checkbox_value==1:
        new_destination_path=xcopy.create_backup_directory(share_folder, destination_path, date)
    else:
        new_destination_path=destination_path

    # Call backup method for coping data from source to destination
    try:
        xcopy.backup(share_folder, source_path, new_destination_path, file_obj, exclude)
        file_obj.write("Scheduler completed successfully..\n")

    except Exception as e:

        # Write the error message of the scheduler to log file
        file_obj.write("Scheduler failed to copy all data..\nProblem in connection..Please check connection..!!\n")
        # #file_obj.write("Error while scheduling")
        # return

    # Write the details of scheduler to log file
    file_obj.write("Total skipped unmodified file:")
    file_obj.write(str(xcopy.skipped_unmodified_count))
    file_obj.write("\n")
    file_obj.write("Total skipped file:")
    file_obj.write(str(xcopy.skipped_file))
    file_obj.write("\n")
    file_obj.write("Total copied file:")
    file_obj.write(str(xcopy.copy_count))
    file_obj.write("\n")
    file_obj.write("Total skipped folder:")
    file_obj.write(str(xcopy.skipped_folder))
    file_obj.write("\n")
    # file_obj.write(scheduler_next_run)
    file_obj.close()

Answer 1

There is some awkwardness in your source code, but I won't spend time on that. For instance, if you want to find the source_path, it's better to use a for loop with break/else:

for detail in scheduler_detail:
    source_path = detail[2]
    break  # found
else:
    # not found: raise an exception
    ...

Some advice:

• Try to separate the user interface code and the sub-processing, avoid mixing the two.
• Use exceptions and exception handlers.
• If you want portable code: avoid system call (there are no pgrep on Windows).

---

Since your application is packaged in a virtualenv (I make the assumption cx_freeze does this kind of thing), you have no access to the system-wide Python. You even don't have that on Windows. So you need to use the packaged Python (this is a best practice anyway).

If you want to call a Python script like a subprocess, that means you have two packaged applications: you need to create an exe for the main application and for the scheduler.py script. But, that's not easy to communicate with it.

Another solution is to use multiprocessing to spawn a new Python process. Since you don't want to wait for the end of processing (which may be long), you need to create daemon processes. The way to do that is explained in the multiprocessing module.

Basically:

import time
from multiprocessing import Process

def f(name):
    print('hello', name)

if __name__ == '__main__':
    p = Process(target=f, args=('bob',))
    p.daemon = True
    p.start()

    # let it live and die, don't call: `p.join()`
    time.sleep(1)

Of course, we need to adapt that with your problem.

Here is how I would do that (I removed UI-related code for clarity):

import scheduler


class SchedulerError(Exception):
    pass


class YourClass(object):
    def start_scheduler_action(self, scheduler_id, scheduler_name, list_index):
        scheduler_detail = db.get_scheduler_detail_using_id(scheduler_id)
        for detail in scheduler_detail:
            source_path = detail[2]
            break
        else:
            raise SchedulerError("Invalid Path", "Missing source path", parent=self.new_frame)

        if not os.path.exists(source_path):
            raise SchedulerError("Invalid Path", "Please select valid path", parent=self.new_frame)

        p = Process(target=scheduler.scheduler_copy, args=('source_path',))
        p.daemon = True
        p.start()

        self.db.add_process_id(scheduler_id, p.pid)

To check if your process is still running, I recommend you to use psutil. It's really a great tool!

You can define your scheduler.py script like that:

def scheduler_copy(source_path):
    ...

---

Multiprocessing vs Threading Python

Quoting this answer: https://stackoverflow.com/a/3044626/1513933

The threading module uses threads, the multiprocessing module uses processes. The difference is that threads run in the same memory space, while processes have separate memory. This makes it a bit harder to share objects between processes with multiprocessing. Since threads use the same memory, precautions have to be taken or two threads will write to the same memory at the same time. This is what the global interpreter lock is for.

Here, the advantage of multiprocessing over multithreading is that you can kill (or terminate) a process; you can't kill a thread. You may need psutil for that.

Answer 2

This is not an exact solution you are looking for, but following suggestion should be preferred for two reasons.

1. These are more pythonic way
2. subprocess is slightly expensive

Suggestions you can consider

1. Don't use subprocess for fetching system path. Try check os.getenv('PATH') to get env variable & try to find if python is in the path. For windows, one has to manually add Python path or else you can directly check in Program Files I guess

2. For checking process ID's you can try psutils. A wonderful answer is provided here at how do I get the process list in Python?

3. Calling another script from a python script. This does not look cool. Not bad, but I would not prefer this at all.

4. In above code, line - if sys.platform == "win32": has same value in if and else condition ==> you dont need a conditional statement here.

You wrote pretty fine working code to tell you. Keep Coding!