Script performs better when run through PyCharm than when run directly in command line

Question

That very same question was already asked here but got no answers.

I wrote a script that takes very long to complete, and am using a module (tqdm) to give me feedback on its progress. When run on a remote environment through PyCharm, the script performance is between 800 and 1000 iterations per second, which yields an estimated total run time somewhere between 4 and 5 hours.

Problems arise when I try to run the script through a classic Linux command line, on the same remote environment. Under the same circumstances, within the same environment, using the same command, with the only difference that I typed it myself instead of having PyCharm issue it for me, the script performance drops to 200-300 iterations per second, resulting in an estimated 10-16 hours run.

In PyCharm, the run config window for my script looks like this: As you can see, the used interpreter is located on my development server, in the virtual environment used by the whole project.
I had to hide the script name and parameters, which you may find inconvenient, but I don't believe my issue is related to the script itself or its contents anyway.

In order to run the same script through the command line, here is what I do:

• SSH to my server
• cd to my project folder
• activate my environment
• export PYTHONPATH=$(pwd)
• run the same script with the same options

...which should actually result in the same behaviour, right? Well yes, but not at the same speed. There is this 70% performance drop that happens for reasons that I can't figure out, and which seems to be documented nowhere on the Internet.

I've also tried, in an attempt to perfectly mock what Pycharm does:

• copying the exact command output by PyCharm upon running the script, and pasting it in my terminal (to no avail)
• issuing export PYTHONUNBUFFERED=1 before running the script, which I knew would change nothing (it didn't)

I'm losing my mind over this. Is this the result of some shady PyCharm shenanigans?
Any kind of input related to anything PyCharm does which could even remotely speed up some aspect of a Python script execution is appreciated!

Thanks in advance.

Answer 1

I stumbled across the same issue on a windows system and was able to identify the root cause in my case. Maybe this will help you as well for the linux case:

I worked with psutil to find out what might cause the slow down. Here, I found the num_ctx_switches quite useful, as it showed how often the process was "distracted".

import psutil
p = psutil.Process()
print(p.num_ctx_switches())

# script

print(p.num_ctx_switches())

The discrepancy in voluntary context switches (smaller = more focused = better) was proportional to the time difference I observed between Pycharm and the terminal.

I then tried to raise the priority of the python process with various command line arguments and/or system settings, but nothing had a significant impact, the time difference stayed.

Finally, I fell back to using psutil to modify the process priority directly at the start of my script:

import psutil
psutil.Process().nice(psutil.HIGH_PRIORITY_CLASS)
# for linux, the nice value should be -20 to 20,
# where -20 has the highest priority.
# See psutil nice documentation link above.

I assume that windows treats tasks started from terminal as "background" tasks, which easily get interrupted by other "real" applications (such as pycharm, which seems not to be easily interrupted).