Controlling scheduling priority of python threads?

Question

I've written a script that uses two thread pools of ten threads each to pull in data from an API. The thread pool implements this code on ActiveState. Each thread pool is monitoring a Redis database via PubSub for new entries. When a new entry is published, python passes the data to a function that uses python's Subprocess.POpen to execute a PHP shell to do the actual work of calling the API.

This system of launching PHP shells is necessary for functionality with my PHP web app, so launching PHP shells with Python can't be avoided.

This script will only be running on Linux servers.

How do I control the niceness (scheduling priority) of the application's threads?

Edit:

It seems controlling scheduling priority for individual threads in Python isn't possible. Is there a python solution, or at the very least a UNIX command I can run along with my script, to control the priority?

Edit 2:

Well I didn't end up finding a python way to handle it. I'm just running my script with nice now like this:

nice -n 19 python MyScript.py 

Answer 1

I believe that threading priority is not controllable in python due to how they are implemented using a global interpreter lock (GIL). Having said that, even if you could give one thread more CPU processing priority, the python implementation that hands around the GIL would not be aware of this as it handed around the GIL. If you were able to increase niceness in a single thread in your pool (say it is doing a more important job) you would need to use your own implementation of locks to give the higher priority thread access to the GIL more often.

A google search returns this article which I believe is similar to what you are asking

Explains why it doesnt work http://www.velocityreviews.com/forums/t329441-threading-priority.html

Explains the workaround I was suggesting http://bytes.com/topic/python/answers/645966-setting-thread-priorities

Answer 2

It doesn't work, but I tried:

1. getting the parent pid and priority
2. launching threads using concurrent.futures.ThreadPoolExecutor
3. using ctypes to get the (linux) thread id from within the thread(works)
4. using the tid with os.setpriority(os.PRIO_PROCESS,tid,parent_priority+1)
5. calling pool.shutdown() from the parent.

Even with liberal sprinkling of os.sched_yield(), the child threads never actually run past the setpriority().

Reading man pages, it seems threads don't have the capability to change (even their) scheduling priority; you have to do something with "capabilities" to give the thread the "CAP_SYS_NICE" capability. Running the process with root permissions didn't help either; child threads still don't run.