My program can allocate suddenly a lot of RAM based on usage. I want to limit the RAM it can take from the system.
I saw here: Limit RAM usage to python program
But it works only for Unix. Any solution for Windows?
Python does not free memory back to the system immediately after it destroys some object instance. It has some object pools, called arenas, and it takes a while until those are released. In some cases, you may be suffering from memory fragmentation which also causes process' memory usage to grow.
You can use it by putting the @profile decorator around any function or method and running python -m memory_profiler myscript. You'll see line-by-line memory usage once your script exits.
A Job object supports limiting the committed memory of a process. In Python, we can implement this via PyWin32 or ctypes.
Note that prior to Windows 8 a process can only be in a single Job. A couple of common cases where this is a concern include the py.exe launcher (the default association for .py files), which runs python.exe in a Job, and the Task Scheduler service, which runs each task in a Job.
import sys
import warnings
import winerror
import win32api
import win32job
g_hjob = None
def create_job(job_name='', breakaway='silent'):
hjob = win32job.CreateJobObject(None, job_name)
if breakaway:
info = win32job.QueryInformationJobObject(hjob,
win32job.JobObjectExtendedLimitInformation)
if breakaway == 'silent':
info['BasicLimitInformation']['LimitFlags'] |= (
win32job.JOB_OBJECT_LIMIT_SILENT_BREAKAWAY_OK)
else:
info['BasicLimitInformation']['LimitFlags'] |= (
win32job.JOB_OBJECT_LIMIT_BREAKAWAY_OK)
win32job.SetInformationJobObject(hjob,
win32job.JobObjectExtendedLimitInformation, info)
return hjob
def assign_job(hjob):
global g_hjob
hprocess = win32api.GetCurrentProcess()
try:
win32job.AssignProcessToJobObject(hjob, hprocess)
g_hjob = hjob
except win32job.error as e:
if (e.winerror != winerror.ERROR_ACCESS_DENIED or
sys.getwindowsversion() >= (6, 2) or
not win32job.IsProcessInJob(hprocess, None)):
raise
warnings.warn('The process is already in a job. Nested jobs are not '
'supported prior to Windows 8.')
def limit_memory(memory_limit):
if g_hjob is None:
return
info = win32job.QueryInformationJobObject(g_hjob,
win32job.JobObjectExtendedLimitInformation)
info['ProcessMemoryLimit'] = memory_limit
info['BasicLimitInformation']['LimitFlags'] |= (
win32job.JOB_OBJECT_LIMIT_PROCESS_MEMORY)
win32job.SetInformationJobObject(g_hjob,
win32job.JobObjectExtendedLimitInformation, info)
def main():
assign_job(create_job())
memory_limit = 100 * 1024 * 1024 # 100 MiB
limit_memory(memory_limit)
try:
bytearray(memory_limit)
except MemoryError:
print('Success: available memory is limited.')
else:
print('Failure: available memory is not limited.')
return 0
if __name__ == '__main__':
sys.exit(main())
I had about the same problem as the OP, except that I wanted to limit the amount of physical RAM used, not virtual. @eryksun's answer and limit_memory()
function work great, but limit the total amount of allocatable memory (virtual memory). Here is an additional function to his/her code that limits the amount of physical memory (the "working set").
def limit_working_set(memory_limit):
if g_hjob is None:
return
info = win32job.QueryInformationJobObject(g_hjob, win32job.JobObjectBasicLimitInformation)
info['MinimumWorkingSetSize'] = 50 * 4096 # default minimum value
info['MaximumWorkingSetSize'] = memory_limit
info['LimitFlags'] = (win32job.JOB_OBJECT_LIMIT_WORKINGSET)
win32job.SetInformationJobObject(g_hjob, win32job.JobObjectBasicLimitInformation, info)
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With