Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Limit python script RAM usage in Windows

My program can allocate suddenly a lot of RAM based on usage. I want to limit the RAM it can take from the system.

I saw here: Limit RAM usage to python program

But it works only for Unix. Any solution for Windows?

like image 342
BestR Avatar asked Mar 01 '19 16:03

BestR


People also ask

Why is my python script using so much memory?

Python does not free memory back to the system immediately after it destroys some object instance. It has some object pools, called arenas, and it takes a while until those are released. In some cases, you may be suffering from memory fragmentation which also causes process' memory usage to grow.

How do I monitor Python script memory usage?

You can use it by putting the @profile decorator around any function or method and running python -m memory_profiler myscript. You'll see line-by-line memory usage once your script exits.


2 Answers

A Job object supports limiting the committed memory of a process. In Python, we can implement this via PyWin32 or ctypes.

Note that prior to Windows 8 a process can only be in a single Job. A couple of common cases where this is a concern include the py.exe launcher (the default association for .py files), which runs python.exe in a Job, and the Task Scheduler service, which runs each task in a Job.

PyWin32 Example

import sys
import warnings

import winerror
import win32api
import win32job

g_hjob = None

def create_job(job_name='', breakaway='silent'):
    hjob = win32job.CreateJobObject(None, job_name)
    if breakaway:
        info = win32job.QueryInformationJobObject(hjob,
                    win32job.JobObjectExtendedLimitInformation)
        if breakaway == 'silent':
            info['BasicLimitInformation']['LimitFlags'] |= (
                win32job.JOB_OBJECT_LIMIT_SILENT_BREAKAWAY_OK)
        else:
            info['BasicLimitInformation']['LimitFlags'] |= (
                win32job.JOB_OBJECT_LIMIT_BREAKAWAY_OK)
        win32job.SetInformationJobObject(hjob,
            win32job.JobObjectExtendedLimitInformation, info)
    return hjob

def assign_job(hjob):
    global g_hjob
    hprocess = win32api.GetCurrentProcess()
    try:
        win32job.AssignProcessToJobObject(hjob, hprocess)
        g_hjob = hjob
    except win32job.error as e:
        if (e.winerror != winerror.ERROR_ACCESS_DENIED or
            sys.getwindowsversion() >= (6, 2) or
            not win32job.IsProcessInJob(hprocess, None)):
            raise
        warnings.warn('The process is already in a job. Nested jobs are not '
            'supported prior to Windows 8.')

def limit_memory(memory_limit):
    if g_hjob is None:
        return
    info = win32job.QueryInformationJobObject(g_hjob,
                win32job.JobObjectExtendedLimitInformation)
    info['ProcessMemoryLimit'] = memory_limit
    info['BasicLimitInformation']['LimitFlags'] |= (
        win32job.JOB_OBJECT_LIMIT_PROCESS_MEMORY)
    win32job.SetInformationJobObject(g_hjob,
        win32job.JobObjectExtendedLimitInformation, info)

def main():
    assign_job(create_job())
    memory_limit = 100 * 1024 * 1024 # 100 MiB
    limit_memory(memory_limit)
    try:
        bytearray(memory_limit)
    except MemoryError:
        print('Success: available memory is limited.')
    else:
        print('Failure: available memory is not limited.')
    return 0

if __name__ == '__main__':
    sys.exit(main())
like image 161
Eryk Sun Avatar answered Sep 21 '22 09:09

Eryk Sun


I had about the same problem as the OP, except that I wanted to limit the amount of physical RAM used, not virtual. @eryksun's answer and limit_memory() function work great, but limit the total amount of allocatable memory (virtual memory). Here is an additional function to his/her code that limits the amount of physical memory (the "working set").

def limit_working_set(memory_limit):
    if g_hjob is None:
        return
    info = win32job.QueryInformationJobObject(g_hjob, win32job.JobObjectBasicLimitInformation)
    info['MinimumWorkingSetSize'] = 50 * 4096  # default minimum value
    info['MaximumWorkingSetSize'] = memory_limit
    info['LimitFlags'] = (win32job.JOB_OBJECT_LIMIT_WORKINGSET)
    win32job.SetInformationJobObject(g_hjob, win32job.JobObjectBasicLimitInformation, info)
like image 28
cxxl Avatar answered Sep 23 '22 09:09

cxxl