Getting total memory and cpu usage for one python instance

Question

I'm using keras to make and test different types of Neural Nets and need data to compare them. I need data on the cpu and memory used during the training and testing. This is in python and as I looked around I found lot of suggestions for psutil. However everything I see seems to grab the current usage.

What is current usage? Like the amount of memory used at that specific moment? How do I use it to get the total cpu and memory used by the entire program, or at least the portion of the code where the NN is training and testing. Thanks for any help!

Answer 1

psutil is a good recommendation to collect that type of information. If you incorporate this code into your existing keras code, you can collect information about the cpu usage of your process at the time the cpu_times() method is called

import psutil

process = psutil.Process()
print(process.cpu_times())

The meaning of the value returned by cpu_times() is explained here. It is cumulative, so if you want to know how much CPU time your keras code used altogether, just run it before you exit the python script.

To get the memory usage information for your process, at the particular time you make the call to memory_info() you can run this on the same process object we declared before

print(process.memory_info())

The exact meaning of the cpu and memory results depend on what platform you're using. The memory info structure is explained here

A more comprehensive example shows how you could use the Advanced Python Scheduler to take cpu and memory measurements in the background as you run your keras training

import psutil

import time
import os

from apscheduler.schedulers.background import BackgroundScheduler

process = psutil.Process()

def get_info():
    print(process.cpu_times(), process.memory_info())

if __name__ == '__main__':
    scheduler = BackgroundScheduler()
    scheduler.add_job(get_info, 'interval', seconds=3)
    scheduler.start()

    # run the code you want to measure here
    # replace this nonsense loop
    now = time.time()
    finish = now + 60

    while time.time() < finish:
        print("Some progress message: {}".format(time.time()))
        time.sleep(10)