I am using python, and suppose i had some code as below
example.py
import os, psutil
import MySQLdb as mdb
conn = mdb.connect(user='root', passwd='redhat', db='File_Data', host='localhost', charset="utf8")
file_path = "/home/local/user/Module/File_processing/part-file.txt"
p = psutil.Process(os.getpid())
cursor = conn.cursor()
for line in file_open:
result = line.split('\t')
query = "insert into PerformaceReport (campaignID, keywordID, keyword, avgPosition)"
query += " VALUES (%s,%s,'%s',%s)%(result[0],result[1],result[2],result[3])"
cursor.execute( query )
conn.commit()
print p.get_cpu_times()
print p.get_memory_percent()
The above code reads data from text file and saves to database and its working fine
I am running the file with the command python path/to/the/example.py
Now what i am trying to do is finding/capturing the below three values/information
execution time
taken by the python script(example.py
) to read and saves data to the databaseCPU utilization
when the python script(example.py
) has run(executed)Memory usage
when the python script(example.py
) has run(executed)I already googled a lot and found some information here, here and here, each explanation has different approach and i am really confused and unable to understand the approach on how to capture the above three values when the script example.py
has run so asked in SO
Can anyone please provide me an working example(code) that suits/captures the above values for my need i mentioned above, so that i can extend further to my further needs
Edited
Process Followed
1 . For finding execution time
Run the example.py file with the command as below
python -m cProfile -o file.profile example.py
2 . For finding CPU Usage
Executing the top
command in a terminal after the example.py has runned
Result:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1354 root 20 0 138m 17m 7200 S 2.0 0.6 2:40.50 Xorg
2672 shivakri 20 0 581m 19m 12m S 1.3 0.6 0:07.60 gnome-terminal
2587 shivakri 20 0 987m 168m 33m S 1.0 5.6 5:25.26 firefox
2780 shivakri 20 0 1233m 51m 16m S 1.0 1.7 0:16.51 gedit
2163 shivakri 20 0 821m 135m 43m S 0.7 4.5 4:58.74 chrome
1938 shivakri 20 0 567m 15m 11m S 0.3 0.5 0:06.64 metacity
........
........
Now in the above result, how/where can i find the CPU/Memory utilization values, whether i need to take the values %CPU %MEM
from the line that contains gnome-terminal
? (Because we are running the file through terminal ?)
Aslo top
is used for finding total cpu utilization processes that are running, but we need to find the cpu utilization for only example.py script when it is executed/running, i mean how much CPU, Memory is utilized when we run the file(example.py)
Finally what my intention is when i run the python script (example.py) the values such as execution time,CPU utilization taken for running the script,and Memory used for running the script has to be recorded/stored some where(In to a text file) or something
You can use it by putting the @profile decorator around any function or method and running python -m memory_profiler myscript. You'll see line-by-line memory usage once your script exits.
Get current CPU usage using psutil The function psutil. cpu_percent() provides the current system-wide CPU utilization in the form of a percentage. It takes a parameter which is the time interval (seconds). Since CPU utilization is calculated over a period of time it is recommended to provide a time interval.
Python does not free memory back to the system immediately after it destroys some object instance. It has some object pools, called arenas, and it takes a while until those are released. In some cases, you may be suffering from memory fragmentation which also causes process' memory usage to grow.
For time profiling
cd
into the dir that contains example.py
(lets call this exampledir
).python -m cProfile -o example.profile example.py
cd
into the dir where you unpacked RunSnakepython runsnake.py exampledir/example.profile
For CPU Profiling
psutil
psutil.Process
with myProcess = psutil.Process(os.getpid())
myProcess.get_memory_info()
or myProcess.get_ext_memory_info()
or myProcess.get_memory_percent()
as requiredFor memory profiling
easy_install
or pip
example.py
:from meliae import scanner # [[source](http://www.vrplumber.com/programming/runsnakerun/)]
scanner.dump_all_objects( filename ) # you can pass a file-handle if you prefer
runsnakemem fpath
, where fpath
is the path to the file that you used in the code above.This should get you a visual memory profiler similar to What you got with RunSnakeRun.
Hope this helps
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With