Menu
I have a very large python script, 200K, that I would like to use as little memory as possible. It looks something like:
# a lot of data structures r = [34, 78, 43, 12, 99] # a lot of functions that I use all the time def func1(word): return len(word) + 2 # a lot of functions that I rarely use def func1(word): return len(word) + 2 # my main loop while 1: # lots of code # calls functions If I put the functions that I rarely use in a module, and import them dynamically only if necessary, I can't access the data. That's as far as I've gotten.
I'm new at python.
Can anyone put me on the right path? How can I break this large script down so that it uses less memory? Is it worth putting rarely used code in modules and only calling them when needed?
735
Those numbers can easily fit in a 64-bit integer, so one would hope Python would store those million integers in no more than ~8MB: a million 8-byte objects. In fact, Python uses more like 35MB of RAM to store these numbers. Why? Because Python integers are objects, and objects have a lot of memory overhead.
You can use it by putting the @profile decorator around any function or method and running python -m memory_profiler myscript. You'll see line-by-line memory usage once your script exits.
Memory management in Python involves a private heap containing all Python objects and data structures. The management of this private heap is ensured internally by the Python memory manager.
Changing numeric columns to smaller dtype: Instead, we can downcast the data types. Simply Convert the int64 values as int8 and float64 as float8. This will reduce memory usage.
Organizing:
Your python script seems indeed to be huge, maybe you should consider reorganizing your code first, to split it into several modules or packages. It will probably make easier the code profiling and the optimization tasks.
You may want to have a look there:
And possibly:
Optimizing:
There is a lot of things that can be done for optimizing your code ...
For instance, regarding your data structures ... If you make big use of lists or lists comprehensions, you could try to figure out where do you really need lists, and where they might be replaced by non-mutable data structures like tuples or by "volatile" objects, "lazy" containers, like generator expressions.
See:
On these pages, you could find some useful information and tips:
Also, you should study your ways of doing things and wonder whether there is a way to do that less greedily, a way that it's better to do it in Python (you will find some tips in the tag pythonic) ... That is especially true in Python, since, in Python, there is often one "obvious" way (and only one) to do things which are better than the others (see The Zen of Python), which is said to be pythonic. It's not especially related to the shape of your code, but also - and above all - to the performances. Unlike many languages, which promote the idea that there should be many ways to do anything, Python prefers to focus on the best way only. So obviously, there are many ways for doing something, but often, one is really better.
Now, you should also verify whether you are using the best methods for doing things because pythonicality won't arrange your algorithms for you.
But at last, it varies depending on your code and it's difficult to answer without having seen it.
And, make sure to take into account the comments made by eumiro and Amr.
111
This video might give you some good ideas: http://pyvideo.org/video/451/pycon-2011---quot-dude--where--39-s-my-ram--quot-
36
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
