Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python `open` function memory usage

Tags:

python

file

ram

Does Python interpreter load all the file to RAM when it is opened for writing?

with open('file.txt' 'w') as file:
    file.write(some_content)
like image 783
I159 Avatar asked Feb 16 '15 09:02

I159


People also ask

Does Python open use memory?

No. As per the docs, open() wraps a system call and returns a file object, the file contents are not loaded into RAM (unless you invoke, E.G., readlines()).

Does Python Open read entire file into memory?

The read() function is designed to be called once, and it returns the entire contents of the file.

How do I check Python program memory usage?

Working with Python Memory Profiler You can use it by putting the @profile decorator around any function or method and running python -m memory_profiler myscript. You'll see line-by-line memory usage once your script exits.

Why is my Python script using so much memory?

Those numbers can easily fit in a 64-bit integer, so one would hope Python would store those million integers in no more than ~8MB: a million 8-byte objects. In fact, Python uses more like 35MB of RAM to store these numbers. Why? Because Python integers are objects, and objects have a lot of memory overhead.


1 Answers

No. As per the docs, open() wraps a system call and returns a file object, the file contents are not loaded into RAM (unless you invoke, E.G., readlines()).

like image 199
Mark R. Avatar answered Oct 22 '22 09:10

Mark R.