Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is there a way to get a view into a python array.array()?

Tags:

python

I'm generating many largish 'random' files (~500MB) in which the contents are the output of repeated calls to random.randint(...). I'd like to preallocate a large buffer, write longs to that buffer, and periodically flush that buffer to disk. I am currently using array.array() but I can't see a way to create a view into this buffer. I need to do this so that I can feed the part of the buffer with valid data into hashlib.update(...) and to write the valid part of the buffer to the file. I could use the slice operator but AFAICT that creates a copy of the buffer, which isn't what I want.

Is there a way to do this that I'm not seeing?

Update:

I went with numpy as user42005 and hgomersall suggested. Unfortunately this didn't give me the speedups I was looking for. My dirt-simple C program generates ~700MB of data in 11s, while my python equivalent using numpy takes around 700s! It's hard to believe that that's the difference in performance between the two (I'm more likely to believe that I made a naive mistake somewhere...)

like image 520
Allen George Avatar asked Jul 10 '26 16:07

Allen George


1 Answers

I guess you could use numpy: http://www.numpy.org - the fundamental array type in numpy at least supports no-copy views.

like image 135
user422005 Avatar answered Jul 12 '26 05:07

user422005



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!