How to achieve maximum write speed with Python?

Question

I'm writing a program that will do high speed data acquisition. The acquisition card can run at up 6.8 GB/s (It's on PCIe3 x8). Right now I'm trying to stream to a RAM disk to see the max write speed I can achieve with Python.

The card is going to give me 5-10 MB blocks, which I can then write somewhere.

I wrote this piece of code, which writes a 10MB block 500 times to a binary file. I'm using Anaconda2 on Windows 7 64-bit, and I used the profiler from Anaconda's accelerate.

block = 'A'*10*1024*1024
filename = "R:\\test"
f = os.open(filename, os.O_CREAT| os.O_BINARY|os.O_TRUNC|os.O_WRONLY|os.O_SEQUENTIAL)

p = profiler.Profile(signatures=False)
p.enable()
start = time.clock()
for x in range(500):
    os.write(f,block)
transferTime_sec = time.clock() - start
p.disable()
p.print_stats()

print('\nwrote %f MB' % (os.stat(filename).st_size/(1024*1024)))

I tested this on a RAM disk (R:\) and I got the following output:

So I figured, I'm getting something around 2.5 GB/s on RAM. which is not bad but far from max RAM throughput still, but the numbers are consistent. So the low throughput is one problem.

The second problem is, when I test this code with a PCIe SSD (which I had benchmarked with another software at 1090 MB/s sequential write), it gives comparable figures.

This makes me think that it's caching and/or buffering (?) and so I'm just not measuring actual IO. I'm not sure what's going on really as I'm fairly new to python.

So my main question is how to achieve max write speeds, and a side question is why am I getting these numbers?

Answer 1

I don't know if you are still looking after this issue, but I found your question interesting so I gave it a try on a Linux laptop.

I ran your code on python 3.5 and found that you need to have os.O_SYNC flag as well to avoid the buffering issue (basically the os.write function won't return before all data have been written on the disk). I also replace time.clock() by time.time() which give me better results.

import os
import time
import cProfile

def ioTest():
    block = bytes('A'*10*1024*1024, 'utf-8')
    filename = 'test.bin'
    f = os.open(filename, os.O_WRONLY | os.O_CREAT | os.O_TRUNC |
                os.O_SYNC)
    start = time.time()
    for x in range(500):
        os.write(f,block)
    os.close(f)
    transferTime_sec = time.time() - start
    msg = 'Wrote {:0f}MB in {:0.03f}s'
    print(msg.format(os.stat(filename).st_size/1024/1024,
                     transferTime_sec))
cProfile.run('ioTest()')

Also, this post talk about using the os.O_DIRECT flag, which will use DMA and avoid bottlenecks. I had to use the mmap module to make it work on my machine:

import os
import time
import cProfile
import mmap

def ioTest():
    m = mmap.mmap(-1, 10*1024*1024)
    block = bytes('A'*10*1024*1024, 'utf-8')
    m.write(block) filename = 'test.bin'
    f = os.open(filename, os.O_WRONLY | os.O_CREAT | os.O_TRUNC |
                os.O_SYNC, os.O_DIRECT)
    start = time.time()
    for x in range(500):
        os.write(f,m)
    os.close(f)
    transferTime_sec = time.time() - start
    msg = 'Wrote {:0f}MB in {:0.03f}s.'
    print(msg.format(os.stat(filename).st_size/1024/1024,
                     transferTime_sec))
cProfile.run('ioTest()')

This reduced the writing time on my machine by 40%... not bad. I didn't used os.O_SEQUENTIAL and os.O_BINARY that are not available on my machine.

[Edit]: I found how to use the os.O_DIRECT flag from this site which explains it very well and in depth. I strongly recommend reading this if you are interesting in performance and direct IO in Python.