Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

What would be an ideal buffer size? [duplicate]

Tags:

c++

c

io

Possible Duplicate:
How do you determine the ideal buffer size when using FileInputStream?

When reading raw data from a file (or any input stream) using either the C++'s istream family's read() or C's fread(), a buffer has to be supplied, and a number of how much data to read. Most programs I have seen seem to arbitrarily chose a power of 2 between 512 and 4096.

  1. Is there a reason it has to/should be a power of 2, or this just programer's natural inclination to powers of 2?
  2. What would be the "ideal" number? By "ideal" I mean that it would be the fastest. I assume it would have to be a multiple of the underlying device's buffer size? Or maybe of the underlying stream object's buffer? How would I determine what the size of those buffers is, anyway? And once I do, would using a multiple of it give any speed increase over just using the exact size?

EDIT
Most answers seem to be that it can't be determined at compile time. I am fine with finding it at runtime.

like image 549
Baruch Avatar asked May 22 '12 08:05

Baruch


People also ask

What is the optimal buffer size?

All that said, there's no “industry standard” buffer size and sample rate, as it's all dependent on your computer's processing power. However, recording at 128 to 256 at a sample rate of 48kHz is acceptable for most home recording on modern-day computers.

Is 1024 a good buffer size?

To eliminate latency, lower your buffer size to 64 or 128. This will give your CPU little time to process the input and output signals, giving you no delay. Likewise, when it's time for mixing, nothing's better than a larger buffer, such as 1024, which will give your CPU the time it needs to process.

What is the best buffer size for streaming?

A buffer size less than 0.5 s results in a severe loss of QoE for all users. A buffer size of 2-4 s offers a good QoE for the average user and any sensitive user. Increasing the buffer size further decreases the QoE.

How Big Should TCP buffer be?

The default buffer size is 8 KB. The maximum size is 8 MB (8096 KB). The optimal buffer size depends on several network environment factors including types of switches and systems, acknowledgment timing, error rates and network topology, memory size, and data transfer size.


1 Answers

SOURCE:
How do you determine the ideal buffer size when using FileInputStream?

Optimum buffer size is related to a number of things: file system block size, CPU cache size and cache latency.

Most file systems are configured to use block sizes of 4096 or 8192. In theory, if you configure your buffer size so you are reading a few bytes more than the disk block, the operations with the file system can be extremely inefficient (i.e. if you configured your buffer to read 4100 bytes at a time, each read would require 2 block reads by the file system). If the blocks are already in cache, then you wind up paying the price of RAM -> L3/L2 cache latency. If you are unlucky and the blocks are not in cache yet, the you pay the price of the disk->RAM latency as well.

This is why you see most buffers sized as a power of 2, and generally larger than (or equal to) the disk block size. This means that one of your stream reads could result in multiple disk block reads - but those reads will always use a full block - no wasted reads.

Ensuring this also typically results in other performance friendly parameters affecting both reading and subsequent processing: data bus width alignment, DMA alignment, memory cache line alignment, whole number of virtual memory pages.

like image 58
ravi Avatar answered Sep 22 '22 04:09

ravi