what is the speed differential of binary versus text file i/o?

Question

I am developing a financial app that stores data in a file. I have noticed that text file i/o is pretty slow. If I switch to a binary format, how much faster will it be? 2x? 10x? I could write my own timing tests but I am hoping someone else has already done it. Thanks in advance.

Answer 1

The bottleneck is moving data to/from disk. Whether it is text or binary is not really the issue; what you need to do is minimize the amount of data.

If your data consists of a lot of numeric values, you may see some significant savings. For example, the number 1234567890 is ten bytes of text, but could fit into a four-byte binary integer. However, note that the value 1 is only one byte of text, but would still take four bytes in a binary int.

You may also want to consider time spent converting between text and binary. Reading and writing "raw" arrays of ints and doubles will be faster than converting to and from text, but then you need to worry about byte ordering, differences in sizes for different platforms/compilers, structure padding, etc.

So, you need to look at your application's data to determine if it will be significantly smaller in a binary encoding and what issues are involved in converting data between formats.

The other big issue is fixed-size vs. variable-sized records when doing random access. If records are fixed in size, then it is easy to go to the Nth record in a file. If they are variable in size, then you may have to read and parse records 1 to N-1 before reading record N.

Answer 2

Thanks for your thoughts, but as I am looking for numbers I decided to run my own tests.

Here are my results:

                    emulator   droid
integer text         5.478    0.028
integer binary       0.112    0.002
double text          6.546    0.187
double binary        0.117    0.003

integer text/bin      49       14

double text/bin       56       62

The top four rows are times in seconds to write 100 numbers. The ratios are in the last two rows. So the answer is that it is between 14 and 62 times faster to write binary versus text. I did not test read performance.

For me this is good news and bad news. Good because I can speed up the i/o a lot, bad because dealing with binary files is a pain.