Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Is StreamReader.Readline() really the fastest method to count lines in a file?

While looking around for a while I found quite a few discussions on how to figure out the number of lines in a file.

For example these three:
c# how do I count lines in a textfile
Determine the number of lines within a text file
How to count lines fast?

So, I went ahead and ended up using what seems to be the most efficient (at least memory-wise?) method that I could find:

private static int countFileLines(string filePath)
{
    using (StreamReader r = new StreamReader(filePath))
    {
        int i = 0;
        while (r.ReadLine() != null) 
        { 
            i++; 
        }
        return i;
    }
}

But this takes forever when the lines themselves from the file are very long. Is there really not a faster solution to this?

I've been trying to use StreamReader.Read() or StreamReader.Peek() but I can't (or don't know how to) make the either of them move on to the next line as soon as there's 'stuff' (chars? text?).

Any ideas please?


CONCLUSION/RESULTS (After running some tests based on the answers provided):

I tested the 5 methods below on two different files and I got consistent results that seem to indicate that plain old StreamReader.ReadLine() is still one of the fastest ways... To be honest, I'm perplexed after all the comments and discussion in the answers.

File #1:
Size: 3,631 KB
Lines: 56,870

Results in seconds for File #1:
0.02 --> ReadLine method.
0.04 --> Read method.
0.29 --> ReadByte method.
0.25 --> Readlines.Count method.
0.04 --> ReadWithBufferSize method.

File #2:
Size: 14,499 KB
Lines: 213,424

Results in seconds for File #1:
0.08 --> ReadLine method.
0.19 --> Read method.
1.15 --> ReadByte method.
1.02 --> Readlines.Count method.
0.08 --> ReadWithBufferSize method.

Here are the 5 methods I tested based on all the feedback I received:

private static int countWithReadLine(string filePath)
{
    using (StreamReader r = new StreamReader(filePath))
    {
    int i = 0;
    while (r.ReadLine() != null)
    {
        i++;
    }
    return i;
    }
}

private static int countWithRead(string filePath)
{
    using (StreamReader _reader = new StreamReader(filePath))
    {
    int c = 0, count = 0;
    while ((c = _reader.Read()) != -1)
    {
        if (c == 10)
        {
        count++;
        }
    }
    return count;
    }            
}

private static int countWithReadByte(string filePath)
{
    using (Stream s = new FileStream(filePath, FileMode.Open))
    {
    int i = 0;
    int b;

    b = s.ReadByte();
    while (b >= 0)
    {
        if (b == 10)
        {
        i++;
        }
        b = s.ReadByte();
    }
    return i;
    }
}

private static int countWithReadLinesCount(string filePath)
{
    return File.ReadLines(filePath).Count();
}

private static int countWithReadAndBufferSize(string filePath)
{
    int bufferSize = 512;

    using (Stream s = new FileStream(filePath, FileMode.Open))
    {
    int i = 0;
    byte[] b = new byte[bufferSize];
    int n = 0;

    n = s.Read(b, 0, bufferSize);
    while (n > 0)
    {
        i += countByteLines(b, n);
        n = s.Read(b, 0, bufferSize);
    }
    return i;
    }
}

private static int countByteLines(byte[] b, int n)
{
    int i = 0;
    for (int j = 0; j < n; j++)
    {
    if (b[j] == 10)
    {
        i++;
    }
    }

    return i;
}
like image 299
sergeidave Avatar asked Jan 09 '13 17:01

sergeidave


People also ask

What is the use of StreamReader?

StreamReader is designed for character input in a particular encoding, whereas the Stream class is designed for byte input and output. Use StreamReader for reading lines of information from a standard text file. This type implements the IDisposable interface.

What does StreamReader return?

StreamReader. ReadLine() method reads a line of characters from the current stream and returns the data as a string.

Is StreamReader disposable?

Here is one classic block of code where stream and stream reader are used. They both are disposable.

When manipulating a file What is the difference between foreach and ReadLines methods?

Explanation: The difference between the method foreach and the method readlines is that the method foreach is associated with a block. However, unlike the method readlines, the method foreach does not return an array. 9.


2 Answers

No, it is not. Point is - it materializes the strings, which is not needed.

To COUNT it you are much better off to ignore the "string" Part and to go the "line" Part.

a LINE is a seriees of bytes ending with \r\n (13, 10 - CR LF) or another marker.

Just run along the bytes, in a buffered stream, counting the number of appearances of your end of line marker.

like image 153
TomTom Avatar answered Sep 21 '22 05:09

TomTom


The best way to know how to do this fast is to think about the fastest way to do it without using C/C++.

In assembly there is a CPU level operation that scans memory for a character so in assembly you would do the following

  • Read big part (or all) of the file into memory
  • Execute the SCASB command
  • Repeat as needed

So, in C# you want the compiler to get as close to that as possible.

like image 44
Hogan Avatar answered Sep 21 '22 05:09

Hogan