I am trying to read some text files, where each line needs to be processed. At the moment I am just using a StreamReader, and then reading each line individually.
I am wondering whether there is a more efficient way (in terms of LoC and readability) to do this using LINQ without compromising operational efficiency. The examples I have seen involve loading the whole file into memory, and then processing it. In this case however I don't believe that would be very efficient. In the first example the files can get up to about 50k, and in the second example, not all lines of the file need to be read (sizes are typically < 10k).
You could argue that nowadays it doesn't really matter for these small files, however I believe that sort of the approach leads to inefficient code.
First example:
// Open file using(var file = System.IO.File.OpenText(_LstFilename)) { // Read file while (!file.EndOfStream) { String line = file.ReadLine(); // Ignore empty lines if (line.Length > 0) { // Create addon T addon = new T(); addon.Load(line, _BaseDir); // Add to collection collection.Add(addon); } } }
Second example:
// Open file using (var file = System.IO.File.OpenText(datFile)) { // Compile regexs Regex nameRegex = new Regex("IDENTIFY (.*)"); while (!file.EndOfStream) { String line = file.ReadLine(); // Check name Match m = nameRegex.Match(line); if (m.Success) { _Name = m.Groups[1].Value; // Remove me when other values are read break; } } }
The standard way of reading a line of text in C is to use the fgets function, which is fine if you know in advance how long a line of text could be.
The C library function char *fgets(char *str, int n, FILE *stream) reads a line from the specified stream and stores it into the string pointed to by str. It stops when either (n-1) characters are read, the newline character is read, or the end-of-file is reached, whichever comes first.
This means that even a tab ( \t ) in the format string can match a single space character in the input stream. Each call to fscanf() reads one line from the file.
It's simpler to read a line and check whether or not it's null than to check for EndOfStream all the time.
However, I also have a LineReader
class in MiscUtil which makes all of this a lot simpler - basically it exposes a file (or a Func<TextReader>
as an IEnumerable<string>
which lets you do LINQ stuff over it. So you can do things like:
var query = from file in Directory.GetFiles("*.log") from line in new LineReader(file) where line.Length > 0 select new AddOn(line); // or whatever
The heart of LineReader
is this implementation of IEnumerable<string>.GetEnumerator
:
public IEnumerator<string> GetEnumerator() { using (TextReader reader = dataSource()) { string line; while ((line = reader.ReadLine()) != null) { yield return line; } } }
Almost all the rest of the source is just giving flexible ways of setting up dataSource
(which is a Func<TextReader>
).
You can write a LINQ-based line reader pretty easily using an iterator block:
static IEnumerable<SomeType> ReadFrom(string file) { string line; using(var reader = File.OpenText(file)) { while((line = reader.ReadLine()) != null) { SomeType newRecord = /* parse line */ yield return newRecord; } } }
or to make Jon happy:
static IEnumerable<string> ReadFrom(string file) { string line; using(var reader = File.OpenText(file)) { while((line = reader.ReadLine()) != null) { yield return line; } } } ... var typedSequence = from line in ReadFrom(path) let record = ParseLine(line) where record.Active // for example select record.Key;
then you have ReadFrom(...)
as a lazily evaluated sequence without buffering, perfect for Where
etc.
Note that if you use OrderBy
or the standard GroupBy
, it will have to buffer the data in memory; ifyou need grouping and aggregation, "PushLINQ" has some fancy code to allow you to perform aggregations on the data but discard it (no buffering). Jon's explanation is here.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With