I am using string.split() in my C# code for reading tab separated file. I am facing "OutOfMemory exception" as mentioned below in code sample.
Here I would like to know why problem is coming for file having size 16 MB?
This is right approach or not?
using (StreamReader reader = new StreamReader(_path))
{
//...........Load the first line of the file................
string headerLine = reader.ReadLine();
MeterDataIPValueList objMeterDataList = new MeterDataIPValueList();
string[] seperator = new string[1]; //used to sepreate lines of file
seperator[0] = "\r\n";
//.............Load Records of file into string array and remove all empty lines of file.................
string[] line = reader.ReadToEnd().Split(seperator, StringSplitOptions.RemoveEmptyEntries);
int noOfLines = line.Count();
if (noOfLines == 0)
{
mFileValidationErrors.Append(ConstMsgStrings.headerOnly + Environment.NewLine);
}
//...............If file contains records also with header line..............
else
{
string[] headers = headerLine.Split('\t');
int noOfColumns = headers.Count();
//.........Create table structure.............
objValidateRecordsTable.Columns.Add("SerialNo");
objValidateRecordsTable.Columns.Add("SurveyDate");
objValidateRecordsTable.Columns.Add("Interval");
objValidateRecordsTable.Columns.Add("Status");
objValidateRecordsTable.Columns.Add("Consumption");
//........Fill objValidateRecordsTable table by string array contents ............
int recordNumber; // used for log
#region ..............Fill objValidateRecordsTable.....................
seperator[0] = "\t";
for (int lineNo = 0; lineNo < noOfLines; lineNo++)
{
recordNumber = lineNo + 1;
**string[] recordFields = line[lineNo].Split(seperator, StringSplitOptions.RemoveEmptyEntries);** // Showing me error when we split columns
if (recordFields.Count() == noOfColumns)
{
//Do processing
}
I use my own. It has been tested with 10 unit tests..
public static class StringExtensions
{
// the string.Split() method from .NET tend to run out of memory on 80 Mb strings.
// this has been reported several places online.
// This version is fast and memory efficient and return no empty lines.
public static List<string> LowMemSplit(this string s, string seperator)
{
List<string> list = new List<string>();
int lastPos = 0;
int pos = s.IndexOf(seperator);
while (pos > -1)
{
while(pos == lastPos)
{
lastPos += seperator.Length;
pos = s.IndexOf(seperator, lastPos);
if (pos == -1)
return list;
}
string tmp = s.Substring(lastPos, pos - lastPos);
if(tmp.Trim().Length > 0)
list.Add(tmp);
lastPos = pos + seperator.Length;
pos = s.IndexOf(seperator, lastPos);
}
if (lastPos < s.Length)
{
string tmp = s.Substring(lastPos, s.Length - lastPos);
if (tmp.Trim().Length > 0)
list.Add(tmp);
}
return list;
}
}
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With