I am creating a method in C# which generates a text file for a Google Product Feed. The feed will contain upwards of 30,000 records and the text file currently weighs in at ~7Mb.
Here's the code I am currently using (some lines removed for brevity's sake).
public static void GenerateTextFile(string filePath) { var sb = new StringBuilder(1000); sb.Append("availability").Append("\t"); sb.Append("condition").Append("\t"); sb.Append("description").Append("\t"); // repetitive code hidden for brevity ... sb.Append(Environment.NewLine); var items = inventoryRepo.GetItemsForSale(); foreach (var p in items) { sb.Append("in stock").Append("\t"); sb.Append("used").Append("\t"); sb.Append(p.Description).Append("\t"); // repetitive code hidden for brevity ... sb.AppendLine(); } using (StreamWriter outfile = new StreamWriter(filePath)) { result.Append("Writing text file to disk.").AppendLine(); outfile.Write(sb.ToString()); } }
I am wondering if StringBuilder is the right tool for the job. Would there be performance gains if I used a TextWriter instead?
I don't know a ton about IO performance so any help or general improvements would be appreciated. Thanks.
To be able to open such large CSV files, you need to download and use a third-party application. If all you want is to view such files, then Large Text File Viewer is the best choice for you. For actually editing them, you can try a feature-rich text editor like Emacs, or go for a premium tool like CSV Explorer.
Solution 1: Download a Dedicated Large File Viewer If all you need to do is read the large file, you can download a dedicated large file viewer such as the Large Text File Viewer. Such tools will open large text files with ease.
File I/O operations are generally well optimized in modern operating systems. You shouldn't try to assemble the entire string for the file in memory ... just write it out piece by piece. The FileStream
will take care of buffering and other performance considerations.
You can make this change easily by moving:
using (StreamWriter outfile = new StreamWriter(filePath)) {
to the top of the function, and getting rid of the StringBuilder
writing directly to the file instead.
There are several reasons why you should avoid building up large strings in memory:
StringBuilder
has to increase its capacity as you write to it, resulting in reallocation and copying of memory.StringBuilder
contents to a file you have to use ToString()
which effectively doubles the memory consumption of the process since both copies must be in memory for a period of time. This operation may also fail if your address space is sufficiently fragmented, such that a single contiguous block of memory cannot be allocated.Just move the using
statement so it encompasses the whole of your code, and write directly to the file. I see no point in keeping it all in memory first.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With