Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Combine multiple files into single file

Code:

static void MultipleFilesToSingleFile(string dirPath, string filePattern, string destFile)
{
    string[] fileAry = Directory.GetFiles(dirPath, filePattern);

    Console.WriteLine("Total File Count : " + fileAry.Length);

    using (TextWriter tw = new StreamWriter(destFile, true))
    {
        foreach (string filePath in fileAry)
        {
            using (TextReader tr = new StreamReader(filePath))
            {
                tw.WriteLine(tr.ReadToEnd());
                tr.Close();
                tr.Dispose();
            }
            Console.WriteLine("File Processed : " + filePath);
        }

        tw.Close();
        tw.Dispose();
    }
}

I need to optimize this as its extremely slow: takes 3 minutes for 45 files of average size 40 — 50 Mb XML file.

Please note: 45 files of an average 45 MB is just one example, it can be n numbers of files of m size, where n is in thousands & m can be of average 128 Kb. In short, it can vary.

Could you please provide any views on optimization?

like image 958
Pratik Avatar asked Jan 25 '13 15:01

Pratik


People also ask

How do I consolidate files?

Step 1: Open all files (workbooks) that contain the data you want to consolidate. Step 2: Ensure the data is organized in the same way (see example below). Step 3: On the Data ribbons, select Data Tools and then Consolidate. Step 4: Select the method of consolidation (in our example, it's Sum).

How do you put multiple documents into one PDF?

Convert multiple files into a single PDF.Select Combine Files. Drag and drop your files into the conversion frame. You can also locate your files manually. Highlight the files you want to combine and select Save.


1 Answers

General answer

Why not just use the Stream.CopyTo(Stream destination) method?

private static void CombineMultipleFilesIntoSingleFile(string inputDirectoryPath, string inputFileNamePattern, string outputFilePath)
{
    string[] inputFilePaths = Directory.GetFiles(inputDirectoryPath, inputFileNamePattern);
    Console.WriteLine("Number of files: {0}.", inputFilePaths.Length);
    using (var outputStream = File.Create(outputFilePath))
    {
        foreach (var inputFilePath in inputFilePaths)
        {
            using (var inputStream = File.OpenRead(inputFilePath))
            {
                // Buffer size can be passed as the second argument.
                inputStream.CopyTo(outputStream);
            }
            Console.WriteLine("The file {0} has been processed.", inputFilePath);
        }
    }
}

Buffer size adjustment

Please, note that the mentioned method is overloaded.

There are two method overloads:

  1. CopyTo(Stream destination).
  2. CopyTo(Stream destination, int bufferSize).

The second method overload provides the buffer size adjustment through the bufferSize parameter.

like image 76
Sergey Vyacheslavovich Brunov Avatar answered Oct 14 '22 22:10

Sergey Vyacheslavovich Brunov