gzipstream memory stream to file

Question

I am trying to compress JSON files using Gzip compression to be sent to another location. It needs to process 5,000 - 10,000 files daily, and I don't need the compressed version of the file on the local machine (they are actually being transferred to AWS S3 for long-term archiving).

Since I don't need them, I am trying to compress to a memory stream and then use that to write to AWS, rather than compress each one to disk. Whenever I try to do this, the files are broken (as in, when I open them in 7-Zip and try to open the JSON file inside, I get "Data error File is Broken).

The same thing happens when I try to write the memory stream to a local file, so I'm trying to solve that for now. Here's the code:

string[] files = Directory.GetFiles(@"C:\JSON_Logs");

foreach(string file in files)
{
    FileInfo fileToCompress = new FileInfo(file);
    using (FileStream originalFileStream = fileToCompress.OpenRead())
    {
        using (MemoryStream compressedMemStream = new MemoryStream())
        {
            using (GZipStream compressionStream = new GZipStream(compressedMemStream, CompressionMode.Compress))
            {
                originalFileStream.CopyTo(compressionStream);
                compressedMemStream.Seek(0, SeekOrigin.Begin);
                FileStream compressedFileStream = File.Create(fileToCompress.FullName + ".gz");

                //Eventually this will be the AWS transfer, but that's not important here
                compressedMemStream.WriteTo(compressedFileStream); 
            }
        }
    }      
}

Answer 1

Rearrange your using statements so the GZipStream is definitely done by the time you read the memory stream contents:

foreach(string file in files)
{
    FileInfo fileToCompress = new FileInfo(file);
    using (MemoryStream compressedMemStream = new MemoryStream())
    {
        using (FileStream originalFileStream = fileToCompress.OpenRead())
        using (GZipStream compressionStream = new GZipStream(
            compressedMemStream, 
            CompressionMode.Compress,
            leaveOpen: true))
        {
            originalFileStream.CopyTo(compressionStream);
        }
        compressedMemStream.Seek(0, SeekOrigin.Begin);

        FileStream compressedFileStream = File.Create(fileToCompress.FullName + ".gz");
        //Eventually this will be the AWS transfer, but that's not important here
        compressedMemStream.WriteTo(compressedFileStream); 
    }
}

Disposing a stream takes care of flushing and closing it.