How can I split a text file using PowerShell?

Question

I need to split a large (500 MB) text file (a log4net exception file) into manageable chunks like 100 5 MB files would be fine.

I would think this should be a walk in the park for PowerShell. How can I do it?

Answer 1

A word of warning about some of the existing answers - they will run very slow for very big files. For a 1.6 GB log file I gave up after a couple of hours, realising it would not finish before I returned to work the next day.

Two issues: the call to Add-Content opens, seeks and then closes the current destination file for every line in the source file. Reading a little of the source file each time and looking for the new lines will also slows things down, but my guess is that Add-Content is the main culprit.

The following variant produces slightly less pleasant output: it will split files in the middle of lines, but it splits my 1.6 GB log in less than a minute:

$from = "C:\temp\large_log.txt" $rootName = "C:\temp\large_log_chunk" $ext = "txt" $upperBound = 100MB   $fromFile = [io.file]::OpenRead($from) $buff = new-object byte[] $upperBound $count = $idx = 0 try {     do {         "Reading $upperBound"         $count = $fromFile.Read($buff, 0, $buff.Length)         if ($count -gt 0) {             $to = "{0}.{1}.{2}" -f ($rootName, $idx, $ext)             $toFile = [io.file]::OpenWrite($to)             try {                 "Writing $count to $to"                 $tofile.Write($buff, 0, $count)             } finally {                 $tofile.Close()             }         }         $idx ++     } while ($count -gt 0) } finally {     $fromFile.Close() }