Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Split a file using windows batch script

I have a csv file and i need to split it in to n files such that each split file should not exceed 100 mb. I need to achieve it in windows batch script. I tried the below way but its taking lot of time as my unsplit file is in GBs

@echo off
setlocal enableextensions enabledelayedexpansion
set count=1
set maxbytesize=100000000
set size=1
type NUL > output_1.csv

FOR /F  "tokens=*" %%i in (myfile.csv) do (
FOR /F "usebackq" %%A in ('!filename!_!count!.csv') do (
set size=%%~zA) 
if !size! LSS !maxbytesize! (
echo %%i>>!filename!_!count!.csv) else (
set /a count+=1 
echo %%i>>!filename!_!count!.csv 
))

please let me know if there is a better way to achieve this. I cant go to any other scripting languages as my server is windows

like image 431
chethan Avatar asked Dec 06 '12 13:12

chethan


People also ask

How do I split a file into parts in Windows?

First up, right-click the file you want to split into smaller pieces, then select 7-Zip > Add to Archive. Give your archive a name. Under Split to Volumes, bytes, input the size of split files you want. There are several options in the dropdown menu, although they may not correspond to your large file.

How do I split a file into multiple files?

Open the Zip file. Open the Tools tab. Click the Split Size dropdown button and select the appropriate size for each of the parts of the split Zip file. If you choose Custom Size in the Split Size dropdown list, another small window will open and allow you to enter in a custom size specified in megabytes.


1 Answers

This would do the trick assuming your lines are roughly the same size.

Its advantage is that it is only a 2 pass solution, One for counting the lines and the other for printing them.

@rem echo off

@rem usage: batchsplit.bat <file-to-split> <size-limit>
@rem it will generate files named <file-to-split>.part_NNN

setlocal EnableDelayedExpansion

set FILE_TO_SPLIT=%1
set SIZE_LIMIT=%2

for /f %%s in ('dir /b %FILE_TO_SPLIT%') do set SIZE=%%~Zs
for /f %%c in ('type "%FILE_TO_SPLIT%"^|find "" /v /c') do set LINE_COUNT=%%c

set /a AVG_LINE_SIZE=%SIZE%/%LINE_COUNT%
set /a LINES_PER_PART=%SIZE_LIMIT%/%AVG_LINE_SIZE%

set "cmd=findstr /R /N "^^" %FILE_TO_SPLIT%"

for /f "tokens=1,2* delims=:" %%a in ('!cmd!') do @(
    set /a ccc = %%a / %LINES_PER_PART%
    echo %%b >> %FILE_TO_SPLIT%.part_!ccc!
)

save it as batchsplit.bat and run it using:

batchsplit.bat myfile.csv 100000000
like image 172
Amnon Avatar answered Oct 18 '22 18:10

Amnon