Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Batch split a text file

I have this batch file to split a txt file:

@echo off
for /f "tokens=1*delims=:" %%a in ('findstr /n "^" "PASSWORD.txt"') do for /f "delims=~" %%c in ("%%~b") do >"text%%a.txt" echo(%%c
pause

It works but it splits it line by line. How do i make it split it every 5000 lines. Thanks in advance.

Edit:

I have just tried this:

@echo off
setlocal ENABLEDELAYEDEXPANSION
REM Edit this value to change the name of the file that needs splitting. Include the extension.
SET BFN=passwordAll.txt
REM Edit this value to change the number of lines per file.
SET LPF=50000
REM Edit this value to change the name of each short file. It will be followed by a number indicating where it is in the list.
SET SFN=SplitFile

REM Do not change beyond this line.

SET SFX=%BFN:~-3%

SET /A LineNum=0
SET /A FileNum=1

For /F "delims==" %%l in (%BFN%) Do (
SET /A LineNum+=1

echo %%l >> %SFN%!FileNum!.%SFX%

if !LineNum! EQU !LPF! (
SET /A LineNum=0
SET /A FileNum+=1
)

)
endlocal
Pause
exit

But i get an error saying: Not enough storage is available to process this command

like image 367
09stephenb Avatar asked Mar 31 '26 05:03

09stephenb


1 Answers

This will give you the a basic skeleton. Adapt as needed

@echo off
    setlocal enableextensions disabledelayedexpansion

    set "nLines=5000"
    set "line=0"

    for /f "usebackq delims=" %%a in ("passwords.txt") do (
        set /a "file=line/%nLines%", "line+=1"
        setlocal enabledelayedexpansion
        for %%b in (!file!) do (
            endlocal
            >>"passwords_%%b.txt" echo(%%a
        )
    )

    endlocal

EDITED

As the comments indicated, a 4.3GB file is hard to manage. for /f needs to load the full file into memory, and the buffer needed is twice this size as the file is converted to unicode in memory.

This is a fully ad hoc solution. I've not tested it over a file that high, but at least in theory it should work (unless 5000 lines needs a lot of memory, it depends of the line length)

AND, with such a file it will be SLOW

@echo off
    setlocal enableextensions disabledelayedexpansion

    set "line=0"
    set "tempFile=%temp%\passwords.tmp"

    findstr /n "^" passwords.txt > "%tempFile%"
    for /f %%a in ('type passwords.txt ^| find /c /v "" ') do set /a "nFiles=%%a/5000"

    for /l %%a in (0 1 %nFiles%) do (
        set /a "e1=%%a*5", "e2=e1+1", "e3=e2+1", "e4=e3+1", "e5=e4+1"
        setlocal enabledelayedexpansion
        if %%a equ 0 (
            set "e=/c:"[1-9]:" /c:"[1-9][0-9]:" /c:"[1-9][0-9][0-9]:" /c:"!e2![0-9][0-9][0-9]:" /c:"!e3![0-9][0-9][0-9]:" /c:"!e4![0-9][0-9][0-9]:" /c:"!e5![0-9][0-9][0-9]:" "
        ) else (
            set "e=/c:"!e1![0-9][0-9][0-9]:" /c:"!e2![0-9][0-9][0-9]:" /c:"!e3![0-9][0-9][0-9]:" /c:"!e4![0-9][0-9][0-9]:" /c:"!e5![0-9][0-9][0-9]:" "
        )
        for /f "delims=" %%e in ("!e!") do (
            endlocal & (for /f "tokens=1,* delims=:" %%b in ('findstr /r /b %%e "%tempFile%"') do @echo(%%c)>passwords_%%a.txt
        )
    )

    del "%tempFile%" >nul 2>nul

    endlocal

EDITED, again: Previous code will not correctly work for lines starting with a colon, as it has been used as a delimiter in the for command to separate line numbers from data.

For an alternative, still pure batch but still SLOW

@echo off
    setlocal enableextensions disabledelayedexpansion

    set "nLines=5000"
    set "line=0"
    for /f %%a in ('type passwords.txt^|find /c /v ""') do set "fileLines=%%a"

    < "passwords.txt" (for /l %%a in (1 1 %fileLines%) do (
        set /p "data="
        set /a "file=line/%nLines%", "line+=1"
        setlocal enabledelayedexpansion
        >>"passwords_!file!.txt" echo(!data!
        endlocal
    ))

    endlocal
like image 82
MC ND Avatar answered Apr 02 '26 23:04

MC ND



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!