Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to replace line-endings in VIM

How can I replace all line-endings in big file (>100MB)? I have tried to do

:%s/\n/, /g

but it's too slow.

like image 481
Frankovskyi Bogdan Avatar asked Mar 04 '10 14:03

Frankovskyi Bogdan


4 Answers

So, I went through and tested/timed some of the answers that were given by other people, plus a python answer of my own. Here is what I got:

tr:

> time tr "\n" "," < lines > line
real    0m1.617s
user    0m0.100s
sys     0m1.520s

python:

> time python -c 'import sys; print sys.stdin.read().replace("\n",", "),' < lines > line
real    0m1.663s
user    0m0.060s
sys     0m1.610s

awk:

> time awk '{printf("%s, ", $0)}' lines > line                                 
real    0m1.998s
user    0m0.390s
sys     0m1.600s

perl:

> time perl -e 'while (<>) { chomp; print "$_, " }' lines > line
real    0m2.100s
user    0m0.590s
sys     0m1.510s

sed:

> time sed 's/$/, /g' lines > line                                             
real    0m6.673s
user    0m5.050s
sys     0m1.630s

Here is the file I used:

> ls -lh lines
-rw-r--r-- 1 some one 101M 2010-03-04 19:54 lines
> wc -l < lines
1300000
> head -n 3 < lines
The pretty pink puma pounced on the unsuspecting aardvark, the scientist watched.
The pretty pink puma pounced on the unsuspecting aardvark, the scientist watched.
The pretty pink puma pounced on the unsuspecting aardvark, the scientist watched.
> head -n 1 < lines | wc -c
82

Originally the timings were taken in cygwin, they have now been taken with fully updated ubuntu 9.10. Also, the text files size was increased to 100 megs, with lines 80ish characters wide. As you can see pretty much anything other than sed is a good idea.

like image 145
Seamus Connor Avatar answered Nov 08 '22 19:11

Seamus Connor


:%s/$/, / followed by a :1,$j might be faster. Otherwise, do it in an external utility:

perl -e 'while (<>) { chomp; print "$_, " }' input_file > output_file

awk '{printf("%s, ", $0)}' input_file > output_file

Don't know off the top of my head which would be fastest.

like image 20
Cascabel Avatar answered Nov 08 '22 18:11

Cascabel


Use this Perl script to go through your file; it'd be faster than holding everything in memory with VIM. Just pipe output to a new file.

#!/usr/local/bin/perl

while (<>) {
  $_ =~ s/\n/,/g;
  print $_;
}
like image 2
OverClocked Avatar answered Nov 08 '22 18:11

OverClocked


Do you have to do this in vim?

There is nice Unix utility that does character based translation. It'c called tr. Some reference.

In your case it would be:

tr "\n" "," &lt input_file &gt output_file
like image 1
pajton Avatar answered Nov 08 '22 19:11

pajton