Menu
I have inherited this sed script snippet that attempts to remove certain empty spaces:
s/[\s\t]*|/|/g
s/|[\s\t]*/|/g
s/[\s] *$//g
s/^|/null|/g
that operates on a file that is around 1Gb large. This script runs for 2 hours on our unix server. Any ideas how to speed it up?
Notes that the \s stands for a space and \t stands for a tab, the actual script uses the actual space and tab and not those symbols
The input file is a pipe delimited file and is located locally not on the network. The 4 lines are in a file executed with sed -f
558
If you want to run faster than that then you need to put your script in a custom block and set it to run without screen refresh, the script will run as fast as it can at the expense of any other scripts and as the name implies even the expense of drawing the screen.
sed is a stream editor; it only processes one line at a time. This means that its memory footprint is minuscule. Unlike a text editor such as emacs or vim , it does not need to maintain the entire copy of the file in memory.
Sed command in unix is not working on file size greater than 3GB - Stack Overflow. Stack Overflow for Teams – Start collaborating and sharing organizational knowledge.
I find awk much faster than sed . You can speed up grep if you don't need real regular expressions but only simple fixed strings (option -F). If you want to use grep, sed, awk together in pipes, then I would place the grep command first if possible.
The best I was able to do with sed, was this script:
s/[\s\t]*|[\s\t]*/|/g
s/[\s\t]*$//
s/^|/null|/
In my tests, this ran about 30% faster than your sed script. The increase in performance comes from combining the first two regexen and omitting the "g" flag where it's not needed.
However, 30% faster is only a mild improvement (it should still take about an hour and a half to run the above script on your 1GB data file). I wanted to see if I could do any better.
In the end, no other method I tried (awk, perl, and other approaches with sed) fared any better, except -- of course -- a plain ol' C implementation. As would be expected with C, the code is a bit verbose for posting here, but if you want a program that's likely going to be faster than any other method out there, you may want to take a look at it.
In my tests, the C implementation finishes in about 20% of the time it takes for your sed script. So it might take about 25 minutes or so to run on your Unix server.
I didn't spend much time optimizing the C implementation. No doubt there are a number of places where the algorithm could be improved, but frankly, I don't know if it's possible to shave a significant amount of time beyond what it already achieves. If anything, I think it certainly places an upper limit on what kind of performance you can expect from other methods (sed, awk, perl, python, etc).
Edit: The original version had a minor bug that caused it to possibly print the wrong thing at the end of the output (e.g. could print a "null" that shouldn't be there). I had some time today to take a look at it and fixed that. I also optimized away a call to strlen() that gave it another slight performance boost.
127
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
