insert header to a file

Question

I would like to hear your directions on how to insert lines of header (all lines in a file) to another file (more bigger, several GB). I prefer the Unix/awk/sed ways of do that job.

# header I need to insert to another, they are in a file named "header".


##fileformat=VCFv4.0
##fileDate=20090805
##source=myImputationProgramV3.1
##reference=1000GenomesPilot-NCBI36
##phasing=partial
##INFO=<ID=NS,Number=1,Type=Integer,Description="Number of Samples With Data">
##INFO=<ID=DP,Number=1,Type=Integer,Description="Total Depth">
##INFO=<ID=AF,Number=.,Type=Float,Description="Allele Frequency">
##INFO=<ID=AA,Number=1,Type=String,Description="Ancestral Allele">
##INFO=<ID=DB,Number=0,Type=Flag,Description="dbSNP membership, build 129">
##INFO=<ID=H2,Number=0,Type=Flag,Description="HapMap2 membership">
##FILTER=<ID=q10,Description="Quality below 10">
##FILTER=<ID=s50,Description="Less than 50% of samples have data">
##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
##FORMAT=<ID=GQ,Number=1,Type=Integer,Description="Genotype Quality">
##FORMAT=<ID=DP,Number=1,Type=Integer,Description="Read Depth">
##FORMAT=<ID=HQ,Number=2,Type=Integer,Description="Haplotype Quality">
#CHROM POS     ID        REF ALT    QUAL FILTER INFO 

Answer 1

header="/name/of/file/containing/header"
for file in "$@"
do
    cat "$header" "$file" > /tmp/xx.$$
    mv /tmp/xx.$$ "$file"
done

You might prefer to locate the temporary file on the same file system as the file you are editing, but anything that requires inserting data at the front of the file is going to end up working very close to this. If you are going to be doing this all day, every day, you might assemble something a little slicker, but the chances are the savings will be minuscule (fractions of a second per file).

If you really, really must use sed, then I suppose you could use:

header="/name/of/file/containing/header"
for file in "$@"
do
    sed -e "0r $header" "$file" > /tmp/xx.$$
    mv /tmp/xx.$$ "$file"
done

The command reads the content of header 'after' line 0 (before line 1), and then everything else is passed through unchanged. This isn't as swift as cat though.

An analogous construct using awk is:

header="/name/of/file/containing/header"
for file in "$@"
do
    awk '{print}' "$header" "$file" > /tmp/xx.$$
    mv /tmp/xx.$$ "$file"
done

This simply prints each input line on the output; again, not as swift as cat.

One more advantage of cat over sed or awk; cat will work even if the big files are mainly binary data (it is oblivious to the content of the files). Both sed and awk are designed to handle data split into lines; while modern versions will probably handle even binary data fairly well, it is not what they are designed for.

Answer 2

I did it all with a Perl script, because I had to traverse a directory tree and handle various file types differently. The basic script was

#!perl -w
process_directory(".");

sub process_directory {
    my $dir = shift;
    opendir DIR, $dir or die "$dir: not a directory\n";
    my @files = readdir DIR;
    closedir DIR;
    foreach(@files) {
        next if(/^\./ or /bin/ or /obj/);  # ignore some directories
        if(-d "$dir/$_") {
            process_directory("$dir/$_");
        } else {
            fix_file("$dir/$_");
        }
    }
}

sub fix_file {
    my $file = shift;
    open SRC, $file or die "Can't open $file\n";
    my $file = "$file-f";
    open FIX, ">$fix" or die "Can't open $fix\n";
    print FIX <<EOT;
        -- Text to insert
EOT
    while(<SRC>) {
        print FIX;
    }
    close SRC;
    close FIX;
    my $oldfile = $file;
    $oldFile =~ s/(.*)\.\(\w+)$/$1-old.$2/;
    if(rename $file, $oldFile) {
        rename $fix, $file;
    }
}

Share and enjoy! Or not -- I'm no Perl hacker, so this is probably double-plus-unoptimal Perl code. Still, it worked for me!