Get last line of a huge gziped file

Question

In a database backup process I generate a text dumpfile. As database is quite huge, dump file is huge too so I compresses it with gzip. Compression is done inline while dump is generated (thanks Unix pipe !).

At the process end, I check dump file validity by watching the last line and check the "Dump completed" string presence. In my script I do it by extracting last line into a variable:

str=`zcat ${PATHSAVE}/dumpFull.sql.gz | tail -n1`

As database dump file is huge (currently more than 200Gb) this end process check take huge time to run (currently more than 180 minutes).

I'm searching a way to extract quicker the last line of my .gz file ... any idea anyone ?

Note 1: For explain context, we can say database is MySql community, backup tool is mysqldump, generated dumpfile is a full text file. OS is CentOs. Backup script is Bash shell script.

Note 2: I'm aware about Percona xtraBackup but in my case I want to use mysqldump for this specific backup job. Time need for restauration is not an issue.

Answer 1

This is a job for a fifo (a pipe) and the tee command. Use this when making your backup.

mkfifo mypipe
tail mypipe -1 > lastline.txt & mysqldump whatever | tee mypipe | gzip >dump.gz
rm mypipe

What's going on?

mkfifo mypipe puts a fifo object into your current working directory. It looks like a file that you can write to, and read from, at the same time.

tail mypipe -1 >lastline.txt uses tail to read whatever you write to mypipe and save the last line in a file.

mysqldump whatever | tee mypipe | gzip >dump.gz does your dump operation, and pipes the output to the tee command. Tee writes the output to mypipe and pipes it along to gzip.

The & between the two parts of the command causes both parts to run at the same time.

rm mypipe gets rid of the fifo object.

Charles Duffy pointed out that some shells (including bash) have process substitution, and so your command can be simpler if you're using one of those shells.

 mysqldump whatever | tee >(tail -1 > lastline.txt ) | gzip >dump.gz

In this case the shell creates its own pipe for you.

Credit: Pipe output to two different commands