Transparently mount a tar.gz archive with Python

Question

How can I mount a tar.gz archive transparently with Python?

I have a tar.gz archive whose contents have to be read by an external program. The contents will only be needed temporarily. I could just unpack it to a temporary folder and point my external program there to read it. Afterwards, I could just delete the temp folder again. However, the archives may be large (>1 GB when extracted) so that unpacking them will take up a lot of space on the disk. My server is rather weak regarding HD performance and I cannot waste space ad lib but it does have a lot of RAM and CPU power.

That's why I want to try to mount the archive transparently without unpacking it entirely. I came across archivemount which seems to do exactly what I want. Is there a way to do what archivemount does in pure Python? No subprocess.call "solutions", please. It should run on 64-bit Linux.

I believe there should be a smart way to use tarfile to access archive's contents and then fusepy to create a user-space file system which exposes the contents of the archive. Has anyone already put these pieces together? Any ideas?

If you think that this is not a good idea, please post relevant comments. If you know what is better, please comment.

Answer 1

As of version 0.3.1 of my ratarmount module, you can use it or take a look at its source to mount a .tar.gz in Python. The gzip seeking support is from the dependency indexed_gzip. Ratarmount itself is based on tarindexer, which implements the idea to use tarfile to get offsets and then seek to it. But, ratarmount adds a FUSE layer among other usability and performance features.

You can install ratarmount from PyPI:

pip3 install --user ratarmount

and then call its command line interface directly from python like so:

import ratarmount
ratarmount.cli( [ '--help' ] )
ratarmount.cli( [ pathToTar, pathToMountPoint ] )

The heart of the module is as you already surmised tarfile, which is used to iterate over all TarInfo objects and create a list of filepath,offset,size, which then can be used to seek directly to the offset in the raw tar file and the simply read the next size bytes. This works because TAR is that simple of a format.

Here is the unoptimized and very bare core idea:

import sys
import tarfile
from indexed_gzip import IndexedGzipFile

targzfile = sys.argv[1]
filetoprint = sys.argv[2]

index = {} # path : ( offset, size )

file = IndexedGzipFile( targzfile )
for tarinfo in tarfile.open( fileobj = file, mode = 'r|' ):
    index[tarinfo.name] = ( tarinfo.offset_data, tarinfo.size )

# at this point you could save or load the index for faster consecutive file seeks

file.seek( index[filetoprint][0] )
sys.stdout.buffer.write( file.read( index[filetoprint][1] ) )

The above example was tested to work with:

wget -O- 'https://ftp.mozilla.org/pub/firefox/releases/70.0/linux-x86_64/en-US/firefox-70.0.tar.bz2' | bzip2 -d -c | gzip > firefox.tgz
python3 minimal-example.py firefox.tgz firefox/updater.ini