Make Python script combined with linux packages easy installable for end-user

Question

I wrote a python script that uses numpy, multiprocessing, tqdm and a feq other Python libraries. Additionally, I run packages (e.g. samtools, bwa, GATK) set are necessary to be installed in linux (apt-get install).

I'd like to somehow wrap all these dependencies up to make the final installation as user-friendly and stable as possible.

It seems as pip is not an option here as non-python-packages are included for my example.

Maybe Docker or creating a conda environment with all these dependecies might be possible but I did not really get how to manage this.

Answer 1

I think a good starting point will be a Dockerfile, in which you start from an Ubuntu image and then install your desired dependencies(samtools, bwa) and also the python packages (placed in a requirements.txt)

• pip freeze > requirements.txt to have your desired python packages (numpy, tqdm etc)

• create a Dockerfile (which resides in the same directory as requirements.txt) e.g:

  FROM ubuntu:16.04
  
  RUN apt-get update && \
  apt-get upgrade -y && \
  apt-get install -y python && \
  apt-get install -y wget \
  curl \
  bc \
  unzip \
  less \
  bedtools \
  samtools \
  openjdk-8-jdk \
  tabix \
  bwa\
  python-pip\
  software-properties-common && \
  apt-get -y clean  && \
  apt-get -y autoclean  && \
  apt-get -y autoremove
  
  RUN mkdir -p /usr/src/app
  
  WORKDIR /usr/srv/app
  
  COPY . /usr/srv/app
  
  RUN pip install -r requirements.txt
  
  CMD ["/bin/bash"]
  

• build your docker image docker built --tag repository/imagename . (DOT is the current directory)

• start image docker run -dti --name test repository/imagename
• enter in the container to start working docker attach test
• as you can see, I tested it and inside the container I have all the desired tools and packages

---

• If you want to add new tools just add them in the Dockerfile and if you want to add python packages just expand the requirements.txt file.

• If you have a python script that does something and you need to include it into the Docker container , just put it in the same directory with the Dockerfile and requirements.txt and update the Dockerfile (in order to have a container that starts your python script), more precisely in the Dockerfile the last line will be CMD [ "python", "./my_script.py" ]

Answer 2

I wouldn't go as far as installing software onto some computer/server because it might break other software running on that system. Instead, I would create a deb package and set dependencies on which software has to be installed to guide the user through installation.

1. Create the deb directory structure
The directory deb holds final scripts/binaries for the package. source has your python code.

$ cd $HOME/path/to/some/directory
$ mkdir -p ./deb/DEBIAN ./source
$ touch ./Makefile ./source/Makefile ./deb/DEBIAN/{control,preinst} ./source/hello-world.py
$ chmod +x ./deb/DEBIAN/preinst

2. Add files
Below I've added the content of every file. You should be able to implement it to your needs.

./Makefile: This file is responsible for generating the source and the deb.

PACKAGE         := hello-world

DEB_DIR         := deb
DEB_FILE        := $(PACKAGE).deb

SOURCE_DIR      := source
SOURCE_IN       := $(SOURCE_DIR)/$(PACKAGE).py
SOURCE_OUT      := $(DEB_DIR)/usr/bin/

.PHONY: all clean
all:
        @make -C $(SOURCE_DIR) CROSS_COMPILE=$(CROSS_COMPILE)
        @mkdir -p $(SOURCE_OUT)
        @cp -r $(SOURCE_IN) $(SOURCE_OUT)
        @chmod -R a-s $(DEB_DIR)
        @dpkg-deb --build $(DEB_DIR) $(DEB_FILE)

clean:
        @rm -rf $(DEB_FILE) $(SOURCE_OUT)
        @make -C $(SOURCE_DIR) clean

./deb/DEBIAN/control: The control file for Debian's package manager.

Package: hello-world
Version: 1
Section: misc
Priority: optional
Architecture: all
Depends: libc6 (>= 2.24), python (>=2.7.13), virtualenv (>=1.11.6)
Maintainer: Your name <[email protected]>
Description: This is a deb package to install an application

./deb/DEBIAN/preinst: The file in which you can install python packages. It is run before the installation of hello-world.py. Again, make sure you don't just (re)install/update a library. It could cause incompatibility problems in other software.

#! /bin/bash
virtualenv -p /usr/bin/python3 venv/
source venv/bin/activate
pip install -r ./requirements.txt

./source/Makefile: A file I used to test this setup.

MYFILE=hello-world.py
all:
        @echo "#!/usr/bin/python\n\
        print('Hello world!')" > $(MYFILE)
        @chmod +x $(MYFILE)
clean:
        @rm $(MYFILE)

3. Installation and removal Installing is very easy, just use dpkg.

dpkg -i hello-world.deb # for installation
dpkg -r hello-world     # for removal

A nice feature to have is you don't need to worry about versioning your software, since it's done in the control file. Debian has a pretty good tutorial if you're interested in reading more.

EDIT:
Added virtualenv to example. Python packages should be installed in a virtual environment instead of globally.