How are apt-get Repositories Hosted/Managed/Architected?

Question

Whether I'm setting up a new VPS somewhere or installing an Ubuntu ISO on an old computer I have sitting around, when I want to get my favorite programming enviornment setup all I need to do is type

sudo apt-get install foo baz bar

and then my computer -- does something -- and at the end I usually have the software I'm after.

How does this work? Is apt-get downloading precompiled binaries? Or is it downloading source and building things for me? Or some sort of other packaged thing its installing? Or all three? Regardless of which, who's building these binaries, managing these build recipes, or putting together these packages? Whatever apt-get is doing -- where are the bits its sending me served from?

I'm vaguely aware that my composer/VPS will have a list of apt-get repositories it checks for packages. Are these just feeds hosted on an HTTP server somewhere that point at things? Or something else? Who hosts these feeds?

When I'm using a package manager like composer (for PHP), I know that there's a master list of packages at packagist.org composer repository, that a user can specify other packages repositories, and that repositories only point to version control systems where a package's source files are stored. I'd like to have a similar high level understanding of how apt-get works.

I realize this is an impossibly broad question -- pointers towards where in the FM I could read about how everything gets tied together would be a perfectly acceptable answer to me. (although if you're passionate about linux package management please don't let that stop you from answering)

Answer 1

My experience is based on Debian and Ubuntu:

Where does apt-get get software from?

By choosing a distribution like Ubuntu 16.04 for your system you are basically installing a bootstrap image that has package repositories configured. apt-get gets software for installing/updating from those repositories. Repositories for a system have to be configured in file /etc/apt/sources.list. For example, my installation of Ubuntu 16.04 on a VPS at a german provider contains the following lines which point to its own mirrored package repositories:

deb http://mirror.hetzner.de/ubuntu/packages  xenial           main restricted universe multiverse
deb http://mirror.hetzner.de/ubuntu/packages  xenial-backports main restricted universe multiverse
deb http://mirror.hetzner.de/ubuntu/packages  xenial-updates   main restricted universe multiverse
deb http://mirror.hetzner.de/ubuntu/security  xenial-security  main restricted universe multiverse

(i left some out) There are many mirrored repositories that are run by universities, companies, hosting providers, etc.

What's in a repository?

Each of those repositories has one or more index files that contain lists of all packages in it. So apt-get can determine which packages are available and solve dependencies. apt-get installs/updates software from those repositories by downloading packages via FTP or HTTP and installing them with the program dpkg.

The package format used by all Debian-based distributions (like Ubuntu) is .deb. It contains binaries so there are different .deb files for each architecture the distribution supports like "amd64" or "arm64" which obviously has to match the architecture of your system. You can also get .deb packages that contain the program source to build the software by yourself (lines starting with deb-src in /etc/apt/sources.list).

Who makes packages?

Each package is maintained by one or more package maintainers. Those take releases of the original software - which is called "upstream" in this context - and package them as .deb to be put onto a package repository. A whole toolchain for packaging exists which is basically automated testing/building/archiving based on a package recipe.