In Docker, apt-get install fails with "Failed to fetch http://archive.ubuntu.com/ ... 404 Not Found" errors. Why? How can we get past it?

Question

My team uses Docker (with ubuntu:14.04 base image) for local development and we often have to rebuild some or all of our images. But we often get failures downloading packages with apt-get install, even immediately after running apt-get -y update. For instance, today I see

Err http://archive.ubuntu.com/ubuntu/ trusty-security/main libxml2 amd64 2.9.1+dfsg1-3ubuntu4.7
  404  Not Found [IP: 91.189.88.161 80]
Err http://archive.ubuntu.com/ubuntu/ trusty-security/main libxml2-dev amd64 2.9.1+dfsg1-3ubuntu4.7
  404  Not Found [IP: 91.189.88.161 80]
Fetched 84.7 MB in 1min 6s (1281 kB/s)
Unable to correct missing packages.
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/libx/libxml2/libxml2_2.9.1+dfsg1-3ubuntu4.7_amd64.deb  404  Not Found [IP: 91.189.88.161 80]

E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/libx/libxml2/libxml2-dev_2.9.1+dfsg1-3ubuntu4.7_amd64.deb  404  Not Found [IP: 91.189.88.161 80]

E: Aborting install.

Apparently the specific version of a particular package has been deleted from the archive and replaced with a slightly differently named patch version. For instance, the above error is looking for libxml2_2.9.1+dfsg1-3ubuntu4.7_amd64.deb but the version on the server is libxml2_2.9.1+dfsg1-3ubuntu4.8_amd64.deb.

Often this is solvable by removing the base image (docker rmi ubuntu:14.04) and rebuilding; the newly downloaded ubuntu image has the correct patch number and finds the right archive file. But even this doesn't always work -- probably due to a delay between a new minor upgrade to Ubuntu's dependency db and the deployment of that new ubuntu:14.04 image onto Docker Hub.

We've tried using apt-get flags --fix-missing and --fix-broken and those don't consistently work either.

Any other ideas?

apt-get install fails with Not Found error because package removed from repository is a similar problem but the accepted answer is unacceptable because it's not possible to be automated. Our daily development process, including automatic build and deploy, is all scripted and using Docker and it's not practical to hack around inside a Dockerfile every time a particular archive goes missing (then remove the hack after a few hours or days).

---

In response to @prateek05, here's the /etc/apt/sources.list from the official ubuntu:14.04 docker image:

root@72daa1942714:/# cat /etc/apt/sources.list
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.

deb http://archive.ubuntu.com/ubuntu/ trusty main restricted
deb-src http://archive.ubuntu.com/ubuntu/ trusty main restricted

## Major bug fix updates produced after the final release of the
## distribution.
deb http://archive.ubuntu.com/ubuntu/ trusty-updates main restricted
deb-src http://archive.ubuntu.com/ubuntu/ trusty-updates main restricted

## Uncomment the following two lines to add software from the 'universe'
## repository.
## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team. Also, please note that software in universe WILL NOT receive any
## review or updates from the Ubuntu security team.
deb http://archive.ubuntu.com/ubuntu/ trusty universe
deb-src http://archive.ubuntu.com/ubuntu/ trusty universe
deb http://archive.ubuntu.com/ubuntu/ trusty-updates universe
deb-src http://archive.ubuntu.com/ubuntu/ trusty-updates universe

## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
# deb http://archive.ubuntu.com/ubuntu/ trusty-backports main restricted
# deb-src http://archive.ubuntu.com/ubuntu/ trusty-backports main restricted

deb http://archive.ubuntu.com/ubuntu/ trusty-security main restricted
deb-src http://archive.ubuntu.com/ubuntu/ trusty-security main restricted
deb http://archive.ubuntu.com/ubuntu/ trusty-security universe
deb-src http://archive.ubuntu.com/ubuntu/ trusty-security universe
# deb http://archive.ubuntu.com/ubuntu/ trusty-security multiverse
# deb-src http://archive.ubuntu.com/ubuntu/ trusty-security multiverse

Answer 1

You have stated that your Dockerfile contains RUN apt-get -y update as its own RUN instruction. However, due to build caching, if all changes to the Dockerfile occur later in the file, when docker build is run, Docker will reuse the intermediate image created the last time RUN apt-get -y update executed instead of running the command again, and so any recently-added or -edited apt-get install lines will be using old data, leading to the errors you've observed.

There are two ways to fix this:

1. Pass the --no-cache option to docker build, forcing every statement in the Dockerfile to be run every time the image is built.

2. Rewrite the Dockerfile to combine the apt-get commands in a single RUN instruction: RUN apt-get update && apt-get install foo bar .... This way, whenever the list of packages to install is edited, docker build will be forced to re-execute the entire RUN instruction and thus rerun apt-get update before installing.

The Dockerfile best practices page actually has an entire section on apt-get commands in Dockerfiles. I suggest you read it.

Answer 2

The issue could be potentially with the ubuntu sources

Check /etc/apt/sources.list

If you see deb http://archive.ubuntu.com/ubuntu main universe restricted multiverse , that could be the potential issue.

Fix that by replacing it with deb http://archive.ubuntu.com/ubuntu/ trusty main universe restricted multiverse

Alternatively it could be the mirror itself doesn't respond. archive.ubuntu.com

Name:   archive.ubuntu.com
Address: 91.189.88.152
Name:   archive.ubuntu.com
Address: 91.189.88.161
Name:   archive.ubuntu.com
Address: 91.189.88.149

Replace archive.ubuntu.com with a more trusted mirror say us.archive.ubuntu.com

Name:   us.archive.ubuntu.com
Address: 91.189.91.23
Name:   us.archive.ubuntu.com
Address: 91.189.91.26

---

(edit by the oiriginal asker):

Thanks, prateek05! My Dockerfile now starts:

FROM ubuntu:14.04
RUN sed -i'' 's/archive\.ubuntu\.com/us\.archive\.ubuntu\.com/' /etc/apt/sources.list
RUN apt-get -y update

and it seems to be working. But since this is a sporadic issue, only time will tell...

Answer 3

In my case the problem was caused by the parent image since it had not cleared the apt cache properly.

I solve the problem including cleaning commands before the first apt update ...

RUN apt clean && \
    rm -rf /var/lib/apt/lists/* && \
    apt update && \
    ...

Hope this helps

Answer 4

found something that works!

so i found this: https://www.mail-archive.com/[email protected]/msg5682159.html

The solution is to create a pinning_file with the contents

# Written by ubuntu-advantage-tools
Package: *
Pin: release o=UbuntuESM, n=trusty
Pin-Priority: never

then add

COPY pinning_file /etc/apt/preferences.d/ubuntu-esm-infra-trusty

to you Dockerfile before you run the sudo apt-get -y update