In case you encounter this error message while creating docker images I want to draw your attention to a more unusual reason (DNS), how to fix it, and the somewhat embarrassing root cause in my case.
Have fun.
When setting up a docker installation on an Ubuntu Server 20.04 LTS system the build process for a docker image that had worked fine on my desktop computer failed with this misleading error message:
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.4/main: temporary error (try again later)
Strangely enough I could wget this file from the command line.
If you google this error the standard advice is to update docker. I did (to version 20.10.12) ... but it didn't help though.
With problems like this it's always good advise to try to replicate it with the most simple setup possible.
In this case:
- Get a small Linux system image from the repo (alpine)
- and start a command line inside the container: sh
- try to install a package (curl) within the container
$ docker run -it alpine:3.4 sh
/ # apk --update add curl
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.4/main: temporary error (try again later)
WARNING: Ignoring APKINDEX.167438ca.tar.gz: No such file or directory
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
ERROR: http://dl-cdn.alpinelinux.org/alpine/v3.4/community: temporary error (try again later)
WARNING: Ignoring APKINDEX.a2e6dac0.tar.gz: No such file or directory
ERROR: unsatisfiable constraints:
curl (missing):
required by: world[curl]
Docker did download the alpine image but inside the container downloading the APKINDEX failed. And yeah - I did wait and tried again later... no luck.
Back inside the container I tried:
/ # ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=117 time=14.639 ms
64 bytes from 8.8.8.8: seq=1 ttl=117 time=13.921 ms
64 bytes from 8.8.8.8: seq=2 ttl=117 time=13.956 ms
^C
/ # ping google.com
ping: bad address 'google.com'
Which means: I can reach the internet from inside the container but the DNS resolution obviously isn't working. Let's see who is resolving:
/ # cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients directly to
# all known uplink DNS servers. This file lists all configured search domains.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 129.168.0.1
search local
Who is 129.168.0.1 and how did it became the nameserver? To be honest - it was my fault (more on that later).
Using the only text editor available in a base alpine install I changed it to
/ # vi /etc/resolv.conf
nameserver 8.8.8.8
And yes, when using vi I always have to think very hard how to get the changes written back to disk.
Trying to add the package now works like a charm:
/ # apk --update add curl
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
(1/4) Installing ca-certificates (20161130-r0)
(2/4) Installing libssh2 (1.7.0-r0)
(3/4) Installing libcurl (7.60.0-r1)
(4/4) Installing curl (7.60.0-r1)
Executing busybox-1.24.2-r14.trigger
Executing ca-certificates
So the problem is definitely a faulty DNS name server .... How can this be fixed?
If I start the container with the --dns option ...
docker run --dns 8.8.8.8 -it alpine:3.4 sh
apk add runs without a problem. And if I check /etc/resolv.conf it says 8.8.8.8
Slight problem: --dns works with docker run but not with docker build.
You have to tell docker itself to use a different DNS server.
Googles first advise is modifying /etc/default/docker like this
DOCKER_OPTS="--dns=my-private-dns-server-ip --dns=8.8.8.8"
But Ubuntu 20.04 LTS uses systemd and in this case these settings are ignored.
You have to create an override files for this systemd service using
sudo systemctl edit docker.service
with the following content
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd --dns 8.8.8.8 -H fd:// --containerd=/run/containerd/containerd.sock
I first located the original docker.service file, copied the ExecStart line, and added the dns option.
The first empty ExecStart is needed to clear its original value before setting the new one... good to know (thanks - herbert.cc).
Everything worked.
So - who is 129.168.0.1? Well, it's a typo. It should read 192.168.0.1 - my cable modem.
I later found it in /etc/netplan/00-installer-config.yaml which sets up the machine's IP address, gateway, DNS resolver, etc.
I must have made this typo while installing the system onto the hard drive using the Ubuntu installer.
But why did the internet connection work at all? I could download files... the docker image for example.
My specific setup made the system use a fixed IP address (as servers usually need one) but it did NOT disable DHCP.
So eventually the DHCP server updated the faulty DNS resolver setting with the correct value and all worked fine.
It seems that docker samples the DNS nameserver during boot-up at a time after the yml-file had set the wrong value and before the DHCP server could replace it with the correct one. It then hands this value to the docker build and docker run processes instead of the nameserver currently in use. As those values are usually identical nobody does notice.
I don't know if I would call this a bug but it is unexpected behaviour.
Now you know.
Useful links
- https://serverfault.com/questions/612075/how-to-configure-custom-dns-server-with-docker
- https://serverfault.com/questions/1020732/docker-settings-in-ubuntu-20-04
- https://docs.docker.com/config/daemon/systemd/
- https://www.herbert.cc/blog/systemctl-docker-settings/
