If you are using Docker Hub, you will be aware of pull-request limits that are being enforced since November 2. Limits are determined based on the account type: If you are using the free tier of Docker Hub, you can only execute 100 pulls per 6 hours and per client IP for anonymous clients. Authenticated users are limited to 200 pulls within the same time frame.
From a user perspective, it is hard to predict if and when these limits will be reached. Since many CI/CD systems are using anonymous pulls, this yields the risk of unpredicted outages on random new pulled containers when the limit is reached.
With our products Kubermatic Kubernetes Platform (KKP) and Kubermatic KubeOne, users can easily mitigate the risk by setting up a Registry Mirror. In this blog post, we explain how to. But before we start, let’s first take a look at what a pull actually means.
What Is a Pull?
According to Docker, pulls are defined as the following:
- A pull request is defined as up to two GET requests on registry manifest URLs (/v2/*/manifests/*)
- A normal image pull makes a single manifest request
- A pull request for a multi-arch image makes two manifest requests
- HEAD requests are not counted
- Limits are applied based on the user doing the pull, and not based on the image being pulled or its owner
Mitigate the Risk With Registry Mirroring
To avoid any problems with the Docker Hub rate limits, we recommend setting up an internal registry mirror that can be used by Kubermatic Kubernetes Platform and Kubermatic KubeOne. Example enterprise-ready registries are Harbour or Jfrog Artifactory
Example Registry Mirror
Kubermatic Kubernetes Platform (KKP) contains an image-loader utility that can be used to collect and mirror all Docker images used by KKP. The utility works by faking the reconciliation and then checking the created Deployments/DaemonSets/… for the referenced images. A simple example use would be:
./image-loader -registry my.local.registry:5000
This will go through all reconciling functions as well as all addons to collect the Docker images.
Note: This does not include Helm charts, so Grafana for example will not be included here.
Helm charts are handled differently. For these, we currently have a helper script for our own offline test setup. This script renders all Helm charts, parses the YAML for Docker image references and then does what the image-loader did, basically: pull, retag, push to another registry.
An example on how to use it:
export TARGET_REGISTRY=my.registry.local:5000
helm template charts/cert-manager | ./hack/retag-images.sh
helm template charts/nginx-ingress-controller | \
./hack/retag-images.sh
helm template charts/oauth | ./hack/retag-images.sh
Care must be taken to run the helm-template with the same values as the YAML file. Otherwise, as the YAML file is later used to install the charts, the process could accidentally miss some images (e.g. Thanos images as Thanos is not enabled by default in Prometheus).
It is then necessary to configure all Helm charts to use these newly mirrored images. For Grafana this could look like this:
grafana:
image:
repository: "my.registry.local:5000/grafana/grafana"
utilImage:
repository: "my.registry.local:5000/kubermatic/util"
Registry Configuration with Kubermatic Kubernetes Platform
KKP can be configured with an overwriteRegistry flag (in the KubermaticConfiguration CRD). This flag contains the host/port of the Docker registry to use for all images used on the seed (i.e. inside the user cluster control plane namespace). Additionally, this registry is also used for the Container Linux Update Operator running inside the user cluster.
The flag cannot be used to overwrite more than the host, so docker.io/grafana/grafana becomes myregistry.corp.local/grafana/grafana, but the path part (/grafana/grafana) cannot be changed. Likewise, it’s also not possible to have different overwrites per registry (i.e. users cannot overwrite docker.io but leave quay.io intact).
Registry Configuration With Kubermatic KubeOne
Kubermatic KubeOne can be configured with the RegistryConfiguration API. The RegistryConfiguration API allows users to override the registry configuration. The API is similar to KKP, but allows for more flexibility. Settings specified in the RegistryConfiguration can be applied to all components deployed by Kubermatic KubeOne and to kubeadm.
type RegistryConfiguration struct {
OverrideRegistry string `json:"overrideRegistry,omitempty"`
OverrideRegistryOrg bool `json:"overrideRegistryOrg,omitempty"`
OverrideRegistryIgnore []string `json:"overrideRegistryIgnore,omitempty"`
}
The OverrideRegistry provides the same functionality as in KKP. The OverrideRegistryOrg allows users to enable overriding the organization/user (e.g. /calico/cni could be changed to /custom-mirror/cni). The OverrideRegistryIgnore slice can be used to specify registries to be ignored (e.g. if user provider k8s.gcr.io and quay.io, those registries would be used instead of registry provided in the OverrideRegistry field).
Learn More
- Documentation for Kubermatic Kubernetes Platform
- Documentation for Kubermatic KubeOne
- About Docker Rate Limits
