Note: This probably only applies to OpenShift/OKD 4.13+

Introduction⌗

This post is about configuring your cluster so that only the mirror registries (pull through caches in this case) are used for pulling any images. By default, specified mirrors are used only when their respective sources are unavailable.

Pull through caches are helpful to reduce network usage since the image is pulled only once from the Internet. An added advantage is a significant decrease in cluster startup time which is very useful for homelabbers like me who shut down their machines often.

Creating the pull through cache registries⌗

I’m using a simple docker compose file for creating multiple pull through caches for the following registries:

registry-1.docker.io
registry.k8s.io
quay.io
gcr.io
ghcr.io

version: '3.9'
services:
  registry-quay.io:
    image: registry:2
    environment:
      - REGISTRY_PROXY_REMOTEURL=https://quay.io
      - REGISTRY_HTTP_ADDR=0.0.0.0:443
      - REGISTRY_HTTP_TLS_CERTIFICATE=/certs/tls.crt
      - REGISTRY_HTTP_TLS_KEY=/certs/tls.key
    restart: always
    sysctls:
      - net.ipv4.ip_unprivileged_port_start=0
    container_name: registry-quay.io
    ports:
      - "443:5000"
    networks:
      gapped_network:
        ipv4_address: 10.0.0.4
    volumes:
      - ./certs:/certs
  registry-docker.io:
    image: registry:2
    environment:
      - REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io
      - REGISTRY_HTTP_ADDR=0.0.0.0:443
      - REGISTRY_HTTP_TLS_CERTIFICATE=/certs/tls.crt
      - REGISTRY_HTTP_TLS_KEY=/certs/tls.key
    restart: always
    sysctls:
      - net.ipv4.ip_unprivileged_port_start=0
    container_name: registry-docker.io
    ports:
      - "443:5000"
    networks:
      gapped_network:
        ipv4_address: 10.0.0.5
    volumes:
      - ./certs:/certs
  registry-registry.k8s.io:
    image: registry:2
    environment:
      - REGISTRY_PROXY_REMOTEURL=https://registry.k8s.io
      - REGISTRY_HTTP_ADDR=0.0.0.0:443
      - REGISTRY_HTTP_TLS_CERTIFICATE=/certs/tls.crt
      - REGISTRY_HTTP_TLS_KEY=/certs/tls.key
    restart: always
    sysctls:
      - net.ipv4.ip_unprivileged_port_start=0
    container_name: registry-registry.k8s.io
    ports:
      - "443:5000"
    networks:
      gapped_network:
        ipv4_address: 10.0.0.6
    volumes:
      - ./certs:/certs
  registry-gcr.io:
    image: registry:2
    environment:
      - REGISTRY_PROXY_REMOTEURL=https://gcr.io
      - REGISTRY_HTTP_ADDR=0.0.0.0:443
      - REGISTRY_HTTP_TLS_CERTIFICATE=/certs/tls.crt
      - REGISTRY_HTTP_TLS_KEY=/certs/tls.key
    restart: always
    sysctls:
      - net.ipv4.ip_unprivileged_port_start=0
    container_name: registry-gcr.io
    ports:
      - "443:5000"
    networks:
      gapped_network:
        ipv4_address: 10.0.0.7
    volumes:
      - ./certs:/certs
  registry-ghcr.io:
    image: registry:2
    environment:
      - REGISTRY_PROXY_REMOTEURL=https://ghcr.io
      - REGISTRY_HTTP_ADDR=0.0.0.0:443
      - REGISTRY_HTTP_TLS_CERTIFICATE=/certs/tls.crt
      - REGISTRY_HTTP_TLS_KEY=/certs/tls.key
    restart: always
    sysctls:
      - net.ipv4.ip_unprivileged_port_start=0
    container_name: registry-ghcr.io
    ports:
      - "443:5000"
    networks:
      gapped_network:
        ipv4_address: 10.0.0.8
    volumes:
      - ./certs:/certs
networks:
  gapped_network:
    name: gapped_network
    driver: macvlan
    driver_opts:
      parent: eth1
    ipam:
      config:
        - subnet: "10.0.0.0/24"
          gateway: "10.0.0.1"

The certs directory holds the certificates for enabling SSL/TLS.

The macvlan driver is used so that I can use IP addresses from my virtual network for the docker containers created. Ofcourse I have ensured that these IP addresses are not assigned by the DHCP server that I use since that would cause conflicts.

install-config.yaml⌗

The only relevant configuration parameters here are imageDigestSources (for specifying the mirrors) and additionalTrustBundle (for providing the CA certificate that we created for the pull through caches).

imageDigestSources:
- mirrors:
  - quay.internal.nanibot.net
  source: quay.io
- mirrors:
  - registry-1.internal.nanibot.net
  source: registry-1.docker.io
- mirrors:
  - registry-k8s.internal.nanibot.net
  source: registry.k8s.io
- mirrors:
  - gcr.internal.nanibot.net
  source: gcr.io
- mirrors:
  - ghcr.internal.nanibot.net
  source: ghcr.io
additionalTrustBundle: |
  -----BEGIN CERTIFICATE-----
  <----CERT-DATA-HERE------->
  -----END CERTIFICATE-----

ImageDigestMirrorSet and ImageTagMirrorSet⌗

Even though we’ve provided the mirrors in install-config.yaml, they won’t be used if the source (for example, quay.io) is reachable.

To ensure that our cluster pulls images only from our mirrors and not from the original sources, we have to modify the existing ImageDigestMirrorSet to include the field mirrorSourcePolicy. We also add an ImageTagMirrorSet so that the mirrors are used when images are pulled using tags.

File: image-digest-mirror-set.yaml

apiVersion: config.openshift.io/v1
kind: ImageDigestMirrorSet
metadata:
  name: image-digest-mirror
spec:
  imageDigestMirrors:
  - mirrorSourcePolicy: NeverContactSource
    mirrors:
    - quay.internal.nanibot.net
    source: quay.io
  - mirrorSourcePolicy: NeverContactSource
    mirrors:
    - registry-1.internal.nanibot.net
    source: registry-1.docker.io
  - mirrorSourcePolicy: NeverContactSource
    mirrors:
    - registry-k8s.internal.nanibot.net
    source: registry.k8s.io
  - mirrorSourcePolicy: NeverContactSource
    mirrors:
    - gcr.internal.nanibot.net
    source: gcr.io
  - mirrorSourcePolicy: NeverContactSource
    mirrors:
    - ghcr.internal.nanibot.net
    source: ghcr.io

File: image-tag-mirror-set.yaml

apiVersion: config.openshift.io/v1
kind: ImageTagMirrorSet
metadata:
  name: image-tag-mirror
spec:
  imageTagMirrors:
  - mirrorSourcePolicy: NeverContactSource
    mirrors:
    - quay.internal.nanibot.net
    source: quay.io
  - mirrorSourcePolicy: NeverContactSource
    mirrors:
    - registry-1.internal.nanibot.net
    source: registry-1.docker.io
  - mirrorSourcePolicy: NeverContactSource
    mirrors:
    - registry-k8s.internal.nanibot.net
    source: registry.k8s.io
  - mirrorSourcePolicy: NeverContactSource
    mirrors:
    - gcr.internal.nanibot.net
    source: gcr.io
  - mirrorSourcePolicy: NeverContactSource
    mirrors:
    - ghcr.internal.nanibot.net
    source: ghcr.io

Verifying the changes⌗

An entry before applying the above changes

File: /etc/containers/registries.conf

...
[[registry]]
  prefix = ""
  location = "gcr.io"

  [[registry.mirror]]
    location = "gcr.internal.nanibot.net"
    pull-from-mirror = "digest-only"
...

The same entry after applying the changes

...
[[registry]]
  prefix = ""
  location = "gcr.io"
  blocked = true

  [[registry.mirror]]
    location = "gcr.internal.nanibot.net"
    pull-from-mirror = "digest-only"

  [[registry.mirror]]
    location = "gcr.internal.nanibot.net"
    pull-from-mirror = "tag-only"
...

Reducing OpenShift/OKD cluster startup time using pull through caches

Introduction⌗

Creating the pull through cache registries⌗

install-config.yaml⌗

ImageDigestMirrorSet and ImageTagMirrorSet⌗

Verifying the changes⌗