zot Registry

Running zot, the OCI-native registry: storage layouts, sync and mirroring, authentication, extensions, and air-gapped distribution.

Overview

zot is a CNCF sandbox OCI image registry that ships as a single static Go binary with no external dependencies — no database, no object store, no message broker, nothing to bootstrap. Point it at a directory and it serves an OCI distribution-spec API over an OCI image-layout on disk. That is the whole design: the storage format is the spec's storage format, so the bytes under rootDirectory are a valid OCI layout that skopeo copy oci:…, oras or a tarball can consume directly.

The niche is the middle ground. registry:2 is a bare blob store: no search, no UI, no scanning, no mirroring, garbage collection only with the registry stopped. Harbor is a Kubernetes-scale product with Postgres, Redis, a job service and half a dozen containers. zot gives you the useful 80% of Harbor — a UI, GraphQL search, Trivy-backed CVE reporting, signature verification, sync, per-repo RBAC, online GC with retention policies — in one process you can also run on a Raspberry Pi at the edge.

What actually differentiates it:

• OCI-native storage. No Docker schema2 translation layer, no proprietary layout. The on-disk tree is an OCI image layout per repository.
• Sync. Pull-through mirroring and scheduled replication are first-class, with content filters, semver tag selection and signature copying. This is the feature people adopt zot for.
• Online garbage collection with declarative retention policies, rather than the stop-the-world offline GC of registry:2.
• Config-file driven. There is essentially no flag surface: zot serve config.json and that is it. Everything is JSON (or YAML) and validated up front.
• Composable builds. A zot-minimal build strips every extension into an 82 MB binary; the full build is 225 MB because it statically links Trivy, the UI bundle and a GraphQL server.

This sheet is zot-specific. For skopeo usage, Docker Hub, GHCR, ECR, Artifact Registry, ACR, Harbor and Quay, plus the generic credential-helper, mirroring and air-gap patterns, see the Container Registries cheatsheet. For Trivy itself — DB management, scanner behaviour, ignore policies, SBOM formats — see the Trivy cheatsheet.

⤢ zoom

Versions and builds

Everything below is verified against zot v2.1.20 (released 2026-08-04, go1.26.5, distribution-spec 1.1.1) unless flagged otherwise.

Each release publishes these binaries for Linux, macOS, FreeBSD and Windows on amd64 and arm64, plus a zot-schema.json:

Binary	Size (linux/amd64)	What it is
zot-linux-amd64	225 MB	Full build: every extension compiled in
zot-linux-amd64-minimal	82 MB	Registry core only, no extensions
zot-linux-amd64-debug	larger	Full build without symbol stripping
zli-linux-amd64	14 MB	Client CLI for querying a zot instance
zb-linux-amd64	9 MB	Benchmarking / load-generation tool
zxp-linux-amd64	small	Prometheus exporter sidecar for minimal builds

The binary reports which extensions it carries. This is the fastest way to tell what you are actually running:

zot --version
# {"level":"info","message":"version","distribution-spec":"1.1.1",
#  "commit":"v2.1.20-0-g3b5796d",
#  "binary-type":"-events-imagetrust-lint-metrics-mgmt-profile-scrub-search-sync-ui-userprefs",
#  "go version":"go1.26.5"}

zot-minimal --version
# ... "binary-type":"minimal" ...

The leading - in the full build's binary-type is a separator artefact, not a negation — those eleven names are the extensions that are present.

Install and Run

# Binary. Pick the release and platform explicitly; there is no "latest" redirect for assets.
VER=v2.1.20
sudo curl -fsSL -o /usr/bin/zot \
  https://github.com/project-zot/zot/releases/download/$VER/zot-linux-amd64
sudo chmod +x /usr/bin/zot && sudo chown root:root /usr/bin/zot

# zli and zb are client-side tools — no need to install them on the server
curl -fsSL -o ~/.local/bin/zli \
  https://github.com/project-zot/zot/releases/download/$VER/zli-linux-amd64
chmod +x ~/.local/bin/zli

There is no upstream apt or dnf repository. Package the binary yourself or use the container image.

# Container. Images are on GHCR, one per platform — the arch is in the image NAME,
# these are not multi-arch manifest lists.
podman run -d --name zot -p 5000:5000 \
  -v ./config.json:/etc/zot/config.json:ro,Z \
  -v zot-data:/var/lib/zot:Z \
  ghcr.io/project-zot/zot-linux-amd64:v2.1.20
#   ghcr.io/project-zot/zot-minimal-linux-amd64:v2.1.20   — minimal variant
#   ghcr.io/project-zot/zb-linux-amd64:v2.1.20            — benchmark tool

podman inspect ghcr.io/project-zot/zot-linux-amd64:v2.1.20 --format \
  'Entrypoint={{.Config.Entrypoint}} Cmd={{.Config.Cmd}} User={{.Config.User}} Exposed={{.Config.ExposedPorts}} Vols={{.Config.Volumes}}'
# Entrypoint=[/usr/local/bin/zot-linux-amd64] Cmd=[serve /etc/zot/config.json]
# User=0 Exposed=map[] Vols=map[]

Three things about the image that matter and are not obvious. It runs as root (User=0) — add --user and make rootDirectory writable by that UID, or run it rootless under Podman. It declares no EXPOSE, so the published port must match whatever http.port says and nothing warns you if they disagree. It declares no VOLUME, so forgetting the mount leaves you with a container-lifetime scratch store. And http.address must be 0.0.0.0 in a container, not the 127.0.0.1 every upstream example uses, or the port publish silently reaches nothing.

Minimal versus full

The extensions are the entire size difference, and the cost of the full build is almost all Trivy's static vulnerability-scanning machinery plus the embedded UI. The runtime cost of enabling them is a different question: search builds and maintains a metadata index, and cve downloads and holds a multi-gigabyte Trivy database (see CVE scanning).

The footgun: zot-minimal accepts a config containing an extensions block without complaint. zot verify reports it valid and never mentions the block, so a CI gate that only runs verify passes. serve then starts and the extension routes return 404 forever. It does say so, but only at startup and only in the log: one warn line per skipped extension, fifteen of them on a config enabling search, ui and metrics, buried in the normal boot chatter.

# Same config, minimal binary
zot-minimal verify config.json
# {"level":"info","message":"config file is valid"}    <-- no mention of extensions

zot-minimal serve config.json &
# {"level":"warn","message":"skipping enabling search extension because given zot binary
#  doesn't include this feature,please build a binary that does so"}
# ... and one of these per extension: metrics, scrub, sync, lint, userprefs, ui, mgmt, trust
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v2/                     # 200
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/v2/_zot/ext/search      # 404
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/                        # 404  (no UI)

Check binary-type before you debug an extension that "does not work".

Configuration

zot is configured entirely by file. zot serve <config> is the only meaningful invocation; the sole runtime flag is --force-reparse, which discards the fast-restart stamp and rebuilds metaDB from storage.

zot serve /etc/zot/config.json      # run
zot verify /etc/zot/config.json     # validate without starting
zot schema > zot-schema.json        # dump the JSON Schema for the whole config
zot scrub /etc/zot/config.json      # offline integrity check (server must be stopped)
zot verify-feature retention /etc/zot/config.json    # run GC + retention once, offline

zot schema is the authoritative reference — it is generated from the Go structs, so it is never stale. Feed it to an editor's JSON language server and you get completion and validation on config.json for free.

Structure

Four top-level sections carry almost everything, plus scheduler and cluster:

{
  // Advertised distribution-spec version. Use the current one; not a compatibility switch.
  "distSpecVersion": "1.1.1",
  "storage": {
    "rootDirectory": "/var/lib/zot",   // required; everything else has a default
    "dedupe": true,                    // default true
    "gc": true,                        // default true
    "gcDelay": "1h",                   // default 1h — grace period before a blob is collectable
    "gcInterval": "1h",                // default 1h — how often the periodic GC task runs
    "commit": false,                   // default false; fsync every write. Costly, rarely needed
    "maxRepos": 0,                     // 0 = unlimited
    "subPaths": {}
  },
  "http": {
    "address": "127.0.0.1",            // no default that suits a container; set it deliberately
    "port": "8080",                    // a string, not a number
    "realm": "zot",                    // no default; shows in the WWW-Authenticate challenge
    "externalUrl": "https://reg.example.com",   // required for OpenID callbacks and the UI
    "readTimeout": "60s", "writeTimeout": "60s",             // both default 60s
    "compat": [],                      // ["docker2s2"] to accept Docker schema2 — see below
    "tls": {}, "auth": {}, "accessControl": {}, "ratelimit": {}
  },
  "log": {
    "level": "info",                   // debug | info | warn | error
    "output": "/var/log/zot/zot.log",  // unset = stdout
    "audit": "/var/log/zot/audit.log"  // separate authn/authz audit trail; unset = none
  },
  "extensions": {},
  "scheduler": { "numWorkers": 3 }     // default: 4 x NumCPU
}

YAML is accepted too (config.yaml), with identical keys. JSON is what every upstream example uses.

Keys are matched case-insensitively — rootDirectory, rootdirectory and ROOTDIRECTORY all bind to the same field — which is why the schema lists each name twice. Do not rely on this; write the documented camelCase.

Validation

zot verify is strict about shape and loose about substance.

# Unknown keys are rejected outright — typos cannot slip through
zot verify bad.json
# Error: decoding failed due to the following error(s):
#
# 'Storage' has invalid keys: dedup

# Semantic cross-checks run too
zot verify authz-without-authn.json
# Error: invalid server config: access control config requires one of htpasswd, ldap,
# openid or mTLS authentication or using only 'anonymousPolicy' policies

File references are checked inconsistently. The secret files verify needs to parse — the LDAP and OpenID credentialsFile, and sessionKeysFile — are opened and a missing one is a hard error:

zot verify ldap.json
# Error: invalid server config
# open /etc/zot/ldap-credentials.json: no such file or directory

But the files it only hands to something else are never stat'ed. A config naming a nonexistent htpasswd file or TLS certificate passes verify and fails at serve. Validate in CI, then smoke-test the real start.

Hot reload — and the two traps

zot watches its config file with fsnotify and reloads on change: it stops background tasks, swaps the config, and restarts them. Auth, authorisation, sync registries, extension settings and log level all take effect without dropping connections.

# Edit the file in place and the running server picks it up
vi /etc/zot/config.json
# stdout: {"level":"info","message":"config file changed, trying to reload config"}

Two failure modes, both silent, both verified against v2.1.20:

SIGHUP shuts zot down. It is registered alongside SIGTERM and SIGINT as a shutdown signal, not a reload signal. systemctl reload zot — if you have written an ExecReload=/bin/kill -HUP $MAINPID — stops your registry.

kill -HUP "$(pidof zot)"
# log: {"level":"info","message":"received signal","signal":1}
# process exits

Only an in-place write triggers a reload. The watcher tests for fsnotify.Write exactly. Writing a temp file and renaming it over the config — the atomic-update idiom, and precisely what a Kubernetes ConfigMap projection does — produces CREATE/RENAME events and is ignored:

# Reloads
python3 -c 'f=open("/etc/zot/config.json","r+"); f.seek(0); f.write(new); f.truncate()'

# Does NOT reload — no log line, old config stays live
cp new.json /etc/zot/config.json.tmp && mv /etc/zot/config.json.tmp /etc/zot/config.json

On Kubernetes, restart the pod on a ConfigMap change (a checksum annotation on the pod template) rather than expecting the projected file to hot-reload.

Also note the reload message goes to stdout, not to log.output. If you redirect stdout to /dev/null under systemd you will never see whether a reload happened.

Storage

On-disk layout

Each repository is a self-contained OCI image layout directory under rootDirectory:

/var/lib/zot/
├── meta.db                      # metaDB: tags, timestamps, CVE results, stars, API keys (boltdb)
├── cache.db                     # dedupe cache: digest -> canonical path (boltdb)
├── _trivy/                      # CVE extension only: vulnerability + fanal databases
└── infra/
    └── alpine/
        ├── oci-layout           # {"imageLayoutVersion":"1.0.0"}
        ├── index.json           # tags live here, as org.opencontainers.image.ref.name annotations
        ├── .uploads/            # in-flight chunked uploads
        └── blobs/
            └── sha256/
                ├── 871ac0ef…   # manifest
                ├── b66e0ce6…   # config
                └── 22df59ff…   # layer

# The repository directory is a valid OCI layout — consume it directly, no registry needed
skopeo copy oci:/var/lib/zot/infra/alpine:3.22 docker://other.example.com/infra/alpine:3.22

Blobs are created 0600, directories 0700. The registry expects to own the tree; do not share it with another process.

Tags are annotations inside index.json, not directory entries, so nothing in the filesystem layout tells you what a tag points at without reading that file. Separately, meta.db is a projection of the tree, not the tree itself, and the two can drift. By default they do not: zot walks storage on every start, so a repository directory dropped in out of band is in metaDB — and in search, the UI and GC — the moment you restart. It is fastRestart: true that opens the gap. That skips the walk when a stamp matches:

{"level":"info","message":"metaDB fast-restart stamp matches, skipping full storage parse","component":"metadb"}

and after that an out-of-band repository is served by /v2/_catalog and /v2/<repo>/tags/list, which read storage, but is invisible to GraphQL, the UI and anything else backed by metaDB. zot serve --force-reparse ignores the stamp and rebuilds; that is the repair, and the only reason the flag exists.

Deduplication

dedupe (default true) hard-links identical blobs across repositories within the same store. The dedupe cache in cache.db maps each digest to its canonical path.

# Same layer pushed to two repos
ls -li /var/lib/zot/infra/alpine/blobs/sha256/22df59ff… \
       /var/lib/zot/team-b/alpine/blobs/sha256/22df59ff…
# 3740 -rw------- 2 zot zot 3904689 … infra/alpine/blobs/sha256/22df59ff…
# 3740 -rw------- 2 zot zot 3904689 … team-b/alpine/blobs/sha256/22df59ff…
#  ^inode          ^link count 2 — one copy on disk

Consequences worth knowing:

• Dedupe is per store. Blobs are not shared across subPaths, because each subpath is a separate store with its own cache.
• It needs a filesystem that supports hard links across the tree. It does nothing on object storage; with storageDriver set, dedupe is implemented in the cache layer instead and remoteCache governs whether that cache is shared.
• Turning dedupe off on an existing store triggers a restore pass on next start, unlinking every deduped blob back into independent copies. That is an expensive, disk-hungry, one-way-ish operation. zot records completion in a _restore_complete marker so it is not repeated.
• hydrateBlobOnRead, which restores the pre-conformance behaviour where a HEAD or ranged GET could hard-link a blob from the dedupe cache into the requesting repository, is not in v2.1.20 — it is on main and unreleased. Putting it in a config today fails validation outright: 'Storage' has invalid keys: hydrateblobonread.

Garbage collection and retention

zot's GC is online — it runs against a live registry, unlike registry:2 which requires a read-only or stopped instance.

"storage": {
  "rootDirectory": "/var/lib/zot", "gc": true,
  "gcDelay": "2h",                  // blobs younger than this are never collected
  "gcInterval": "1h",               // periodic sweep cadence
  "gcTimeWindow": "01:00-08:00"     // optional: only sweep inside this daily window
}

gcDelay exists to protect in-flight pushes: a blob uploaded but not yet referenced by a manifest looks like garbage. Setting it too low will corrupt concurrent pushes; the default 1h is a reasonable floor. gcTimeWindow is empty by default, meaning GC may run at any time — set it if the sweep competes with your build peak.

GC removes unreferenced blobs. Deciding which manifests become unreferenced is the retention policy's job:

"storage": {
  "rootDirectory": "/var/lib/zot",
  "gc": true,
  "retention": {
    "dryRun": false,          // true = log what would be deleted, delete nothing
    "delay": "24h",           // grace period for untagged manifests; default 1h
    "policies": [
      {
        // First matching policy wins — order matters, most specific first
        "repositories": ["infra/*", "prod/*"],
        "deleteReferrers": false, "deleteUntagged": false,
        "keepTags": [
          { "patterns": ["^v\\d+\\.\\d+\\.\\d+$"] },                 // all release tags
          { "patterns": [".*-rc\\d+$"], "pulledWithin": "168h" }     // RCs pulled this week
        ]
      },
      {
        // Catch-all. Without one, anything not matched by an earlier policy is kept forever.
        "repositories": ["**"],
        "deleteReferrers": true, "deleteUntagged": true,
        "keepTags": [
          { "mostRecentlyPushedCount": 10, "mostRecentlyPulledCount": 10, "pulledWithin": "720h" }
        ],
        "keepUntagged": { "mostRecentlyPushedCount": 5 }
      }
    ]
  }
}

Matching is precise and worth stating exactly:

• Repositories: first match wins. Policies are scanned in order and the first pattern that matches the repository supplies the whole policy; later policies are not consulted. Order most specific first, and put the ** catch-all last. Patterns are doublestar globs — * matches one path segment, ** any depth.
• keepTags entries are ORed, and the first matching entry supplies the rule set. Within one entry the conditions are ANDed. patterns are Go regular expressions matched against the tag.

The defaults when no policy matches a repository — including when retention is absent entirely — are not "do nothing", which is the trap:

Behaviour	Default with no matching policy
deleteUntagged	true — untagged manifests are expired whenever gc is on
deleteReferrers	false — referrer artefacts are kept
Tag retention	none — tagged images are never expired

So the out-of-the-box posture is: untagged manifests are cleaned up, tagged ones accumulate forever. Adding a keepTags policy is what starts expiring tags; adding "deleteUntagged": false is what stops the untagged sweep. A policy set with no ** catch-all leaves unmatched repositories on these defaults, not on your policy — which is either what you wanted or a silent surprise, and nothing tells you which.

Rehearse before you commit:

# dryRun: true in the config, then run the whole GC + retention pass once, offline
systemctl stop zot
zot verify-feature retention /etc/zot/config.json -t 120s
# {"level":"info","message":"applied policy","module":"retention","dry-run":true,
#  "repository":"infra/debian","mediaType":"application/vnd.oci.image.manifest.v1+json",
#  "digest":"sha256:bbee9b20…"}
systemctl start zot

verify-feature retention needs metaDB access, so the server must be stopped. -i overrides gcInterval and -t bounds how long it waits for tasks.

Subpaths — multi-tenant layouts

subPaths maps a URL prefix onto a different store, with its own root directory, storage driver, dedupe setting and retention policy:

"storage": {
  "rootDirectory": "/var/lib/zot",
  "dedupe": true,
  "subPaths": {
    "/infra":   { "rootDirectory": "/mnt/fast-nvme/zot-infra",
                  "dedupe": true, "gc": true, "gcDelay": "2h", "gcInterval": "1h" },
    "/archive": { "rootDirectory": "/mnt/bulk/zot-archive",
                  "dedupe": true, "remoteCache": true,
                  "storageDriver": { "name": "s3", "rootdirectory": "/zot-archive",
                                     "region": "eu-west-2", "bucket": "zot-archive",
                                     "forcepathstyle": false, "secure": true } }
  }
}

A push to infra/alpine lands in /mnt/fast-nvme/zot-infra; everything else lands in the default store. This is how you give tenants separate volumes, quotas and backup schedules without running separate registries. fastRestart is a top-level storage setting only and is not honoured under subPaths.

Remote storage backends

S3, GCS and Azure Blob via the distribution storage drivers, plus a separate cache driver for dedupe and metadata:

"storage": {
  "rootDirectory": "/var/lib/zot",   // still required — used for local temp and metaDB
  "dedupe": true,
  "remoteCache": true,
  "storageDriver": { "name": "s3", "rootdirectory": "/zot", "region": "eu-west-2",
                     "bucket": "zot-storage", "forcepathstyle": false,
                     "secure": true, "skipverify": false },
  "cacheDriver":   { "name": "redis", "url": "redis://cache.internal:6379",
                     "keyprefix": "zot" }
}

// DynamoDB cache — the S3-native pairing, and what scale-out clustering expects.
// Every table name is configurable; these are the conventional values.
"cacheDriver": { "name": "dynamodb", "endpoint": "https://dynamodb.eu-west-2.amazonaws.com",
                 "region": "eu-west-2", "cacheTablename": "ZotBlobTable",
                 "repoMetaTablename": "ZotRepoMetadataTable",
                 "imageMetaTablename": "ZotImageMetaTable",
                 "repoBlobsInfoTablename": "ZotRepoBlobsInfoTable",
                 "userDataTablename": "ZotUserDataTable",
                 "versionTablename": "ZotVersion", "apiKeyTablename": "ZotApiKeyTable" }

// GCS / Azure Blob
"storageDriver": { "name": "gcs", "rootdirectory": "/zot", "bucket": "zot-storage",
                   "credentialsFile": "/etc/zot/gcs-credentials.json" }
"storageDriver": { "name": "azure", "rootdirectory": "/zot",
                   "accountname": "myazurestorageaccount", "container": "zot-storage",
                   "credentials": { "type": "default_credentials" } }

Notes that bite:

• Storage-driver keys are lowercase, no camelCase (rootdirectory, forcepathstyle, regionendpoint) because they pass straight through to the distribution driver. The zot-level keys around them are camelCase. Mixing the two conventions is the most common S3 config error.
• rootDirectory is still mandatory with a remote driver — metaDB and temporary state stay local.
• redirectBlobURL (default false) makes zot issue a 307 to a presigned object-store URL instead of proxying the blob. It moves the bytes off your registry's egress path, at the cost of exposing bucket URLs to clients.
• Set remoteCache: true and a shared cacheDriver when more than one zot fronts the same bucket, or the two instances will disagree about which blobs exist.

Authentication and Authorisation

These are two separate blocks and they compose in a specific order: http.auth establishes who you are; http.accessControl decides what you may do. Configuring the second without the first is rejected at startup unless every policy is an anonymousPolicy.

⤢ zoom

htpasswd

The simplest option, and the one every other mechanism composes with.

# -B is mandatory: zot supports bcrypt hashes ONLY. MD5 (-m, the apache default on some
# platforms), SHA1 (-s) and plaintext (-p) entries are parsed but never match, so those
# users simply cannot log in and nothing says why.
# -n prints to stdout rather than writing the file, so redirect explicitly.
htpasswd -bnB alice 's3cret'  > /etc/zot/htpasswd
htpasswd -bnB bob   'hunter2' >> /etc/zot/htpasswd
chown root:zot /etc/zot/htpasswd && chmod 640 /etc/zot/htpasswd

The htpasswd file has its own watcher, separate from the config reloader — debounced, with a stat-polling fallback, and deliberately tolerant of the Remove/Create/Rename burst a Kubernetes Secret update produces. So adding a user takes effect without a restart, and unlike the main config it does survive an atomic rename.

// failDelay: seconds to stall a failed auth; default 0. Set it — free anti-brute-force.
"http": { "auth": { "htpasswd": { "path": "/etc/zot/htpasswd" }, "failDelay": 5 } }

LDAP, SSO, bearer and mTLS

"http": {
  "externalUrl": "https://reg.example.com",   // REQUIRED for OpenID callbacks and UI links
  "tls": { "cert": "/etc/zot/server.crt", "key": "/etc/zot/server.key",
           "cacert": "/etc/zot/ca.crt" },     // cacert present == client-cert verification on
  "auth": {
    "failDelay": 5,
    "apikey": true,                           // users can mint zak_… tokens for CLI use
    "sessionKeysFile": "/etc/zot/sessionKeys.json",

    "ldap": {
      "address": "ldap.example.org", "port": 636,
      "startTLS": false,                      // true for STARTTLS on 389; false for LDAPS on 636
      "baseDN": "ou=Users,dc=example,dc=org",
      "userAttribute": "uid",
      "userGroupAttribute": "memberOf",       // populates groups for accessControl
      "userFilter": "(objectClass=person)", "subtreeSearch": true,
      "caCert": "/etc/zot/ldap-ca.pem",
      "skipVerify": false, "insecure": false, // never true in production
      "credentialsFile": "/etc/zot/ldap-credentials.json"   // {"bindDN":…,"bindPassword":…}
    },
    "openid": {
      "callbackAllowOrigins": ["https://ui.example.com"],
      "providers": { "oidc": { "name": "Corporate SSO",
                               "issuer": "https://sso.example.com/realms/main",
                               "credentialsFile": "/etc/zot/oidc-credentials.json",
                               "scopes": ["openid", "email", "groups"] } }
    },
    "bearer": { "realm": "https://auth.example.com/auth/token",
                "service": "zot.example.com",
                "cert": "/etc/zot/auth-pubkey.crt" },   // public key verifying the JWTs
    "mtls": { "identityAttributes": ["CommonName"] }
              // or "uriSanPattern": "^spiffe://example\\.org/ns/prod/sa/(.+)$"
  }
}

• LDAP credentials live in a separate credentialsFile so it can carry tighter permissions; it is watched alongside the main config, so a rotated bind password reloads without a restart. userGroupAttribute is what makes directory groups usable in accessControl.
• OpenID is browser-interactive login for the UI and zli. It does not replace basic auth for docker login — keep htpasswd or an API key for machine clients. Provider keys are google, gitlab and the generic oidc; github is handled as plain OAuth2. Endpoints live under /zot/auth/ (login, logout, apikey, callback).
• sessionKeysFile matters as soon as there is more than one replica: without a shared, persistent key pair every restart invalidates every session, and two instances behind a load balancer will not accept each other's cookies.
• mTLS maps a certificate field to a zot identity that accessControl then references. identityAttributes defaults to ["CommonName"]; for SPIFFE workloads match the URI SAN instead — uriSanPattern's first capture group becomes the identity. Note there is no separate mTLS switch: adding cacert to the TLS block is what turns on client-certificate verification.

Access control

"http": {
  "auth": { "htpasswd": { "path": "/etc/zot/htpasswd" }, "failDelay": 5 },
  "accessControl": {
    "groups": {
      "platform": { "users": ["alice", "carol"] }
      // LDAP/OIDC groups arrive from the directory and need no declaration here
    },
    "repositories": {
      "**":        { "defaultPolicy": ["read"],       // any authenticated user
                     "anonymousPolicy": [] },         // unauthenticated: nothing
      "infra/**":  { "policies": [
                       { "groups": ["platform"], "actions": ["read","create","update","delete"] },
                       { "users": ["ci-bot"],    "actions": ["read","create"] } ],
                     "defaultPolicy": ["read"] },
      "public/**": { "anonymousPolicy": ["read"] }    // unauthenticated pulls allowed here only
    },
    "adminPolicy": { "users": ["alice"], "actions": ["read","create","update","delete"] }
  }
}

The four actions are read, create, update and delete. create is a push of something new; update is overwriting an existing tag — separating them is how you make tags effectively immutable for CI while still allowing new ones.

Verified behaviour that is easy to get wrong:

# /v2/_catalog is FILTERED per identity, never 401 — anonymous sees only what it may read
curl -s http://reg:8080/v2/_catalog
# {"repositories":["public/alpine"]}
curl -s -u alice:s3cret http://reg:8080/v2/_catalog
# {"repositories":["infra/alpine","public/alpine"]}

# An anonymous request for a protected manifest is 401, not 403
curl -s -o /dev/null -w '%{http_code}\n' http://reg:8080/v2/infra/alpine/manifests/3.22   # 401

# bob has defaultPolicy read but no create — the push fails at the first blob upload
podman push --creds bob:hunter2 reg:8080/infra/alpine2:3.22
# Error: writing blob: initiating layer upload to /v2/infra/alpine2/blobs/uploads/ …:
#        requested access to the resource is denied

Exactly one repository pattern applies: zot picks the longest matching pattern by string length and evaluates only that entry. infra/** (8 characters) beats ** for infra/alpine, and nothing from the ** entry is inherited — so a pattern with policies but no defaultPolicy grants nothing to users outside those policies, even if the broader pattern would have. That is usually the intent, and occasionally a surprise when the CI account stops being able to pull. adminPolicy is evaluated separately as a fallback grant, so an admin is permitted regardless of which repository pattern matched.

Enable the audit log whenever accessControl is in play; it is the only record of who did what:

"log": { "level": "info", "output": "/var/log/zot/zot.log", "audit": "/var/log/zot/audit.log" }

{"level":"info","message":"HTTP API Audit","subject":"alice","action":"PUT",
 "object":"/v2/infra/alpine/manifests/3.22","status":201,"clientIP":"10.0.2.14:55784"}

CEL conditions and rate limiting

Policies may carry conditions — CEL expressions evaluated per request — for time-boxing access or enforcing digest-pinned pushes. They compile at config validation, so a broken expression is caught by zot verify rather than at request time.

"prod/**": {
  "policies": [ {
    "users": ["alice"], "actions": ["read", "create", "update"],
    "conditions": [
      { "expression": "req.time < timestamp(\"2027-03-31T23:59:59Z\")",
        "message": "alice's prod access expires end of Q1 2027" },
      { "expression": "req.referenceType == \"digest\"",
        "message": "prod pushes must use digest references, not mutable tags" }
    ]
  } ]
}

// Unrelated but adjacent: a global and per-method request-rate cap
"http": { "ratelimit": { "rate": 100, "methods": [ { "method": "PATCH", "rate": 10 } ] } }

Sync and Mirroring

zot's headline feature. Two modes, and they combine on the same registry entry.

⤢ zoom

On-demand (onDemand: true) is pull-through caching. A request for an image zot does not hold triggers a synchronous fetch from upstream, which is then stored locally and served from cache thereafter.

Scheduled (pollInterval: "6h") is push-free replication. zot walks the upstream catalogue on a timer and copies whatever matches the content filters, whether or not anyone has asked for it. This is what you use to warm an air-gapped or edge site.

// preserveDigest below needs http.compat set, so both keys are shown. The default
// "compat": [] makes zot refuse to start: "can not use PreserveDigest option
// without enabling http.Compat".
"http": { "address": "0.0.0.0", "port": "8080", "compat": ["docker2s2"] },

"extensions": {
  "sync": {
    "enable": true,
    "credentialsFile": "/etc/zot/sync-credentials.json",
    "downloadDir": "/var/lib/zot-sync-tmp",     // staging dir; needed for cloud storage backends
    "registries": [
      {
        // Pull-through cache for Docker Hub
        "urls": ["https://registry-1.docker.io"],
        "onDemand": true, "tlsVerify": true,
        "maxRetries": 5, "retryDelay": "1s", "maxRetryDelay": "30s", "syncTimeout": "10m",
        // There is no concurrency or rate-limit knob here — regclient's transport settings
        // are not exposed, and inventing a key makes the whole config fail to load.
        "content": [ { "prefix": "library/**" } ]
      },
      {
        // Scheduled mirror of a curated set
        "urls": ["https://ghcr.io"],
        "onDemand": false, "pollInterval": "6h", "tlsVerify": true,
        "onlySigned": true,           // refuse to mirror unsigned images
        "preserveDigest": true,       // copy without re-encoding; digests match upstream
                                      // exactly. Requires the http.compat above.
        "certDir": "/etc/zot/upstream-certs",
        "maxRetries": 3, "retryDelay": "5m",
        "content": [
          { "prefix": "myorg/platform/**",
            "tags": { "semver": true, "excludeRegex": ".*-(rc|alpha|beta)\\d*$" } },
          { "prefix": "myorg/vendor/*",
            "destination": "/vendor", "stripPrefix": true,   // myorg/vendor/foo -> vendor/foo
            "tags": { "regex": "^v4\\." } }
        ]
      }
    ]
  }
}

// /etc/zot/sync-credentials.json — keyed by upstream host[:port]
{
  "registry-1.docker.io": { "username": "syncbot", "password": "…" },
  "ghcr.io":              { "username": "syncbot", "password": "ghp_…" }
}

Content filters

content is what turns "mirror the internet" into something bounded. Each entry is matched against the upstream repository name:

Key	Effect
prefix	Glob over repository names. * is one segment, ** is any depth. Required.
destination	Local prefix to write under, instead of the upstream name
stripPrefix	Remove the matched prefix before applying destination
tags.regex	Go regex; only matching tags are copied
tags.excludeRegex	Go regex; matching tags are skipped
tags.semver	Only copy tags that parse as semantic versions

The three tag filters are a chain applied in a fixed order — regex, then excludeRegex, then semver — so excludeRegex can only ever narrow what regex already admitted.

With no content block at all, an onDemand registry will proxy anything requested — which is the point for a Docker Hub cache and a serious hole for anything else. A scheduled registry with no content mirrors the entire upstream catalogue.

Signatures

Cosign signatures and Notation/referrers artefacts are copied along with the image; onlySigned: true makes zot refuse to mirror anything that has none. syncLegacyCosignTags (default true) also copies the old sha256-<digest>.sig tag form that pre-referrers cosign used. Pair onlySigned with the trust extension and you have a workable supply-chain gate at the edge.

Verifying pull-through works

# Fresh zot with onDemand from Docker Hub
curl -s http://127.0.0.1:8080/v2/_catalog
# {"repositories":[]}

podman pull --tls-verify=false 127.0.0.1:8080/library/busybox:1.37   # ~8s: fetched upstream
podman pull --tls-verify=false 127.0.0.1:8080/library/busybox:1.37   # ~4s: served from cache

curl -s http://127.0.0.1:8080/v2/_catalog
# {"repositories":["library/busybox"]}

Air-gapped distribution

The sync path only works where there is a path. For a genuine air gap, the OCI layout on disk is the transfer medium — see the Container Registries cheatsheet for the general pattern. zot's contribution is that the source and destination are both already OCI layouts:

# High side: copy straight out of zot's storage, no registry API involved
tar -C /var/lib/zot -cf transfer.tar infra/

# Low side: unpack into the target zot's rootDirectory, then rebuild metaDB
systemctl stop zot
tar -C /var/lib/zot -xf transfer.tar
zot serve /etc/zot/config.json --force-reparse

--force-reparse is belt-and-braces: on a default config the restart's own storage walk picks the new repositories up, but if the low side runs fastRestart: true the walk is skipped and metaDB never learns about them, so search and the UI stay empty. Passing the flag costs one walk and removes the question.

Where a diode or one-way link exists, the usual shape is a DMZ zot with onDemand: true against the internet, and an inside zot with pollInterval against the DMZ, filtered hard by content.

Extensions

All extensions require the full build. Each takes "enable": true; several are also implicitly enabled by others (mgmt and userprefs routes appear whenever search or ui is on).

"extensions": {
  "search":  { "enable": true, "cve": { "updateInterval": "24h" } },
  "ui":      { "enable": true },
  "metrics": { "enable": true, "prometheus": { "path": "/metrics" } },
  "scrub":   { "enable": true, "interval": "24h" },
  "trust":   { "enable": true, "cosign": true, "notation": true },
  "lint":    { "enable": true, "mandatoryAnnotations": ["org.opencontainers.image.source"] },
  "events":  { "enable": true, "source": "reg.example.com", "typePrefix": "com.example.registry",
               "sinks": [ { "type": "nats", "address": "nats://events:4222",
                            "channel": "registry", "timeout": "10s" } ] }
}

Search (GraphQL)

Exposed at /v2/_zot/ext/search, backed by metaDB. This is what zli and the UI both talk to.

curl -s -X POST -H 'Content-Type: application/json' \
  -d '{"query":"{ ImageList(repo: \"infra/debian\") { Results { RepoName Tag Digest Size IsSigned Vulnerabilities { MaxSeverity Count } } } }"}' \
  http://127.0.0.1:8080/v2/_zot/ext/search | jq -c
# {"data":{"ImageList":{"Results":[{"RepoName":"infra/debian","Tag":"12",
#  "Digest":"sha256:bbee9b20…","Size":"51058682","IsSigned":false,
#  "Vulnerabilities":{"MaxSeverity":"CRITICAL","Count":101}}]}}}

The query surface, as of v2.1.20:

Query	Purpose
GlobalSearch(query, filter, requestedPage)	Fuzzy search across repos and images
ImageList(repo), Image(image), ExpandedRepoInfo(repo)	Repository and image detail
RepoListWithNewestImage	Catalogue ordered by most recent push
ImageListForDigest(id)	Which images contain a given blob digest
CVEListForImage(image, severity, searchedCVE, excludedCVE)	Vulnerabilities in one image
CVEDiffListForImages(minuend, subtrahend)	CVEs present in one image but not another
ImageListForCVE(id), ImageListWithCVEFixed(id, image)	Images affected by / fixed for a CVE
BaseImageList(image), DerivedImageList(image)	Layer-graph ancestry
Referrers(repo, digest, type)	Referrers for a subject
StarredRepos, BookmarkedRepos	User preferences

Request bodies are capped at 1 MiB, with a tighter internal parser token budget — this is a search API, not a general-purpose GraphQL endpoint.

UI

Serves a React console at /. It requires search — enabling ui alone gives you a page that cannot load anything. Set http.externalUrl so OIDC callbacks and copy-to-clipboard pull commands are correct.

Metrics

The full build serves Prometheus text format directly from extensions.metrics:

curl -s http://127.0.0.1:8080/metrics | grep ^zot_ | head -3
# zot_gc_duration_seconds_sum 0
# zot_http_method_latency_seconds_bucket{method="GET",le="0.05"} 10
# zot_http_method_latency_seconds_sum{method="GET"} 0

The minimal build behaves completely differently. /metrics exists but returns an internal JSON dump, not Prometheus text, so a scrape finds nothing:

curl -s http://127.0.0.1:8080/metrics | head -c 120
# {"Counters":[{"Name":"zot.scheduler.generators","Count":3,…

That JSON is for zxp, the exporter sidecar shipped in the same release, which scrapes it and re-exports real Prometheus metrics:

cat > zxp.json <<'EOF'
{ "Server":   { "protocol": "http", "host": "127.0.0.1", "port": "8080" },
  "Exporter": { "port": "8081", "log": { "level": "info" } } }
EOF
zxp config zxp.json &
curl -s http://127.0.0.1:8081/metrics | grep ^zot_ | head -2
# zot_http_method_latency_seconds_bucket{le="+Inf",method="GET"} 1
# zot_http_method_latency_seconds_bucket{le="0.05",method="GET"} 1

Metrics can be put behind auth with accessControl.metrics.users, or opened anonymously while the rest of the registry is not.

Scrub

Walks storage verifying that every blob's content matches its digest — bit-rot and truncated-write detection. interval defaults to 24h; anything under 2h is clamped to 2h with a warning. There is also an offline one-shot, and it is emphatically not the same thing:

# With the server running:
zot scrub /etc/zot/config.json
# {"level":"warn","message":"server is running, in order to perform the scrub command
#  the server should be shut down","error":"server is running"}
# Error: server is running

systemctl stop zot
zot scrub /etc/zot/config.json
# REPOSITORY      TAG  STATUS   AFFECTED BLOB      ERROR
# library/busybox 1.37 ok
# library/busybox      affected 0459f189c19d879d…  bad blob digest

The extension does the same work in-process on a timer; the CLI command exists for forensics on a stopped registry. Note the per-manifest reporting: in a multi-arch index only the tagged manifest shows a TAG, and a corrupt blob is reported once per manifest that references it.

CVE scanning (Trivy)

zot statically links Trivy and reports vulnerabilities through the search extension, the UI and zli.

"extensions": { "search": { "enable": true, "cve": {
  "updateInterval": "24h",             // minimum 2h; how often the vulnerability DB refreshes
  "trivy": {
    "dbRepository": "ghcr.io/aquasecurity/trivy-db",           // default
    "javaDBRepository": "ghcr.io/aquasecurity/trivy-java-db",  // default
    "vulnSeveritySources": ["auto"],                           // default
    "sbom": { "enable": true, "format": "spdx-json" }
  }
} } }

# The databases land under <rootDirectory>/_trivy/ and are not small
du -sh /var/lib/zot/_trivy
# 2.7G   /var/lib/zot/_trivy
ls -la /var/lib/zot/_trivy/db/
# -rw-r--r-- 1 zot zot         153 metadata.json
# -rw-r--r-- 1 zot zot  1348702208 trivy.db      # 1.3 GB, before the Java DB

Limitations you must plan around:

• Size. Budget several gigabytes of rootDirectory for _trivy on top of your images, and expect the first update to take minutes. On an air-gapped host, mirror trivy-db and trivy-java-db into your own registry and repoint dbRepository/javaDBRepository, or scanning never works.
• TMPDIR. Trivy stages the download through the process's temp directory, not through rootDirectory. On a host with a small /tmp tmpfs the update fails with no space left on device and retries in a loop. Set TMPDIR (or systemd's PrivateTmp= target) to somewhere with room.
• A failed DB update is loud at the query and quiet in the pipeline. When the database is missing or its update is failing, queries do not return an empty result set — they refuse:

# DB update failing in the background — server log:
# {"level":"error","message":"failed to download trivy-db to destination dir", ... }
zli cve list library/busybox:1.37 --config local
# [warning] CVE DB is not ready [1] - retry in 3 seconds     (repeats, then gives up)

curl -s -X POST … -d '{"query":"{ CVEListForImage(image: \"library/busybox:1.37\") { CVEList { Id } } }"}' …
# {"errors":[{"message":"cve-db is not present","path":["CVEListForImage"]}],"data":null}

That is the good case. The failure to plan for is the scan, not the query: the background pass logs failed to perform scheduled cve scan for image and moves on, so an image that was never scanned simply has no results to fetch. Alert on the cve-db update completed log line and on zot_scheduler_* task failures, not on the scan results.

• Package-manager metadata only. Same engine, same blind spots as standalone Trivy — static binaries, vendored libraries and language dependencies outside a lockfile are invisible. Distroless and scratch images frequently report zero because there is nothing to enumerate.
• Scanning is a scheduled background pass, not a push-time gate. zot does not block a push on a vulnerability threshold. Enforcement belongs in CI, in an admission controller, or in the lint extension for annotation requirements.

For DB management, ignore policies, SBOM formats and CI wiring, see the Trivy cheatsheet.

Lint, trust and events

Configured in the block above. lint rejects pushes whose manifests lack the annotations in mandatoryAnnotations — the one place zot will refuse content on policy grounds. trust verifies cosign and Notation signatures, adding /v2/_zot/ext/mgmt endpoints for uploading public keys and certificates. events emits CloudEvents to NATS or HTTP sinks on push, delete and other registry activity, which is how you drive a rebuild pipeline or an admission cache off registry changes.

zli and zb

zli

The client CLI. It talks to the search extension, not the distribution API, so it needs a full build with search enabled on the far end.

# Profiles. NB: the older positional syntax (`zli config --list`) is deprecated.
zli config add prod https://reg.example.com
zli config list; zli config show prod; zli config set-default prod
zli config set prod showspinner false

# Repos and images
zli repo list --config prod
zli image list --config prod
zli image name infra/debian:12 --config prod
# REPOSITORY   TAG  OS/ARCH      DIGEST    SIGNED  SIZE
# infra/debian 12   linux/amd64  bbee9b20  false   51MB

zli image digest sha256:22df59ff… --config prod     # which images contain this blob
zli image base    infra/app:1.4  --config prod      # images this one is built on
zli image derived infra/base:12  --config prod      # images built on this one

# CVEs
zli cve list infra/debian:12 --config prod
# CRITICAL 4, HIGH 14, MEDIUM 39, LOW 38, UNKNOWN 6, TOTAL 101
#
# ID              SEVERITY  TITLE                                 VULNERABLE PACKAGE  INSTALL-VER
# CVE-2023-45853  CRITICAL  zlib: integer overflow and result…    zlib1g              1:1.2.13.dfsg-1

zli cve affected CVE-2023-45853 --config prod           # every image carrying it
zli cve fixed infra/debian CVE-2023-45853 --config prod # tags where it is fixed
zli cve diff infra/app:1.4 infra/app:1.3 --config prod  # regression check between two builds

# Search, referrers, server introspection
zli search query 'alpine' --config prod
zli search subject infra/debian:12 --config prod
zli status --config prod
# Server Status: online          Server Version: v2.1.20        Dist Spec Version: 1.1.1
# Built with: events, imagetrust, lint, metrics, mgmt, profile, scrub, search, sync, ui, userprefs

Every command takes --url instead of --config for a one-off, -u user:pass for credentials, and -f json|yaml for machine-readable output. zli status is the quickest way to confirm which extensions a remote instance actually has.

zb

Load generator and conformance-adjacent benchmark. It creates test repositories on the target and cleans them up afterwards unless --skip-cleanup is passed.

zb --list-tests http://127.0.0.1:8080
# Get Catalog; Push Monolith / Push Chunk Streamed / Pull at 1MB, 10MB and 100MB;
# assorted mixed read/write profiles; On-demand Sync 100MB and 1GB

zb -c 10 -n 500 -d /var/tmp/zb --test-regex 'Pull 10MB' http://127.0.0.1:8080
# Test name:            Get Catalog
# Time taken for tests: 3.799534947s
# Requests per second:  10.527604
# Complete requests:    40      Failed requests: 0      2xx responses: 40
# min: 1.281395ms   max: 38.213559ms
# p50: 3.765491ms   p75: 6.335335ms   p90: 19.555196ms   p99: 38.213559ms

-A user:pass for authenticated targets, -o json|ci-cd for machine output, -u to point sync tests at an upstream. Run it against a scratch instance: it writes real data.

OCI Conformance in Practice

zot advertises distribution-spec 1.1.1 and runs the OCI conformance suite in its own CI, referrers API included. In practice that means the OCI-native tools work without special handling, and Docker-era clients need one config key.

# skopeo — see the Container Registries cheatsheet for the full treatment
skopeo copy docker://docker.io/library/alpine:3.22 docker://reg:8080/infra/alpine:3.22
skopeo inspect --raw docker://reg:8080/infra/alpine:3.22
skopeo copy oci:/var/lib/zot/infra/alpine:3.22 dir:/tmp/exported   # storage is a valid OCI layout

# crane
crane copy docker.io/library/alpine:3.22 reg:8080/infra/alpine:3.22
crane catalog reg:8080

# oras — non-image artifacts push and pull unchanged
oras push reg:8080/artifacts/notes:v1 \
  --artifact-type application/vnd.example.config.v1+json notes.json:application/json

# oras — attach a referrer to an existing image
oras attach reg:8080/infra/debian@sha256:bbee9b20… \
  --artifact-type application/vnd.example.sbom.v1+json sbom.json:application/json

Referrers

D=sha256:bbee9b2048ce0dbd26ba87598dd19e335b9de3a3a141cfe48cddf4715d97bbef
curl -s http://reg:8080/v2/infra/debian/referrers/$D | jq -c '.manifests[] | {artifactType, digest}'
# {"artifactType":"application/vnd.example.sbom.v1+json","digest":"sha256:bd4b9076ff2d…"}

# artifactType filtering is supported, and advertised in the response header
curl -sI "http://reg:8080/v2/infra/debian/referrers/$D?artifactType=application/vnd.example.sbom.v1%2Bjson" \
  | grep -i oci-filters
# Oci-Filters-Applied: artifactType

oras discover reg:8080/infra/debian@$D
# 127.0.0.1:8080/infra/debian@sha256:bbee9b20…
# └── application/vnd.example.sbom.v1+json
#     └── sha256:bd4b9076ff2d…

Docker schema2 — the migration footgun

zot is OCI-only by default. A client pushing a Docker Distribution schema2 manifest — old Docker daemons, podman push --format v2s2, some CI images, anything mirroring from a schema2-era registry — is rejected:

podman push --format v2s2 reg:8080/compat/debian:v2s2
# Copying blob sha256:cd482426e567…
# Writing manifest to image destination
# Error: writing manifest: uploading manifest v2s2 to reg:8080/compat/debian: manifest invalid

The fix is one key, and docker2s2 is currently its only valid value:

"http": { "address": "0.0.0.0", "port": "8080", "compat": ["docker2s2"] }

podman push --format v2s2 reg:8080/compat/debian:v2s2     # succeeds
curl -sI -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' \
  http://reg:8080/v2/compat/debian/manifests/v2s2 | grep -i content-type
# Content-Type: application/vnd.docker.distribution.manifest.v2+json

The manifest is stored as-is, with the Docker media type recorded in index.json — zot does not convert it. Enable compat for migration and for clients you do not control; leave it off if you want the registry to enforce OCI-only content.

Deployment

systemd

[Unit]
Description=OCI Distribution Registry
Documentation=https://zotregistry.dev/
After=network-online.target local-fs.target
Wants=network-online.target

[Service]
Type=simple
User=zot
Group=zot
ExecStart=/usr/bin/zot serve /etc/zot/config.json
Restart=on-failure
RestartSec=5s
# zot holds a file descriptor per concurrent blob operation; the default 1024 is far too low
LimitNOFILE=500000
# Trivy stages DB downloads through TMPDIR, not rootDirectory — give it room
Environment=TMPDIR=/var/lib/zot/tmp
# Bound memory so a runaway scan cannot take the host with it
MemoryHigh=8G
MemoryMax=10G
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/zot /var/log/zot
PrivateDevices=yes

[Install]
WantedBy=multi-user.target

Do not add ExecReload=/bin/kill -HUP $MAINPID: SIGHUP terminates zot. There is no reload signal — editing the config file in place is the reload mechanism.

sudo adduser --system --no-create-home --group zot
sudo install -d -o zot -g zot /var/lib/zot /var/log/zot /var/lib/zot/tmp
sudo install -d -o root -g zot -m 750 /etc/zot
sudo -u zot /usr/bin/zot verify /etc/zot/config.json
sudo systemctl daemon-reload && sudo systemctl enable --now zot

TLS and reverse proxies

For a single node, terminate TLS in zot: "tls": { "cert": …, "key": … } under http. Adding "cacert" to that same block turns on client-certificate verification — that is how mTLS is enabled, there is no separate switch.

Behind a proxy, three settings matter more than the rest:

server {
    listen 443 ssl;
    http2 on;
    server_name reg.example.com;
    ssl_certificate     /etc/letsencrypt/live/reg.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/reg.example.com/privkey.pem;

    client_max_body_size 0;               # layers are large; the default 1m breaks pushes

    location / {
        proxy_pass         http://127.0.0.1:8080;
        proxy_set_header   Host              $host;
        proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Proto $scheme;
        proxy_request_buffering off;       # stream uploads rather than spooling to disk
        proxy_read_timeout 900s;
        proxy_send_timeout 900s;
    }
}

Set http.externalUrl to the public URL, or OIDC callbacks and UI-generated pull commands point at the backend address. Raise http.readTimeout and http.writeTimeout above their 60s defaults for very large layers over slow links — they are the server-side counterpart to the proxy timeouts above.

Sizing

• Disk is the binding constraint. Images, plus roughly one extra copy for anything that predates dedupe, plus 3 GB or so for _trivy if CVE scanning is on, plus headroom for sync staging and GC.
• Memory scales with concurrency and metaDB size, not with stored bytes. A few hundred MB idles comfortably; CVE scanning is what spikes it, which is why the unit above caps it.
• File descriptors: LimitNOFILE=500000 is the upstream recommendation and is not excessive under load.
• Scheduler: scheduler.numWorkers defaults to four times the CPU count. Lower it on a small edge node where sync, GC and CVE tasks otherwise starve the request path.

Scale-out clustering

Multiple zot instances can share one object-store backend, with requests routed by repository name using a shared siphash key. Every member carries the same storage block (a storageDriver plus remoteCache: true and a shared cacheDriver) and the same cluster block:

"cluster": {
  "members": ["zot-0:9000", "zot-1:9000", "zot-2:9000"],
  "hashKey": "loremipsumdolors"
}

hashKey must be exactly 16 bytes (a 128-bit siphash key), and both members and hashKey must be identical on every member — this is static membership, not gossip. Put a plain L4 or L7 load balancer in front; the cluster handles internal forwarding to the owning member. Set dedupe: false here: hard-link dedupe is meaningless on object storage and the shared cache does that job instead.

Quick Reference

Task	Command
Start	zot serve /etc/zot/config.json
Validate config	zot verify /etc/zot/config.json
Dump config JSON Schema	zot schema > zot-schema.json
Rebuild metaDB from storage	zot serve /etc/zot/config.json --force-reparse
Offline integrity check	zot scrub /etc/zot/config.json (server stopped)
Run GC + retention once	zot verify-feature retention /etc/zot/config.json -t 120s
Which extensions are compiled in	zot --version → binary-type
Which extensions a remote has	zli status --config prod
Reload config	Edit the file in place (no signal; SIGHUP shuts down)
Catalogue	curl -s $REG/v2/_catalog
Tags	curl -s $REG/v2/<repo>/tags/list
Referrers	curl -s $REG/v2/<repo>/referrers/<digest>
Extension discovery	curl -s $REG/v2/_oci/ext/discover
Server info	curl -s $REG/v2/_zot/ext/mgmt
GraphQL search	POST $REG/v2/_zot/ext/search
Prometheus metrics	curl -s $REG/metrics (full build only)
List images	zli image list --config prod
List CVEs	zli cve list <repo>:<tag> --config prod
CVE regression between builds	zli cve diff <new> <old> --config prod
Benchmark	zb -c 10 -n 500 -d /var/tmp/zb $REG
Export a repo as an OCI layout	tar -C /var/lib/zot -cf out.tar <repo>/

Storage defaults (v2.1.20)

Key	Default	Notes
storage.dedupe	true	Hard links within a store
storage.gc	true	Online, no downtime
storage.gcDelay	1h	Grace period before a blob is collectable
storage.gcInterval	1h	Periodic sweep cadence
storage.gcTimeWindow	unset	Empty means GC may run at any time
storage.commit	false	true fsyncs every write
storage.retention.delay	1h	Grace period for untagged manifests
storage.retention.policies	none	No policy: untagged manifests are expired, tagged ones never are
storage.maxRepos	0	Unlimited
storage.fastRestart	unset	Top-level only; not honoured under subPaths
storage.redirectBlobURL	false	true issues 307s to presigned object-store URLs
http.readTimeout / writeTimeout	60s	Raise for very large layers
http.realm	unset	Shows in the auth challenge
http.auth.failDelay	0	Seconds to stall a failed auth
http.compat	unset	["docker2s2"] to accept Docker schema2
scheduler.numWorkers	4 × NumCPU	Lower on constrained nodes
extensions.search.cve.updateInterval	2h	Shorter values are clamped to 2h with a warning
extensions.scrub.interval	24h	Shorter than 2h is clamped to 2h with a warning

Common Issues and Solutions

Issue	Cause	Solution
Extension endpoints return 404	Running zot-minimal; zot verify accepts the extensions block and serve logs one warn per skipped extension at startup, then ignores them	zot --version and check binary-type; grep the boot log for doesn't include this feature; use the full build
manifest invalid on push from an older client	Docker schema2 manifest, and zot is OCI-only by default	Add "compat": ["docker2s2"] to http
zot verify passes, zot serve fails	verify opens the LDAP/OpenID credentialsFile and sessionKeysFile, but never stats the htpasswd file or TLS certificate	Smoke-test a real start; stat htpasswd and cert paths in CI
SIGHUP / systemctl reload kills the registry	SIGHUP is registered as a shutdown signal	Remove ExecReload=; edit the config file in place to reload
Config change ignored on Kubernetes	The watcher fires only on fsnotify.Write; a ConfigMap projection renames	Restart the pod on a config checksum annotation
access control config requires one of htpasswd, ldap, openid or mTLS	accessControl present with no authentication configured	Add an auth block, or make every policy an anonymousPolicy
A user in the htpasswd file can never log in	Hash is not bcrypt — zot supports bcrypt only, and silently fails to match anything else	Regenerate with htpasswd -B; never -m, -s or -p
Anonymous /v2/_catalog returns 200 with {"repositories":[]}	Working as designed — the catalogue is filtered per identity, never rejected	Nothing to fix; test authorisation against a manifest URL, which returns 401
Push denied but pull works for the same user	defaultPolicy: ["read"] with no create for that repository pattern	Add a policies entry granting create, or widen defaultPolicy
CI account loses access after a pattern is added	Only the longest matching pattern is evaluated; the new one has policies but no defaultPolicy, and nothing is inherited from **	Re-grant explicitly under the more specific pattern
zli cve list loops on CVE DB is not ready, GraphQL returns cve-db is not present	The Trivy DB has never finished downloading — check the log for failed to download trivy-db	Fix TMPDIR/disk/egress; alert on the update task, not on the scan results
Trivy DB update fails with no space left on device in /tmp	Trivy stages the download through TMPDIR, not rootDirectory	Set TMPDIR (systemd Environment=) to a filesystem with several GB free
Sync fails with no space left on device and retries forever	Staging area is undersized, or downloadDir is on a small tmpfs	Point downloadDir at real disk with headroom for the largest image
Sync mirrors far more than expected	No content filters — an onDemand registry proxies anything, a scheduled one copies everything	Always set content with prefix and tags filters
Repository listed by /v2/_catalog but missing from search / UI	fastRestart: true, so the start-up storage walk was skipped and metaDB never saw the out-of-band files. Without fastRestart this does not happen	zot serve … --force-reparse
Old tags accumulate forever despite gc: true	GC reclaims unreferenced blobs; expiring tagged manifests needs a keepTags retention policy, and there is no default	Add retention.policies with keepTags, including a ["**"] catch-all
Untagged manifests disappear without a retention policy	deleteUntagged defaults to true when no policy matches, not false	Add a matching policy with "deleteUntagged": false
Retention deleted more than intended	First matching repository policy wins outright; unmatched repos fall back to the defaults, not to your policy	Set dryRun: true and run zot verify-feature retention before committing
Concurrent pushes corrupt or vanish	gcDelay set below the time a large push takes	Raise gcDelay; keep it well above the slowest realistic push
Disabling dedupe hangs the registry at start-up	Turning dedupe off triggers a full restore pass unlinking every deduped blob	Expect a long start and a large disk spike; it runs once and records _restore_complete
/metrics returns JSON, Prometheus scrape finds nothing	Minimal build exposes an internal JSON dump, not Prometheus text	Run the zxp exporter sidecar, or switch to the full build with the metrics extension
UI loads but shows no images	ui enabled without search	Enable search; the UI is a client of the GraphQL API
OIDC login redirects to 127.0.0.1	externalUrl unset behind a proxy	Set http.externalUrl to the public URL
Sessions drop at random behind a load balancer	No shared sessionKeysFile, so each replica signs cookies differently	Provision a shared key file, or pin sessions at the balancer
S3 backend ignores half the settings	Storage-driver keys are lowercase (rootdirectory, forcepathstyle); camelCase is silently a different key	Use the driver's own spelling inside storageDriver
Large pushes fail behind nginx	Default client_max_body_size and request buffering	client_max_body_size 0; and proxy_request_buffering off;
too many open files under load	Default LimitNOFILE	LimitNOFILE=500000 in the unit file

---

Related Topics

The following topics complement this cheatsheet and would be valuable additions:

1. Container Registries - skopeo in depth, Docker Hub, GHCR, ECR, Artifact Registry, ACR, Harbor and Quay, plus the generic auth, mirroring and air-gap patterns this sheet defers to
2. Trivy - Vulnerability database management, scanner internals, ignore policies, SBOM generation, and the CI gating that zot's built-in scanning deliberately does not do
3. Container Security - Signing with cosign and Notation, SLSA provenance, admission control, and what a registry can and cannot enforce
4. Podman Quadlets and systemd - Running zot itself as a managed service, with healthcheck-gated readiness, volumes and scheduled maintenance units
5. Kubernetes - In-cluster registry deployment, image pull secrets, node-level mirroring, and pointing containerd at a zot pull-through cache
6. Prometheus - Scraping the metrics extension or zxp, and alerting on the background tasks whose silent failure this sheet warns about