Available for day contractsFrom 21st September I have availability for day and half day contracts. Please contact for more information.

Contact →
mikepreston.org
~/writing/git-is-the-source-of-truth-until-it-isnt.md
2026-07-14· 2 min read· #essays

Git is the Source of Truth—Until It Isn't

A confident cartographer holding a pristine map while, through the window behind him, the moonlit landscape has rearranged itself overnight so map and territory no longer agree — 1960s gouache.

Declarative doesn't mean deterministic. We've spent a decade telling ourselves that if it's in Git, it's reality. It isn't. Reality is whatever the cluster decided to do at 03:00 when traffic spiked and your HPA started doing its job.

The classic example: deploy a Deployment with replicas: 3 via ArgoCD, then attach an HorizontalPodAutoscaler that scales between 3 and 20. Traffic arrives. HPA scales to 12. ArgoCD notices drift. ArgoCD "fixes" it back to 3. HPA scales back up. Round and round we go, until either your pager goes off or someone finally reads the docs.

The fix is well-known but worth stating plainly: stop putting replicas in Git for anything fronted by an HPA. Either omit the field entirely, or tell Argo to ignore it:

spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas

That's the easy one. The harder lesson is that HPAs are just the most obvious case of a much broader problem. Anything that mutates live state will fight your GitOps controller eventually:

  • VPA rewriting resource requests
  • Cluster autoscaler annotations
  • Service mesh sidecar injection
  • Cert-manager populating tls.crt
  • Admission webhooks adding labels, tolerations, node selectors
  • Operators reconciling status into spec fields they shouldn't

Every one of these is a controller doing exactly what we asked it to. The conflict isn't a bug — it's the architecture. Kubernetes is a system of competing controllers, each with its own opinion about what the cluster should look like. Git is just one more voice in the chorus, and it's frequently wrong about the things it claims authority over.

So what do we actually do? Be specific about ownership. For each field on each resource, decide which controller owns it, and make every other controller — including Argo — keep its hands off. Server-side apply with field managers helps. ignoreDifferences helps. Discipline helps most.

Treat Git as the source of truth for intent, not state. The cluster is the source of truth for state. Confusing the two is how we end up debugging replica flapping at the weekend.