GitOps with Argo CD

Introduction¶

GitOps: the desired state of your infrastructure lives in Git, and an agent in the cluster continuously reconciles the cluster to match. CI builds and pushes images and bumps a manifest; it never kubectl applys. Argo CD is the common agent.

Why pull, not push¶

• No cluster credentials in CI. CI only needs Git write. Argo CD, inside the cluster, has the deploy permissions.
• Git is the audit log and the rollback mechanism — git revert a bad change.
• Drift is visible. Someone kubectl edits a Deployment → Argo CD shows it OutOfSync (and can revert it).

The Application¶

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/org/deploy.git
    targetRevision: main
    path: apps/api/overlays/prod        # kustomize / helm / plain manifests
  destination:
    server: https://kubernetes.default.svc
    namespace: api
  syncPolicy:
    automated:
      prune: true          # delete resources removed from Git
      selfHeal: true       # revert manual changes to match Git
    syncOptions:
      - CreateNamespace=true
      - ApplyOutOfSyncOnly=true
    retry:
      limit: 5
      backoff: { duration: 10s, factor: 2, maxDuration: 3m }

• automated.selfHeal: true — Argo CD reverts out-of-band changes. Powerful; make sure nothing legitimately needs to mutate those objects (HPA changes replicas — exclude that field with ignoreDifferences).
• prune: true — removing a file from Git deletes the resource. Without it, orphans accumulate.

Scaling to many apps¶

• App-of-apps — one Application whose path contains more Application manifests. A single root app manages dozens.
• ApplicationSet — a generator (list, Git directory, cluster, PR) templates Applications. "One app per directory under apps/", or "deploy this to every cluster labelled env=prod".

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata: { name: all-apps, namespace: argocd }
spec:
  generators:
    - git:
        repoURL: https://github.com/org/deploy.git
        revision: main
        directories: [{ path: "apps/*" }]
  template:
    metadata: { name: '{{path.basename}}' }
    spec:
      project: default
      source: { repoURL: '...', targetRevision: main, path: '{{path}}' }
      destination: { server: 'https://kubernetes.default.svc', namespace: '{{path.basename}}' }
      syncPolicy: { automated: { prune: true, selfHeal: true } }

Secrets¶

Argo CD renders manifests from Git — you can't commit plaintext Secrets. Options:

• Sealed Secrets — encrypt to a cluster-specific key; the ciphertext is safe in Git, the controller decrypts in-cluster.
• External Secrets Operator — commit an ExternalSecret that references Vault/AWS SM; the operator materialises the real Secret.
• SOPS + KSOPS/argocd-vault-plugin — Argo decrypts at render time.

Image updates¶

• Argo CD Image Updater — watches the registry, commits a new tag/digest to Git when a new image appears.
• Or CI bumps the digest in the kustomize/helm values and pushes — simplest, keeps the Git history meaningful.

Verification and troubleshooting¶

argocd app list
argocd app get api
argocd app diff api               # Git vs live
argocd app sync api
argocd app history api            # revisions; argocd app rollback api <id>
kubectl -n argocd logs deploy/argocd-application-controller

• App stuck OutOfSync after sync — a resource Argo can't fully reconcile: a mutating webhook/controller changes it right back (HPA on replicas, a service mesh injecting sidecars). Use ignoreDifferences for those fields, or Replace=false.
• ComparisonError / "rpc error: repo not accessible" — repo credentials (SSH key / token) missing or expired in Argo CD's repo config.
• Sync fails "another operation is already in progress" — a previous sync hung; terminate it (argocd app terminate-op api).
• SharedResourceWarning / resource owned by another app — two Applications manage the same object. Split the paths or use argocd.argoproj.io/tracking-id.
• selfHeal fighting a legitimate change — someone/something needs to mutate a managed field. Either bring that change into Git, or ignoreDifferences, or drop selfHeal for that app.
• Prune deleted something important — a manifest was removed from Git (a bad rebase, a moved file). git revert; consider PruneLast=true and prune guarded by a sync window for critical apps.
• Helm/Kustomize render differs from helm template locally — Argo's build options (--include-crds, value files, kustomize version). Match them in the Application's source.

Related tools and reading¶

• On-site: YAML ↔ JSON converter.
• Related posts: Trunk-based development, Kubernetes version upgrades without drama.