CoreDNS in Kubernetes

Introduction¶

CoreDNS runs as a Deployment in kube-system and is the cluster's DNS server. Pods get its Service IP as their nameserver (via the kubelet), and it resolves Service.namespace.svc.cluster.local names plus forwards everything else upstream.

The Corefile¶

.:53 {
    errors
    health { lameduck 5s }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
        ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf {
        max_concurrent 1000
    }
    cache 30
    loop
    reload
    loadbalance
}

• kubernetes plugin — answers *.cluster.local from the API (Services, Endpoints, and pod A records).
• forward . /etc/resolv.conf — anything not cluster.local goes to the node's upstream resolvers. Point it at explicit resolvers (forward . 1.1.1.1 9.9.9.9) if the node's resolv.conf is unreliable.
• cache 30 — 30s positive/negative cache.
• reload — picks up ConfigMap changes without a restart (~2 min).

The ndots:5 problem¶

The kubelet writes this into every pod's /etc/resolv.conf:

search team-a.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

ndots:5 means: if the name has fewer than 5 dots, try each search domain first before trying it as-is. So curl https://api.example.com (2 dots) generates:

api.example.com.team-a.svc.cluster.local   -> NXDOMAIN
api.example.com.svc.cluster.local          -> NXDOMAIN
api.example.com.cluster.local              -> NXDOMAIN
api.example.com                            -> the real answer (finally)

Four queries (×2 for A+AAAA) for one external hostname. On a busy pod this hammers CoreDNS and adds latency.

Mitigations:

• Use fully-qualified names with a trailing dot in app config: api.example.com. — 0 extra lookups.
• Set dnsConfig on the pod to lower ndots (e.g. ndots: 2) — but then short in-cluster names like api (same namespace) still work via the search list, while api.team-b needs the full api.team-b.svc.cluster.local.
• NodeLocal DNSCache (below) — the biggest win.

NodeLocal DNSCache¶

A DaemonSet running a CoreDNS cache on every node, listening on a link-local IP (169.254.20.10). Pods query the local cache first:

• Cache hits never leave the node → huge latency and load reduction.
• Uses TCP to the upstream CoreDNS (avoids conntrack UDP races that caused the infamous 5s DNS timeouts on some kernels).
• Cache misses forward to the cluster CoreDNS.

Deploy it (the upstream manifest) and update the kubelet's --cluster-dns or use the interface-based setup. Standard on managed clusters as an add-on.

Tuning for scale¶

• Replicas / autoscaling — cluster-proportional-autoscaler scales CoreDNS with node/core count. Default 2 replicas is too few for a big cluster.
• Resources — give CoreDNS a real CPU request; under-provisioned CoreDNS = cluster-wide slow DNS.
• cache — raise TTL if your Services are stable.
• Disable AAAA for IPv4-only clusters with the template plugin to cut query volume in half (pods still ask for both).

Verification and troubleshooting¶

kubectl -n kube-system get pods -l k8s-app=kube-dns
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=50
kubectl -n kube-system get cm coredns -o yaml

kubectl run t --rm -it --image=nicolaka/netshoot -- bash
  nslookup kubernetes.default
  nslookup api.team-a.svc.cluster.local
  dig +search +short api.example.com          # see the search-domain expansion
  cat /etc/resolv.conf

• All in-cluster DNS fails — CoreDNS pods down/crashlooping (check the Corefile for a syntax error — reload logs it), or a NetworkPolicy blocking UDP/TCP 53 to kube-system, or the kube-dns Service has no endpoints.
• nslookup: connection timed out; no servers could be reached — the pod's /etc/resolv.conf nameserver (kube-dns Service IP) isn't reachable: kube-proxy issue, NetworkPolicy, or CNI.
• Intermittent 5s stalls on DNS — the classic conntrack UDP race (net.netfilter.nf_conntrack insert races). NodeLocal DNSCache (TCP upstream) fixes it; or options single-request-reopen in dnsConfig.
• External lookups slow, internal fine — ndots:5 amplification. FQDN with trailing dot, or NodeLocal DNSCache, or lower ndots.
• loop plugin crashes CoreDNS on start — the node's /etc/resolv.conf points back at CoreDNS (a resolution loop). Forward to explicit upstream IPs instead of /etc/resolv.conf.
• Stale Service IP resolved after a Service was recreated — cache TTL; wait cache seconds or flush. Apps that cache DNS forever (some JVM setups) need networkaddress.cache.ttl tuned.

Related tools and reading¶

• On-site: DNS lookup, Port Checker.
• Related posts: Kubernetes Services and Endpoints, resolv.conf and systemd-resolved.