Kubernetes Services and Endpoints

Introduction¶

Pods are ephemeral and get new IPs constantly. A Service gives a stable name and virtual IP that load-balances to a changing set of pods. The set is tracked by EndpointSlices, populated from the Service's label selector.

Service types¶

apiVersion: v1
kind: Service
metadata:
  name: api
spec:
  selector:
    app: api
  ports:
    - name: http
      port: 80          # the Service port
      targetPort: 3000  # the container port

How endpoints get populated¶

The endpoints controller watches for pods matching spec.selector that are Ready (passing their readiness probe) and writes their IP:port into EndpointSlices. Not ready → not in the slice → no traffic.

kubectl get svc api
kubectl get endpointslices -l kubernetes.io/service-name=api
kubectl get endpoints api        # legacy view, still works

If kubectl get endpoints api shows <none>, the Service is pointing at nothing — that's your problem, 90% of the time.

kube-proxy¶

On each node, kube-proxy turns Services + EndpointSlices into forwarding rules:

• iptables mode (default): a chain per Service that DNATs the ClusterIP to a random endpoint. O(n) rules; fine to a few thousand Services.
• IPVS mode: kernel L4 load balancer, better at scale, real algorithms (rr, lc, etc.).
• nftables mode (newer): replacing iptables mode.

DNS: api.myns.svc.cluster.local resolves to the ClusterIP via CoreDNS. api alone works from a pod in the same namespace (search domains).

Debugging "service exists, nothing answers"¶

# 1. Does the Service have endpoints?
kubectl get endpoints api
#   <none>  -> selector doesn't match ready pods. Check:
kubectl get pods -l app=api        # do pods with this label exist and are they Ready?
kubectl describe svc api           # confirm the selector

# 2. Do the ports line up?
#    Service targetPort must equal the container's listening port.

# 3. Test from inside the cluster
kubectl run tmp --rm -it --image=nicolaka/netshoot -- bash
  curl -v http://api.myns.svc.cluster.local
  nslookup api.myns.svc.cluster.local
  curl -v http://<pod-ip>:3000       # does the pod itself answer?

# 4. NetworkPolicy?
kubectl get networkpolicy -n myns

Verification and troubleshooting¶

• endpoints is <none> — selector mismatch, or pods aren't Ready (readiness probe failing → they're excluded on purpose), or the pods are in a different namespace (Services are namespace-scoped).
• Endpoints present, connection refused — targetPort ≠ the port the app listens on, or the app binds 127.0.0.1 inside the container instead of 0.0.0.0.
• Works by pod IP, not by Service IP — kube-proxy issue on the node, or DNS. curl the ClusterIP directly to isolate DNS from proxying.
• Intermittent failures / some requests fail — one endpoint is bad but still Ready (readiness probe too lenient), or a rolling update with no preStop delay. Tighten readiness.
• NodePort works, LoadBalancer stuck <pending> — no cloud controller manager (bare metal — use MetalLB), or a quota/permissions issue provisioning the LB.
• DNS resolves but slowly / 5s timeouts — CoreDNS under-resourced, or ndots:5 causing many failed lookups. Add dnsConfig with a lower ndots, or use FQDNs with a trailing dot.
• Cross-namespace call fails — use the FQDN svc.ns.svc.cluster.local; the short name only searches the caller's namespace.

Related tools and reading¶

• On-site: Port Checker, DNS lookup.
• Related posts: ingress-nginx with TLS, The Kubernetes pod lifecycle.