Kubernetes Jobs and CronJobs

Introduction¶

Deployments keep pods running forever. Jobs run a task to completion and stop. CronJobs create Jobs on a cron schedule. Use them for migrations, batch processing, backups, report generation.

Job¶

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
spec:
  backoffLimit: 4                 # retries before the Job is marked Failed
  activeDeadlineSeconds: 600      # kill the whole Job after 10 min regardless
  ttlSecondsAfterFinished: 3600   # auto-delete the Job (and its pods) 1h after finishing
  template:
    spec:
      restartPolicy: Never        # or OnFailure; NEVER Always for a Job
      containers:
        - name: migrate
          image: myapp:1.4.2
          command: ["python", "-m", "app.migrate"]

• restartPolicy: Never — a failed pod stays (for logs); the Job creates a new pod, up to backoffLimit times, with exponential back-off (10s, 20s, 40s… capped at 6m).
• restartPolicy: OnFailure — the kubelet restarts the container in place; fewer pod objects, but you lose the crashed container's separate logs.
• backoffLimit counts failures across all pods.
• activeDeadlineSeconds is a hard wall on total runtime.

Parallelism¶

spec:
  completions: 20        # need 20 successful pods total
  parallelism: 5         # run 5 at a time

Or indexed completion (completionMode: Indexed) gives each pod a JOB_COMPLETION_INDEX (0–19) so a pod can process "its" shard. Or leave completions unset with parallelism: N for a work-queue pattern (pods pull from a queue, Job succeeds when one pod exits 0 — actually when all running pods finish and at least one succeeded; usually you want the queue to drive it).

CronJob¶

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-report
spec:
  schedule: "0 2 * * *"                 # 02:00 daily (cluster timezone — set .spec.timeZone)
  timeZone: "Asia/Colombo"
  concurrencyPolicy: Forbid            # don't start a new run if the last is still going
  startingDeadlineSeconds: 300         # if the controller missed the slot, still start within 5 min
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      backoffLimit: 2
      activeDeadlineSeconds: 3600
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: report
              image: reports:latest
              args: ["--date", "yesterday"]

• concurrencyPolicy: Allow (default — overlapping runs), Forbid (skip the new one), Replace (kill the running one, start fresh).
• startingDeadlineSeconds: if the CronJob controller was down or busy and missed the scheduled time by more than this, the run is skipped. If unset and the controller misses 100+ schedules, it stops scheduling entirely (a safety valve — but a nasty surprise). Always set it.
• timeZone (1.27+ stable) — before that, CronJobs ran in the kube-controller-manager's timezone (usually UTC).

History and cleanup¶

• CronJobs keep successfulJobsHistoryLimit / failedJobsHistoryLimit old Jobs (default 3/1) for log inspection; older ones are deleted.
• Standalone Jobs are not cleaned up automatically — set ttlSecondsAfterFinished or they pile up (and their completed pods count toward pod limits).

Verification and troubleshooting¶

kubectl get jobs
kubectl get cronjob nightly-report
kubectl get pods -l job-name=db-migrate
kubectl logs job/db-migrate
kubectl describe job db-migrate            # completions, failures, events
kubectl create job --from=cronjob/nightly-report manual-run-1   # trigger a CronJob now

• Job stuck, pods Pending — scheduling (resources, PVC, affinity), not a Job problem. describe the pod.
• Job "Failed" quickly — hit backoffLimit. kubectl logs the failed pods (they persist with restartPolicy: Never). Common: the command exits non-zero, a migration conflict, a missing secret.
• CronJob not running — kubectl get cronjob shows LAST SCHEDULE. If it's old: the schedule syntax is wrong (validate the cron expression), .spec.suspend: true, the controller missed too many (set startingDeadlineSeconds), or the timezone isn't what you think (set timeZone).
• Overlapping CronJob runs causing chaos — concurrencyPolicy: Forbid or Replace.
• Migration CronJob ran twice, corrupted data — concurrencyPolicy: Allow (default) + a slow run. Use Forbid, and make the job idempotent.
• Completed Jobs filling the namespace — standalone Jobs with no ttlSecondsAfterFinished. Add it, or a cleanup CronJob.
• CronJob for a backup: pod evicted mid-run — it's BestEffort (no requests); set resource requests, and activeDeadlineSeconds so a hung run doesn't block the next.

Related tools and reading¶

• On-site: Crontab generator, YAML ↔ JSON converter.
• Related posts: The Kubernetes pod lifecycle, Debugging CrashLoopBackOff.