Varnish Cache in Front of a Site

Introduction¶

Varnish is a reverse-proxy cache that sits between clients and your web app, serving cached responses from memory at very high rates. It speaks HTTP only (no TLS — put Nginx/HAProxy in front for HTTPS), and is configured in its own language, VCL.

Topology¶

client --HTTPS--> [ Nginx/HAProxy: TLS terminate ] --HTTP:80--> [ Varnish ] --HTTP--> [ app ]

Varnish also often does HTTP→HTTPS awareness via an X-Forwarded-Proto header the TLS terminator sets.

The VCL lifecycle (simplified)¶

• vcl_recv — request arrives. Decide: cache lookup (hash), pass (pass — proxy but don't cache), pipe, or synth (return a canned response). Normalise the request here (strip tracking cookies, lowercase Host).
• vcl_hash — build the cache key (default: URL + Host). Add to it here.
• vcl_hit / vcl_miss — object found / not found.
• vcl_backend_response — response from the app. Decide TTL, whether to cache, strip Set-Cookie, set grace/keep.
• vcl_deliver — final touches before sending to the client (add debug headers).

A starter VCL¶

vcl 4.1;

backend default {
    .host = "127.0.0.1";
    .port = "8080";
    .probe = {
        .url = "/healthz";
        .interval = 5s;
        .timeout = 2s;
        .window = 5;
        .threshold = 3;
    }
}

sub vcl_recv {
    # Don't cache non-idempotent methods
    if (req.method != "GET" && req.method != "HEAD") {
        return (pass);
    }
    # Logged-in users / carts: bypass
    if (req.http.Authorization || req.http.Cookie ~ "session=") {
        return (pass);
    }
    # Strip cookies for cacheable content
    unset req.http.Cookie;
    return (hash);
}

sub vcl_backend_response {
    # App said private/no-store? honour it
    if (beresp.http.Cache-Control ~ "(?i)(no-store|private)") {
        set beresp.uncacheable = true;
        return (deliver);
    }
    # Default TTL for HTML if the app didn't set one
    if (beresp.ttl <= 0s) {
        set beresp.ttl = 5m;
    }
    # Ignore app cookies on cacheable responses
    unset beresp.http.Set-Cookie;
    # Serve stale for up to 6h if the backend is sick
    set beresp.grace = 6h;
}

sub vcl_deliver {
    set resp.http.X-Cache = obj.hits > 0 ? "HIT" : "MISS";
}

Cookies — the eternal problem¶

Any Set-Cookie in a response, or a request Cookie, makes Varnish not cache by default. That's why anonymous pages don't cache until you explicitly unset req.http.Cookie for content that's safe, and unset beresp.http.Set-Cookie on cacheable responses. Be surgical: keep the session cookie for the paths that need it (/account, /checkout, /api), strip it elsewhere.

Vary¶

Vary: Accept-Encoding is fine (Varnish handles gzip). Vary: Cookie or Vary: User-Agent explode the cache into near-infinite variants — normalise or strip those headers in vcl_backend_response.

Purging and banning¶

• PURGE — invalidate one exact object:

  sub vcl_recv {
      if (req.method == "PURGE") {
          if (!client.ip ~ purge_acl) { return (synth(405)); }
          return (purge);
      }
  }
  

  curl -X PURGE http://varnish/path/to/page
• Ban — invalidate by pattern (regex on URL/host), lazily applied on next lookup:

  varnishadm "ban req.http.host == example.com && req.url ~ ^/products/"
  

  Bans accumulate; use ban obj.http.x-url ~ ... with the "ban lurker" enabled so old objects are cleaned proactively.

Grace mode¶

beresp.grace keeps an expired object around; if the backend is down or slow, Varnish serves the stale copy while refreshing in the background. This is Varnish's headline resilience feature — tune grace generously.

Verification and troubleshooting¶

varnishstat            # hit rate (cache_hit / (cache_hit + cache_miss)), evictions, backend health
varnishlog -g request -q 'ReqURL ~ "/products"'   # full lifecycle for matching requests
varnishtop -i ReqURL   # most-requested URLs
varnishadm backend.list

• Hit rate near zero — cookies (request or Set-Cookie), Cache-Control: private/no-cache from the app, Vary: Cookie, or short/zero TTL. varnishlog shows TTL and Hit-For-Pass decisions.
• Users see each other's account pages — you unset req.http.Cookie too broadly. Exclude /account, /cart, /api, and anything with a session.
• 503 Backend fetch failed — backend down and no grace object, or the .probe marks it unhealthy. varnishadm backend.list.
• Stale content forever — no purge/ban in the deploy, or bans not being applied (ban lurker disabled and objects never re-looked-up).
• Memory pressure / evictions high (n_lru_nuked) — cache smaller than the working set. Raise -s malloc,<size> (or use -s file), shorten TTLs on bulky low-value objects.
• Compression served twice / broken gzip — let Varnish handle it; don't have both Nginx and the app and Varnish all gzipping. Accept-Encoding normalisation in vcl_recv.

Related tools and reading¶

• On-site: HTTP Header Analyzer.
• Related posts: Nginx content caching, An Nginx reverse proxy with TLS.