Recovering oVirt After Certificates Have Expired

Introduction¶

If oVirt certificates expire before you renew them, several things break at once and some of the normal renewal tooling refuses to run (because its client cert is expired). This is the recovery procedure. Do the before-expiry renewal instead whenever you can.

Symptoms¶

• Admin/VM portal shows a TLS error or the engine (httpd/ovirt-engine) won't start.
• All hosts flip to Non Operational / Connecting — the engine↔VDSM mTLS fails.
• VNC/SPICE consoles fail (ovirt-websocket-proxy cert).
• engine-setup or host enrollment errors with certificate verification failures.
• Running VMs keep running — the engine and its certs are not in the VM data path. This is your breathing room.

Step 1: confirm it's expiry¶

openssl x509 -in /etc/pki/ovirt-engine/ca.pem -noout -dates
openssl x509 -in /etc/pki/ovirt-engine/certs/engine.cer -noout -dates
openssl x509 -in /etc/pki/ovirt-engine/certs/apache.cer -noout -dates
# On a host:
openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -dates

Note whether it's just leaf certs (recoverable with engine-setup) or the CA itself (harder — see the end).

Step 2: the clock-rollback trick (only if tools won't run)¶

If engine-setup itself fails because its client cert is expired, temporarily move the engine host's clock to a date when the certs were still valid, do the renewal, then restore time.

systemctl stop chronyd
timedatectl set-ntp false
# pick a date a few weeks before the cert's notAfter
date -s "2026-05-01 10:00:00"

# ... perform the renewal (step 3) ...

# restore
timedatectl set-ntp true
systemctl start chronyd
chronyc makestep

Keep this window short. Don't start VMs or let hosts reconnect while the clock is wrong.

Step 3: re-issue the engine PKI¶

engine-setup --offline
# Answer YES to renewing certificates / PKI when prompted.

If engine-setup still won't proceed, the documented manual path for your version uses the PKI helper scripts:

/usr/share/ovirt-engine/bin/pki-enroll-request.sh ...   # version-specific
# or the "renew" mode:
names="engine apache websocket-proxy ..."   # per your version's docs

Follow the exact procedure in the oVirt docs for your major version — the script names and flags changed over time. After it completes, restart:

systemctl restart ovirt-engine httpd ovirt-websocket-proxy ovirt-imageio

Step 4: re-enroll every host¶

Hosts still have expired VDSM/libvirt certs. For each host:

1. Compute → Hosts → (host) → Management → Maintenance (if it'll go; if the host is unreachable you may need to force it).
2. Management → Enroll Certificate, or Installation → Reinstall (which redoes PKI fully and is more reliable after expiry).
3. Activate.

If a host won't communicate at all, on the host:

systemctl stop vdsmd
rm -rf /etc/pki/vdsm/certs/* /etc/pki/vdsm/keys/*
# then Reinstall from the engine, which pushes fresh certs

Running VMs on that host keep running through the VDSM restart (VDSM re-attaches to the existing libvirt domains).

Step 5: verify¶

for f in /etc/pki/ovirt-engine/certs/*.cer; do openssl x509 -in "$f" -noout -subject -enddate; done
vdsm-client Host getStats           # on each host
hosted-engine --vm-status           # if hosted-engine

All hosts Up, consoles working, no cert errors in engine.log.

The CA-expired worst case¶

If ca.pem itself is expired, re-issuing leaves off it doesn't help — nothing trusts the CA. You must renew the CA (extends its validity, keeps the same key) then re-issue all leaves and re-enroll all hosts. oVirt ships a documented CA-renewal procedure per version (around pki-* scripts / engine-setup CA renewal support in newer releases). This is a full maintenance window. Take a backup of /etc/pki/ovirt-engine and the engine DB (engine-backup) before touching the CA.

Verification and troubleshooting¶

• Host still Non Operational after reinstall — time skew (the #1 cause — chrony on engine and all hosts), or the host cached the old CA in /etc/pki/ca-trust. update-ca-trust and retry.
• engine-setup completes but portal still TLS-errors — browser cached the old cert / HSTS; try a private window. Or httpd didn't restart.
• Websocket console still broken — its cert/service missed the restart: systemctl restart ovirt-websocket-proxy.
• hosted-engine agent can't start — its broker cert; may need hosted-engine --deploy-style re-registration of the certs, per docs.
• You rolled the clock back and forgot to restore — hosts issue certs dated in the past, sanlock leases get confused. Fix time everywhere, chronyc makestep, restart vdsmd/sanlock.

Related tools and reading¶

• On-site: Certificate decoder, SSL Certificate Checker.
• Related posts: Renewing oVirt certificates before they expire, chrony vs ntpd.