Environment: Red Hat Virtualization 4.4
Issue: All the hosted-engine hosts vdsm certificates are expired. The hosted-engine VM is down and it’s unable to start it using hosted-engine --vm-start because of expired certificates.
Resolution: Since CA of the RHV environment is the RHV manager which is down, we have to get the CA certificates and keys to the host to renew the host certificates.
Option 1: Obtain CA keys and certificates from the engine-backup.
- All the PKI files of the engine are collected while taking
engine-backup. So if the environment has got recent engine-backup, the CA files can be obtained from this.# tar -xvf <engine-backup> # tar -xvf files
The etc/pki/ovirt-engine/ will be having the CA files. This directory has to be copied to the host for renewing the host certificates.
Option 2: Copy out certificates from the hosted-engine disk.
Copy the files out to /root/ of host using virt-copy-out:
Tell libguestfs not to use libvirt since the libvirtd service might be down because of expired certs.
# export LIBGUESTFS_BACKEND=direct
Copy out the files
Block based storage domain:
# virt-copy-out -a /dev/1bfb1005-b98d-4592-9c1d-5c04292584ed/d82c83a7-0957-469d-8e62-d537407c8993 /etc/pki/ovirt-engine/ /root/
username: vdsm@ovirt
password: shibboleth
File-based storage domain:
# virt-copy-out -a /rhev/data-center/SPUUID/1bfb1005-b98d-4592-9c1d-5c04292584ed/images/7ac7f073-e5bc-4f6a-b483-68a16fd9fe25/d82c83a7-0957-469d-8e62-d537407c8993 /etc/pki/ovirt-engine/ /root/
Or
# virt-copy-out -a /rhev/data-center/mnt/NFS_share/1bfb1005-b98d-4592-9c1d-5c04292584ed/images/7ac7f073-e5bc-4f6a-b483-68a16fd9fe25/d82c83a7-0957-469d-8e62-d537407c8993 /etc/pki/ovirt-engine/ /root/
Renew the host certificates.
- Create a CSR for the host using the key.
# cd /root/ovirt-engine/
#openssl req -new -key /etc/pki/vdsm/keys/vdsmkey.pem -out /tmp/01_vdsmkey.csr -passin "pass:mypass" -passout "pass:mypass" -batch -subj "/"
#openssl req -new -key /etc/pki/vdsm/keys/vdsmkey.pem -out /tmp/02_vdsmkey.csr -passin "pass:mypass" -passout "pass:mypass" -batch -subj "/"
#openssl req -new -key /etc/pki/vdsm/keys/vdsmkey.pem -out /tmp/03_vdsmkey.csr -passin "pass:mypass" -passout "pass:mypass" -batch -subj "/"
#openssl req -new -key /etc/pki/vdsm/keys/vdsmkey.pem -out /tmp/04_vdsmkey.csr -passin "pass:mypass" -passout "pass:mypass" -batch -subj "/"
- Find the subject of the old certificate.
# openssl x509 -in /etc/pki/vdsm/certs/vdsmcert.pem -noout -subject
- Sign the CSR using the engine CA. Make sure that the subj is replaced with the output from the command above.
# cd /root/ovirt-engine/
#openssl ca -batch -policy policy_match -config openssl.conf -cert /etc/pki/ovirt-engine/ca.pem -keyfile /etc/pki/ovirt-engine/private/ca.pem -days +3650 -in 01_vdsmkey.csr -out host01_vdsm.cer -startdate "$(date --utc --date "now -1 days" +"%y%m%d%H%M%SZ")" -subj "/O=sysadmin.lk/OU=IT/CN=server01.sysadmin.lk" -utf8
#openssl ca -batch -policy policy_match -config openssl.conf -cert /etc/pki/ovirt-engine/ca.pem -keyfile /etc/pki/ovirt-engine/private/ca.pem -days +3650 -in 02_vdsmkey.csr -out host02_vdsm.cer -startdate "$(date --utc --date "now -1 days" +"%y%m%d%H%M%SZ")" -subj "/O=sysadmin.lk/OU=IT/CN=server02.sysadmin.lk" -utf8
#openssl ca -batch -policy policy_match -config openssl.conf -cert /etc/pki/ovirt-engine/ca.pem -keyfile /etc/pki/ovirt-engine/private/ca.pem -days +3650 -in 03_vdsmkey.csr -out host03_vdsm.cer -startdate "$(date --utc --date "now -1 days" +"%y%m%d%H%M%SZ")" -subj "/O=sysadmin.lk/OU=IT/CN=server03.sysadmin.lk" -utf8
#openssl ca -batch -policy policy_match -config openssl.conf -cert /etc/pki/ovirt-engine/ca.pem -keyfile /etc/pki/ovirt-engine/private/ca.pem -days +3650 -in 04_vdsmkey.csr -out host04_vdsm.cer -startdate "$(date --utc --date "now -1 days" +"%y%m%d%H%M%SZ")" -subj "/O=sysadmin.lk/OU=IT/CN=server04.sysadmin.lk" -utf8
- Copy the signed certificate.
# tar cfJ /tmp/vdsm_pki.tar.xz /etc/pki/vdsm/
# cp /tmp/test_host_vdsm.cer /etc/pki/vdsm/certs/vdsmcert.pem
# cp /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-spice/server-cert.pem
# cp /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-vnc/server-cert.pem
# cp /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/vdsm/libvirt-migrate/server-cert.pem
# cp /etc/pki/vdsm/certs/vdsmcert.pem /etc/pki/libvirt/clientcert.pem
- Restart the libvirtd and vdsmd services.
# systemctl restart vdsmd
# systemctl restart libvirtd
- Wait for around 5 minutes and try to start the VM.
# hosted-engine --vm-start
