Using Let’s Encrypt Certificates

Starting IAM appliance 1.2, we offer built-in support for HTTPS certificates issued by Let’s Encrypt, including their automatic renewal. Let’s Encrypt ("LE") certificates are provided to anyone, free of charge.

We support only a subset of functionality provided by LE. This is intentional.

Supported features

  • IAM appliance maintains only one server certificate.

  • One certificate may contain multiple SANs (aternative names), therefore one certificate can cover multiple domain names.

  • Support for domain-validated certificates using http-01 challenge on port 80/tcp.

    • DNS validation is not supported.

    • ALPN validation is not supported.

  • Wildcard certificates are not supported.

Requirements

  • Your IAM appliance deployment must be able to communicate with LE servers.

  • LE servers must be able to access your IAM appliance’s HTTP port.

  • All your domains must be resolvable in DNS and records must point to correct server.

  • IAM appliance version 1.2 or later.

Deploying LE certificate

Deploying the certificate support consists of:

  1. Installing IAM appliance support package.

  2. Backing up your current certificates.

  3. Restarting IAM appliance services.

  4. Issuing new certificates and deploying them.

  5. Setting up automated renewal.

Installing IAM appliance support package

Installing the support package is very simple, but you have to have working EPEL repository. There was an IAM appliance bug related to inaccessible EPEL/Everything which may affect you. See known issues.

To install support package, run:

[root@localhost ~]# dnf install iam-letsencrypt-support

Backing up your current certificates

As a next step, back up your current TLS certificates.

[root@localhost ~]# mkdir crt-backup
[root@localhost ~]# chmod 700 crt-backup
[root@localhost ~]# cp -v /data/volumes/web-proxy/secrets/current_web_cert.* crt-backup/
'/data/volumes/web-proxy/secrets/current_web_cert.crt' -> 'crt-backup/current_web_cert.crt'
'/data/volumes/web-proxy/secrets/current_web_cert.key' -> 'crt-backup/current_web_cert.key'

Restarting IAM appliance services

By deploying certificates issued by different certificate authority, the whole trust chain, presented by your server, changes. This means that you have to deploy new root CA certificate for appliance services to trust.

Installing iam-letsencrypt-support package already did that for you and deployed a file with ISRG Root X1 root certificate into the /data/volumes-shared/cacerts folder of your appliance. All you have to do now is to restart affected services.

[root@localhost ~]# systemctl restart iam-czechidm
[root@localhost ~]# systemctl restart iam-cas
(and possibly others, depending on your deployment)

Issuing new certificates and deploying them

Now you are ready to request your first certificate and deploy it to the iam-web-proxy service.

Issue the first certificate by running command similar to (see explanation below):

certbot certonly [--dry-run [--test-cert]] --webroot --webroot-path /data/volumes/web-proxy/letsencrypt/ -d iam-d.iamappliance.tld -d iam-d.iamappliance.tld

Explanation:

certbot certonly \ # Running certbot utility with 'certonly' plugin.
  --dry-run \      # Perform challenge validation, but do not write certificates. For testing purposes only.
  --test-cert      # Use testing LE CA, not a production one. For testing purposes only.
  --webroot \      # Use 'webroot' mode to perform domain validation. Do not touch this.
  --webroot-path /data/volumes/web-proxy/letsencrypt/ \    # Where to store domain validation challenges. Do not touch this.
  -d iam-d.iamappliance.tld \    # Domain for the certificate (first occurence of -d also sets the name for the certificate in certbot).
  -d iam-t.iamappliance.tld      # Another domain for the certificate (you can specify many more by repeating the -d parameter).

For example, if we have three domains iam-d.appliance.tld, iam-t.appliance.tld and iam-p.appliance.tld, our command will look like this:

certbot certonly --webroot --webroot-path /data/volumes/web-proxy/letsencrypt/ -d iam-d.appliance.tld -d iam-t.appliance.tld -d iam-p.appliance.tld

The certbot will guide you through the account creation in the Let’s Encrypt authority. Specify an email address that you control. LE sometimes sends important notifications there.

When the certificate is successfully issued, certbot prints out path to it like this:

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/iam-d.appliance.tld/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/iam-d.appliance.tld/privkey.pem

Path to the directory containing certificate, private key, etc. is called lineage. In our example case, the lineage directory is /etc/letsencrypt/live/iam-d.appliance.tld. The name is specific to your deployment, so write it down. We need this information for completing next step.

Now, you have to actually deploy new certificate. Export following shell variable and point it to your lineage directory, like this:

[root@localhost ~]# export RENEWED_LINEAGE=/etc/letsencrypt/live/iam-d.appliance.tld

Then, utilize renewal hook to deploy the certificate:

[root@localhost ~]# /etc/letsencrypt/renewal-hooks/deploy/001_web-proxy-deploy.sh

You can find a log from this operation in the /var/log/letsencrypt/letsencrypt.log, which is the same file as certbot uses.

Instead of calling deployment hook by hand, you can also misuse certbot renew --force-renewal command. As you already obtained a certificate, subsequent invocation of certbot now renews it and, in effect, certbot will also run all configured deployment hooks.

This comes with a price, however. Let’s Encrypt API has limits on number of certificates and such, and you can hit this limit when re-issuing new certificates too frequently.

Lastly, check that your appliance works as expected.

  • Check the appliance web UI and that it serves new certificate and chain correctly.

  • Log-in to appliance services to check that their intercommunication works correctly. All issues should be resolved by restarting the failing service.

Setting up automated renewal

LE certificates have validity of 90 days and they have to be periodically renewed. The certbot command takes care of this and IAM appliance contains a scheduled task to make this happen automatically. All certificate parameters and settings are stored inside certbot (/etc/letsencrypt/) and are reused upon renewal.

Simply enable the scheduled task.

[root@localhost ~]# systemctl enable --now iam-letsencrypt-renew.timer
Created symlink /etc/systemd/system/timers.target.wants/iam-letsencrypt-renew.timer → /usr/lib/systemd/system/iam-letsencrypt-renew.timer.

Changing LE certificate SAN domains

In case you need to add or remove a domain from the certificate, you can do so by invoking the certbot command again. You have to explicitly set certificate name and new list of domain names.

First, find out the name of your certificate inside certbot:

[root@localhost ~]# certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: iam-d.iamappliance.tld
    Serial Number: faf2b63ab66bda2760375cd69c4da9cad823
    Key Type: RSA
    Domains: iam-d.iamappliance.tld iam-t.iamappliance.tld iam-p.iamappliance.tld
    Expiry Date: 2022-10-12 12:31:30+00:00
    Certificate Path: /etc/letsencrypt/live/iam-d.iamappliance.tld/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/iam-d.iamappliance.tld/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Then, request new certificate with updated domain list (we leave out the iam-p.iamappliance.tld domain as we do not want it anymore):

certbot certonly --webroot --webroot-path /data/volumes/web-proxy/letsencrypt/ --cert-name iam-d.appliance.tld -d iam-d.appliance.tld -d iam-t.appliance.tld

Requesting updated certificates with certbot automatically deploys them.

If you do not want them to be deployed right away, edit the /etc/sysconfig/iam-letsencrypt file and set the CERTBOT_WEB_PROXY_DEPLOY_ENABLED to false. Afterwards you have to deploy them manually by:

  1. Setting CERTBOT_WEB_PROXY_DEPLOY_ENABLED to true again.

  2. Exporting the lineage variable export RENEWED_LINEAGE=/etc/letsencrypt/live/$your_certificate_name.

  3. Invoking the deploy hook by hand /etc/letsencrypt/renewal-hooks/deploy/001_web-proxy-deploy.sh.

You can also look into the deploy hook script to find out what is going on and perform all the steps by hand.

Other maintenance tasks

IAM appliance uses out-of-box certbot binary and integrates it by means of hooks and wrapper units. If you need to do anything with your certificates or LE account, you can do it safely, as long as you use certbot command and while you stay within features supported by the appliance. For more information on what is possible, check Certbot documentation.

Removing LE certificate support from the appliance

If you want to remove LE certificates and return back to ones issued by hand, perform these steps.

  1. Disable LE certificate renewal with systemctl disable --now iam-letsencrypt-renew.timer .

  2. Deploy your own HTTPS certificate using this howto.

  3. (Optionally) deactivate your LE account by invoking certbot unregister .

    • Do not perform this step if you use your LE account across multiple machines.

  4. Uninstall support packages with dnf remove iam-letsencrypt-support certbot .

  5. Clean up any .rpmsave files and /etc/letsencrypt and /var/log/letsencrypt directories as necessary.