:title: letsencrypt

.. _letsencrypt:

Let's Encrypt Certificates
##########################

We support provisioning certificates from https://letsencrypt.org for
hosts in the ``opendev.org`` namespace.

At a Glance
===========

:Ansible:
  * :git_file:`playbooks/letsencrypt.yaml`
  * :git_file:`playbooks/roles/letsencrypt-acme-sh-install`
  * :git_file:`playbooks/roles/letsencrypt-request-certs`
  * :git_file:`playbooks/roles/letsencrypt-install-txt-record`
  * :git_file:`playbooks/roles/letsecnrypt-create-certs`
:Resources:
  * https://letsencrypt.org
  * https://github.com/Neilpang/acme.sh
:Chat:
  * #opendev on OFTC

Overview
========

We support automatic provisioning of certificates from Let's Encrypt
to hosts in the ``opendev.org`` domain.

This is implemented in OpenDev via the roles driven from
:git_file:`playbooks/letsencrypt.yaml`.  The overall
actions implemented by the above roles are roughly:

* Hosts that want a certificate use the ``amce.sh`` tool to request it
  from the Let's Encrypt CA.

  Creation or renewal requests receive a TXT record authentication
  value that must be published to prove ownership of the domain.  We
  implement this by making the challenge-request hostname
  ``_acme-challenge.hostname.opendev.org`` a ``CNAME`` record to a
  special "signing domain" ``acme.opendev.org``.

  Note if valid certificates are present and they are not within the
  renewal period (which is most of the time) no further action is
  taken.

* The provided TXT record authentication values are installed and
  published to the ``acme.opendev.org`` domain via the OpenDev
  nameservers.

* The host can now finalise certificate creation.  Let's Encrypt
  checks ``_acme-chellenge.hostname.opendev.org``, which is a
  ``CNAME`` to ``acme.opendev.org``.  Let's Encrypt then enumerates
  the TXT records there, and once finding the required key will return
  the signed keys to the host, which saves them to disk.


Configuring a host to get certificates
======================================

A basic configuration consists of the following steps:

1. Ensure the host is matched by the ``letsencrypt`` group in
   :git_file:`inventory/groups.yaml`.
#. DNS entries for ``_acme-chellenge.hostname`` as a ``CNAME`` to
   ``opendev.org`` must be added and live in the ``opendev.org``
   `zone.db
   <https://opendev.org/opendev/zone-opendev.org/src/branch/master/zones/opendev.org/zone.db>`__
   file.  Follow the other examples to ensure other fields such as
   ``CAA`` records are set too.

   Take care to list `all` hostnames that you wish covered by the
   certificate (e.g. ``hostname01.opendev.org`` and
   ``hostname.opendev.org``)
#. Configure the certificates to be issued to the host.

   The roles look for certificate configuration in a
   ``letsencrypt_certs`` variable defined for each host.  This is
   usually done via specific host variables in
   ``inventory/service/host_vars/<hostname>.opendev.org.yaml``.  For a simple
   host that wants a single certificate to cover its numeric hostname
   and regular ``CNAME`` this would look like ::

     letsencrypt_certs:
       hostname01-opendev-org:
        - hostname01.opendev.org
        - hostname.opendev.org

   This will result in certificate material in
   ``/etc/letsencrypt-certs/hostname01.opendev.org/`` on the host.

   Note that the "certificate name" dictionary keys (just
   ``hostname01-opendev-org`` above) are essentially a free-form
   string, but are used in the next step.  Follow the naming
   conventions for similar hosts.

   For full details, including information on issuing multiple
   certificates for a single host, see
   :git_file:`playbooks/roles/letsencrypt-request-certs/README.rst`.
#. Define a handler for certificate creation and renewal actions.

   When the certificate is created or renewed, the
   ``letsencrypt-create-certs`` role calls a predefined handler so
   action can be taken.  This handler name is constructed by
   prepending ``letsencrypt updated`` to the certificate name above.
   Thus in this example it would be ::

     - name: letsencrypt updated hostname01-opendev-org
       ...

   Usually these handlers are defined centrally in
   :git_file:`playbooks/roles/letsencrypt-create-certs/handlers/main.yaml`
   and common tasks such as restarting Apache have pre-defined tasks
   available for easy import.

   You may choose to define the handler in another way, but it *must*
   exist (Ansible does not have a way to say "call this handler only
   if it exists", thus a missing handler will cause an Ansible error
   at runtime).

Debugging
=========

The Ansible run logs on ``bridge.opendev.org`` should be consulted if
the certificate material is not being created as expected.

Hosts will log their ``acme.sh`` output to
``/var/log/acme.sh/acme.sh.log``

The `G Suite Toolbox Dig <https://toolbox.googleapps.com/apps/dig/>`__
tool can be useful for checking DNS entries from a remote location.

Refreshing keys
===============

In normal operation there should be no need to manually refresh keys
on hosts.  However there have been situations (such as LetsEncrypt
revoking certificates made during a certain period due to bugs) which
may necessitate a manual renewal.

The best way to do this is to move the ``.conf`` files from
``/etc/letsencrypt-certs/<certname>`` on the affected host and allow
the next Ansible pulse to renew.

.. code-block:: console

   # cd /etc/letsencrypt-certs/<name>
   # rename 's/.conf/.conf.old/' *.conf
   # tail -f /var/log/acme.sh/acme.sh.log
   ... watch and should be renewed on next pulse
   # rm *.conf.old

Linaro ARM64 Cloud Cert Renewal
===============================

The Linaro ARM64 cloud relies on Let's Encrypt certs for API endpoints,
but these certs are not automatically provisioned. The reason for this
is that cloud is not completely enrolled into our Ansible automation
(we share management of this install with Linaro and full integration
has not be done). We can manually refresh the SSL certs in this cloud
though.

To access the cloud backend ssh via bridge as root to
``openinfraci.linaro.cloud``.

First we provision a new certificate using acme.sh on the cloud node:

.. code-block:: console

  /root/acme.sh/acme.sh --server letsencrypt --issue \
    --dns dns_aws -d openinfraci.linaro.cloud

Next backup the old cert:

.. code-block:: console

  cp /root/us.linaro.cloud/secret/openinfraci.linaro.cloud.pem \
    /root/us.linaro.cloud/secret/openinfraci.linaro.cloud.pem.$DATE

Copy the new cert into the kolla-ansible secrets:

.. code-block:: console

  cat /root/.acme.sh/openinfraci.linaro.cloud/openinfraci.linaro.cloud.key \
    /root/.acme.sh/openinfraci.linaro.cloud/fullchain.cer \
    > /root/us.linaro.cloud/secret/openinfraci.linaro.cloud.pem

Activate the kolla-ansible virtualenv to run ansible:

.. code-block:: console

  source /root/venv3/bin/activate

Run kolla-ansible to deploy the cert:

.. code-block:: console

  /root/venv3/bin/kolla-ansible -i ~/all-in-one deploy