Deployment Guide

The OpenStack operator requires that you have a functional Kuberentes cluster in order to be able to deploy it. The following steps out-line the installation of a cluster and the operator.

The deployment of the OpenStack operator is highly containerised, even for the components not managed by the operator. The steps to get started involve deploying Docker for running the underlying infrastructure, installing a load balancer to access the Kubernetes API, deploying Kubernetes itself and then the operator which should start the OpenStack deployment.

Install Docker

Docker is used by many different components of the underlying infrastructure, so it must be installed before anything to bootstrap the system.

It must be installed on all the machines that you intend to manage using the OpenStack operator. It will also be used to deploy infrastructure components such as the virtual IP and Ceph.

$ apt-get install -y apt-transport-https ca-certificates curl gnupg-agent software-properties-common
$ curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
$ sudo add-apt-repository "deb https://download.docker.com/linux/debian $(lsb_release -cs) stable"
$ apt-get update
$ apt-get install -y docker-ce
$ apt-mark hold docker-ce

Setup Virtual IP

The virtual IP runs across all controllers in order to allow the Kubernetes API server to be highly available and load balancer. It also becomes one of the interfaces that the Kubernetes ingress listens on where all the OpenStack API endpoints will be exposed.

The recommended way of deploying a virtual IP address is using keepalived running inside Docker in order to make sure that your environment remains clean and easily reproducible.

You should use the following command in order to start up keepalived to host the virtual IP address. These commands should be ran on all your controllers and they assume that you have 3 controllers with IP addresses 10.0.0.1, 10.0.0.2, 10.0.0.3. The following example is what you would run on the 10.0.0.1 machine with a VIP of 10.0.0.200 running on the interface eth0:

$ docker run --cap-add=NET_ADMIN \
             --cap-add=NET_BROADCAST \
             --cap-add=NET_RAW \
             --net=host \
             --env KEEPALIVED_INTERFACE=eth0 \
             --env KEEPALIVED_UNICAST_PEERS="#PYTHON2BASH:['10.0.0.2', '10.0.0.3']" \
             --env KEEPALIVED_VIRTUAL_IPS="#PYTHON2BASH:['10.0.0.200']" \
             --detach \
             --restart always \
             --name keepalived \
             osixia/keepalived:2.0.20

Note

You’ll have to make sure to edit the KEEPALIVED_UNICAST_PEERS environment variable accordingly depending on the host you’re running this on. It should always point at the other hosts.

Setup Load Balancer

The load balancer which will be distributing requests across all of the Kubernetes API servers will be HAproxy.

Note

We do not suggest using HAproxy to distribute load across all of the ingress controllers. The primary reason being that it introduces an extra hop in the network for no large benefit. The ingress should be bound directly on the virtual IP.

The following example assumes that you have 3 controllers, with their IP addresses being 10.0.0.1, 10.0.0.2, 10.0.0.3. It also assumes that all of the Kubernetes API servers will be listening on port 16443 and it will be listening on port 6443.

You’ll have to create a configuration file on the local system first:

$ mkdir /etc/haproxy
$ cat <<EOF | tee /etc/haproxy/haproxy.cfg
listen kubernetes
  mode tcp
  bind 0.0.0.0:6443
  timeout connect 30s
  timeout client 4h
  timeout server 4h
  server ctl1 10.0.0.1:16443 check
  server ctl2 10.0.0.2:16443 check
  server ctl3 10.0.0.3:16443 check
EOF

Once you’ve setup the configuration file, you can start up the containerized instance of HAproxy:

$ docker run --net=host \
             --volume=/etc/haproxy:/usr/local/etc/haproxy:ro \
             --detach \
             --restart always \
             --name=haproxy \
             haproxy:2.2

You’ll also need to make sure that you have a DNS record pointing towards your virtual IP address. It is also recommended that you create a wildcard DNS as well to allow multiple hosts for the ingress without needing extra changes in your DNS, something like this:

cloud.vexxhost.net.  86400   IN      A       10.0.0.200
*.cloud.vexxhost.net 86400   IN      A       10.0.0.200

Install Kubernetes

The recommended container runtime for the operator is containerd, it is also what is used in production. This document outlines the installation of Kubernetes using kubeadm. You’ll need to start by installing the Kubernetes components on all of the systems.

$ curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
$ sudo add-apt-repository "deb https://apt.kubernetes.io/ kubernetes-xenial main"
$ apt-get update
$ apt-get install -y kubelet kubeadm kubectl
$ apt-mark hold containerd.io kubelet kubeadm kubectl
$ containerd config default > /etc/containerd/config.toml
$ systemctl restart containerd

Once this is done, you’ll need to start off by preparing the configuration file for kubeadm, which should look somethig like this:

$ cat <<EOF | tee /etc/kubernetes/kubeadm.conf
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
  bindPort: 16443
nodeRegistration:
  criSocket: /run/containerd/containerd.sock
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
controlPlaneEndpoint: "cloud.vexxhost.net:6443"
apiServer:
  extraArgs:
    oidc-issuer-url: https://accounts.google.com
    oidc-username-claim: email
    oidc-client-id: 1075333334902-iqnm5nbme0c36eir9gub5m62e6pbkbqe.apps.googleusercontent.com
networking:
  podSubnet: 10.244.0.0/16
EOF

Note

The cloud.vexxhost.net should be replaced by the DNS address that you created in the previous step.

The options inside extraArgs are there to allow for OIDC authentication via Google Suite. You should remove those or replace them with your own OIDC provider.

The pod subnet listed there is the one recommended for usage with Calico, which is the supported and tested CNI.

At this point, you should be ready to start and bring up your first control plane node, you can execute the following on any of the controllers:

$ kubeadm init --config /etc/kubernetes/kubeadm.conf --upload-certs

At that point, the cluster will be up and it’s best to add the cluster-admin credentials into the root user for future management:

$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config

Warning

For all of the commands running on other nodes, you’ll need to make sure that you include the following flag or it will use Docker instead of the recommended containerd:

--cri-socket /run/containerd/containerd.sock

You will also need to join the other controllers to this cluster by using the command provided which includes the --control-plane flag. You’ll also need to make sure you add the --apiserver-bind-port 16443 flag or otherwise it will refuse to join (due to port 6443 being used by the load balancer).

Once that is done, you can proceed to the joining the remainder of the nodes using the kubeadm join command that was provided when initializing the cluster.

After you’ve completed the installation of the Kubernetes on all of the node, you can verify that all nodes are present. It’s normal for nodes to be in the NotReady status due to the fact that the CNI is not present yet:

$ kubectl get nodes
NAME   STATUS     ROLES    AGE     VERSION
ctl1   NotReady   master   17m     v1.18.6
ctl2   NotReady   master   6m27s   v1.18.6
ctl3   NotReady   master   5m29s   v1.18.6
kvm1   NotReady   <none>   18s     v1.18.6
kvm2   NotReady   <none>   10s     v1.18.6
kvm3   NotReady   <none>   2s      v1.18.6

Install CNI

The tested and supported CNI for the OpenStack operator is Calico due to it’s high performance and support for NetworkPolicy. You can deploy it onto the cluster by running the following:

$ iptables -I DOCKER-USER -j ACCEPT
$ kubectl apply -f https://docs.opendev.org/vexxhost/openstack-operator/calico.yaml

Note

The first command overrides Docker’s behaviour of disabling all traffic routing if it is enabled, as this is necessary for the functioning on the Kubernetes cluster.

Once the CNI is deployed, you’ll have to make sure that Calico detected the correct interface to build your BGP mesh, you can run this command and make sure that all systems are on the right network:

$ kubectl describe nodes | grep IPv4Address

If they are not on the right IP range or interface, you can run the following command and edit the calico-node DaemonSet:

$ kubectl -n kube-system edit ds/calico-node

You’ll need to add an environment variable to the container definition which skips the interfaces you don’t want, something similar to this:

- name: IP_AUTODETECTION_METHOD
  value: skip-interface=bond0