Deployment Guide¶
The OpenStack operator requires that you have a functional Kuberentes cluster in order to be able to deploy it. The following steps out-line the installation of a cluster and the operator.
The deployment of the OpenStack operator is highly containerised, even for the components not managed by the operator. The steps to get started involve deploying Docker for running the underlying infrastructure, installing a load balancer to access the Kubernetes API, deploying Kubernetes itself and then the operator which should start the OpenStack deployment.
Install Docker¶
Docker is used by many different components of the underlying infrastructure, so it must be installed before anything to bootstrap the system.
It must be installed on all the machines that you intend to manage using the OpenStack operator. It will also be used to deploy infrastructure components such as the virtual IP and Ceph.
$ apt-get install -y apt-transport-https ca-certificates curl gnupg-agent software-properties-common
$ curl -fsSL https://download.docker.com/linux/debian/gpg | sudo apt-key add -
$ sudo add-apt-repository "deb https://download.docker.com/linux/debian $(lsb_release -cs) stable"
$ apt-get update
$ apt-get install -y docker-ce
$ apt-mark hold docker-ce
Setup Virtual IP¶
The virtual IP runs across all controllers in order to allow the Kubernetes API server to be highly available and load balancer. It also becomes one of the interfaces that the Kubernetes ingress listens on where all the OpenStack API endpoints will be exposed.
The recommended way of deploying a virtual IP address is using keepalived
running inside Docker in order to make sure that your environment remains clean
and easily reproducible.
You should use the following command in order to start up keepalived to
host the virtual IP address. These commands should be ran on all your
controllers and they assume that you have 3 controllers with IP addresses
10.0.0.1, 10.0.0.2, 10.0.0.3. The following example is what you
would run on the 10.0.0.1 machine with a VIP of 10.0.0.200 running
on the interface eth0:
$ docker run --cap-add=NET_ADMIN \
--cap-add=NET_BROADCAST \
--cap-add=NET_RAW \
--net=host \
--env KEEPALIVED_INTERFACE=eth0 \
--env KEEPALIVED_UNICAST_PEERS="#PYTHON2BASH:['10.0.0.2', '10.0.0.3']" \
--env KEEPALIVED_VIRTUAL_IPS="#PYTHON2BASH:['10.0.0.200']" \
--detach \
--restart always \
--name keepalived \
osixia/keepalived:2.0.20
Note
You’ll have to make sure to edit the KEEPALIVED_UNICAST_PEERS
environment variable accordingly depending on the host you’re running this
on. It should always point at the other hosts.
Setup Load Balancer¶
The load balancer which will be distributing requests across all of the Kubernetes API servers will be HAproxy.
Note
We do not suggest using HAproxy to distribute load across all of the ingress controllers. The primary reason being that it introduces an extra hop in the network for no large benefit. The ingress should be bound directly on the virtual IP.
The following example assumes that you have 3 controllers, with their IP
addresses being 10.0.0.1, 10.0.0.2, 10.0.0.3. It also assumes that
all of the Kubernetes API servers will be listening on port 16443 and it
will be listening on port 6443.
You’ll have to create a configuration file on the local system first:
$ mkdir /etc/haproxy
$ cat <<EOF | tee /etc/haproxy/haproxy.cfg
listen kubernetes
mode tcp
bind 0.0.0.0:6443
timeout connect 30s
timeout client 4h
timeout server 4h
server ctl1 10.0.0.1:16443 check
server ctl2 10.0.0.2:16443 check
server ctl3 10.0.0.3:16443 check
EOF
Once you’ve setup the configuration file, you can start up the containerized instance of HAproxy:
$ docker run --net=host \
--volume=/etc/haproxy:/usr/local/etc/haproxy:ro \
--detach \
--restart always \
--name=haproxy \
haproxy:2.2
You’ll also need to make sure that you have a DNS record pointing towards your virtual IP address. It is also recommended that you create a wildcard DNS as well to allow multiple hosts for the ingress without needing extra changes in your DNS, something like this:
cloud.vexxhost.net. 86400 IN A 10.0.0.200
*.cloud.vexxhost.net 86400 IN A 10.0.0.200
Install Kubernetes¶
The recommended container runtime for the operator is containerd, it is
also what is used in production. This document outlines the installation of
Kubernetes using kubeadm. You’ll need to start by installing the
Kubernetes components on all of the systems.
$ curl -fsSL https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
$ sudo add-apt-repository "deb https://apt.kubernetes.io/ kubernetes-xenial main"
$ apt-get update
$ apt-get install -y kubelet kubeadm kubectl
$ apt-mark hold containerd.io kubelet kubeadm kubectl
$ containerd config default > /etc/containerd/config.toml
$ systemctl restart containerd
Once this is done, you’ll need to start off by preparing the configuration file
for kubeadm, which should look somethig like this:
$ cat <<EOF | tee /etc/kubernetes/kubeadm.conf
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: InitConfiguration
localAPIEndpoint:
bindPort: 16443
nodeRegistration:
criSocket: /run/containerd/containerd.sock
---
apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
controlPlaneEndpoint: "cloud.vexxhost.net:6443"
apiServer:
extraArgs:
oidc-issuer-url: https://accounts.google.com
oidc-username-claim: email
oidc-client-id: 1075333334902-iqnm5nbme0c36eir9gub5m62e6pbkbqe.apps.googleusercontent.com
networking:
podSubnet: 10.244.0.0/16
EOF
Note
The cloud.vexxhost.net should be replaced by the DNS address that you
created in the previous step.
The options inside extraArgs are there to allow for OIDC authentication
via Google Suite. You should remove those or replace them with your own
OIDC provider.
The pod subnet listed there is the one recommended for usage with Calico, which is the supported and tested CNI.
At this point, you should be ready to start and bring up your first control plane node, you can execute the following on any of the controllers:
$ kubeadm init --config /etc/kubernetes/kubeadm.conf --upload-certs
At that point, the cluster will be up and it’s best to add the
cluster-admin credentials into the root user for future management:
$ mkdir -p $HOME/.kube
$ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
$ sudo chown $(id -u):$(id -g) $HOME/.kube/config
Warning
For all of the commands running on other nodes, you’ll need to make sure
that you include the following flag or it will use Docker instead of the
recommended containerd:
--cri-socket /run/containerd/containerd.sock
You will also need to join the other controllers to this cluster by using the
command provided which includes the --control-plane flag. You’ll also need
to make sure you add the --apiserver-bind-port 16443 flag or otherwise it
will refuse to join (due to port 6443 being used by the load balancer).
Once that is done, you can proceed to the joining the remainder of the nodes
using the kubeadm join command that was provided when initializing the
cluster.
After you’ve completed the installation of the Kubernetes on all of the node,
you can verify that all nodes are present. It’s normal for nodes to be in the
NotReady status due to the fact that the CNI is not present yet:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
ctl1 NotReady master 17m v1.18.6
ctl2 NotReady master 6m27s v1.18.6
ctl3 NotReady master 5m29s v1.18.6
kvm1 NotReady <none> 18s v1.18.6
kvm2 NotReady <none> 10s v1.18.6
kvm3 NotReady <none> 2s v1.18.6
Install CNI¶
The tested and supported CNI for the OpenStack operator is Calico due to it’s
high performance and support for NetworkPolicy. You can deploy it onto
the cluster by running the following:
$ iptables -I DOCKER-USER -j ACCEPT
$ kubectl apply -f https://docs.opendev.org/vexxhost/openstack-operator/calico.yaml
Note
The first command overrides Docker’s behaviour of disabling all traffic routing if it is enabled, as this is necessary for the functioning on the Kubernetes cluster.
Once the CNI is deployed, you’ll have to make sure that Calico detected the correct interface to build your BGP mesh, you can run this command and make sure that all systems are on the right network:
$ kubectl describe nodes | grep IPv4Address
If they are not on the right IP range or interface, you can run the following
command and edit the calico-node DaemonSet:
$ kubectl -n kube-system edit ds/calico-node
You’ll need to add an environment variable to the container definition which skips the interfaces you don’t want, something similar to this:
- name: IP_AUTODETECTION_METHOD
value: skip-interface=bond0