The OpenStack operator requires that you have a functional Kuberentes cluster in order to be able to deploy it. The following steps out-line the installation of a cluster and the operator.
The deployment of the OpenStack operator is highly containerised, even for the components not managed by the operator. The steps to get started involve deploying Docker for running the underlying infrastructure, installing a load balancer to access the Kubernetes API, deploying Kubernetes itself and then the operator which should start the OpenStack deployment.
Docker is used by many different components of the underlying infrastructure, so it must be installed before anything to bootstrap the system.
It must be installed on all the machines that you intend to manage using the OpenStack operator. It will also be used to deploy infrastructure components such as the virtual IP and Ceph.
Setup Virtual IP¶
The virtual IP runs across all controllers in order to allow the Kubernetes API server to be highly available and load balancer. It also becomes one of the interfaces that the Kubernetes ingress listens on where all the OpenStack API endpoints will be exposed.
The recommended way of deploying a virtual IP address is using
running inside Docker in order to make sure that your environment remains clean
and easily reproducible.
You should use the following command in order to start up
host the virtual IP address. These commands should be ran on all your
controllers and they assume that you have 3 controllers with IP addresses
10.0.0.3. The following example is what you
would run on the
10.0.0.1 machine with a VIP of
on the interface
$ docker run --cap-add=NET_ADMIN \ --cap-add=NET_BROADCAST \ --cap-add=NET_RAW \ --net=host \ --env KEEPALIVED_INTERFACE=eth0 \ --env KEEPALIVED_UNICAST_PEERS="#PYTHON2BASH:['10.0.0.2', '10.0.0.3']" \ --env KEEPALIVED_VIRTUAL_IPS="#PYTHON2BASH:['10.0.0.200']" \ --detach \ --restart always \ --name keepalived \ osixia/keepalived:2.0.20
You’ll have to make sure to edit the
environment variable accordingly depending on the host you’re running this
on. It should always point at the other hosts.
Setup Load Balancer¶
The load balancer which will be distributing requests across all of the Kubernetes API servers will be HAproxy.
We do not suggest using HAproxy to distribute load across all of the ingress controllers. The primary reason being that it introduces an extra hop in the network for no large benefit. The ingress should be bound directly on the virtual IP.
The following example assumes that you have 3 controllers, with their IP
10.0.0.3. It also assumes that
all of the Kubernetes API servers will be listening on port
16443 and it
will be listening on port
You’ll have to create a configuration file on the local system first:
$ mkdir /etc/haproxy $ cat <<EOF | tee /etc/haproxy/haproxy.cfg listen kubernetes mode tcp bind 0.0.0.0:6443 timeout connect 30s timeout client 4h timeout server 4h server ctl1 10.0.0.1:16443 check server ctl2 10.0.0.2:16443 check server ctl3 10.0.0.3:16443 check EOF
Once you’ve setup the configuration file, you can start up the containerized instance of HAproxy:
$ docker run --net=host \ --volume=/etc/haproxy:/usr/local/etc/haproxy:ro \ --detach \ --restart always \ --name=haproxy \ haproxy:2.2
You’ll also need to make sure that you have a DNS record pointing towards your virtual IP address. It is also recommended that you create a wildcard DNS as well to allow multiple hosts for the ingress without needing extra changes in your DNS, something like this:
cloud.vexxhost.net. 86400 IN A 10.0.0.200 *.cloud.vexxhost.net 86400 IN A 10.0.0.200
The recommended container runtime for the operator is
containerd, it is
also what is used in production. This document outlines the installation of
kubeadm. You’ll need to start by installing the
Kubernetes components on all of the systems.
Once this is done, you’ll need to start off by preparing the configuration file
kubeadm, which should look somethig like this:
$ cat <<EOF | tee /etc/kubernetes/kubeadm.conf --- apiVersion: kubeadm.k8s.io/v1beta2 kind: InitConfiguration localAPIEndpoint: bindPort: 16443 nodeRegistration: criSocket: /run/containerd/containerd.sock --- apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterConfiguration controlPlaneEndpoint: "cloud.vexxhost.net:6443" apiServer: extraArgs: oidc-issuer-url: https://accounts.google.com oidc-username-claim: email oidc-client-id: 1075333334902-iqnm5nbme0c36eir9gub5m62e6pbkbqe.apps.googleusercontent.com networking: podSubnet: 10.244.0.0/16 EOF
cloud.vexxhost.net should be replaced by the DNS address that you
created in the previous step.
The options inside
extraArgs are there to allow for OIDC authentication
via Google Suite. You should remove those or replace them with your own
The pod subnet listed there is the one recommended for usage with Calico, which is the supported and tested CNI.
At this point, you should be ready to start and bring up your first control plane node, you can execute the following on any of the controllers:
$ kubeadm init --config /etc/kubernetes/kubeadm.conf --upload-certs
At that point, the cluster will be up and it’s best to add the
cluster-admin credentials into the
root user for future management:
$ mkdir -p $HOME/.kube $ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config $ sudo chown $(id -u):$(id -g) $HOME/.kube/config
For all of the commands running on other nodes, you’ll need to make sure
that you include the following flag or it will use Docker instead of the
You will also need to join the other controllers to this cluster by using the
command provided which includes the
--control-plane flag. You’ll also need
to make sure you add the
--apiserver-bind-port 16443 flag or otherwise it
will refuse to join (due to port 6443 being used by the load balancer).
Once that is done, you can proceed to the joining the remainder of the nodes
kubeadm join command that was provided when initializing the
After you’ve completed the installation of the Kubernetes on all of the node,
you can verify that all nodes are present. It’s normal for nodes to be in the
NotReady status due to the fact that the CNI is not present yet:
$ kubectl get nodes NAME STATUS ROLES AGE VERSION ctl1 NotReady master 17m v1.18.6 ctl2 NotReady master 6m27s v1.18.6 ctl3 NotReady master 5m29s v1.18.6 kvm1 NotReady <none> 18s v1.18.6 kvm2 NotReady <none> 10s v1.18.6 kvm3 NotReady <none> 2s v1.18.6
The tested and supported CNI for the OpenStack operator is Calico due to it’s
high performance and support for
NetworkPolicy. You can deploy it onto
the cluster by running the following:
$ iptables -I DOCKER-USER -j ACCEPT $ kubectl apply -f https://docs.opendev.org/vexxhost/openstack-operator/calico.yaml
The first command overrides Docker’s behaviour of disabling all traffic routing if it is enabled, as this is necessary for the functioning on the Kubernetes cluster.
Once the CNI is deployed, you’ll have to make sure that Calico detected the correct interface to build your BGP mesh, you can run this command and make sure that all systems are on the right network:
$ kubectl describe nodes | grep IPv4Address
If they are not on the right IP range or interface, you can run the following
command and edit the
$ kubectl -n kube-system edit ds/calico-node
You’ll need to add an environment variable to the container definition which skips the interfaces you don’t want, something similar to this:
- name: IP_AUTODETECTION_METHOD value: skip-interface=bond0