Copyright 2018 Red Hat This work is licensed under a Creative Commons Attribution 3.0 Unported License. http://creativecommons.org/licenses/by/3.0/legalcode
Post Zuul v3 Third Party CI¶
Our 3rd Party CI solutions have not quite kept up with the pace of Zuul V3 development. OpenStack Infra should provide 3rd Party CI operators with guidance on how to incoporate these new technologies.
There are many external operators who run external continuous-integration systems against projects hosted within the OpenStack Infra gerrit. These have traditionally been referred to as “Third Party CI”.
The reasons are as varied as the operators, but usually involve testing software or hardware components that are not suitable for inclusion in the upstream infrastructure testing environments.
After the acceptance of the “Common OpenStack CI Solution” spec (Common OpenStack CI Solution), effort was put into the puppet-openstackci repository. This acts as an abstraction layer between the OpenStack Infra specific configuration, kept in system-config, and the more specific puppet modules which install and manage indivdual services.
Operators have largely been following the instructions based in https://docs.openstack.org/infra/openstackci/ which promises, via the main entry-point of single_node_ci.pp, to provide a functional zuul and jenkins CI setup.
Unfortunately with the removal of Jenkins and the introduction of nodepool and zuul v3, these instructions have become out of date.
The OpenStack infrastructure team should provide guidance to 3rd Party CI operators about how to utilise this new environment.
Windmill is a
series of ansible roles for the deployment of nodepool, zuul, gerrit.
It uses a range of
openstack/ansible-role-* projects to achieve
this; it is roughly analogous to single_node_ci.pp
We will focus on creating a playbook(s) and related documentation that does an all-in-one setup that focuses on 3rd party CI specifics.
Gating across Fedora, Xenial (soon bionic) and possibility of other platforms too.
Roles are generic enough anyone can build out a zuul stack
Currently using well tested path of pypi and OS packages
About 90% done
Propper documentation to come
Would need further updates for dealing with containerisation of services
OpenStack Infra is not directly running these services in production, which leaves windows for issues to creep in.
To declare this spec as complete will require work across a combination of documentation, Windmill and related projects.
The target audience is defined as an experienced Linux administrator, with basic familiarity as a user of OpenStack clouds, Gerrit and Ansible.
3rd Party CI will require
A single (Ubuntu-based?) host to run Zuul, nodepool and nodepool-builder
Access to an OpenStack cloud
ability to upload images
resources to run at least one testing node
For success, they should be able to complete the following with a combination of documentation and using the provided tools:
Register an account for use with Gerrit
Have an environment for running Windmill.
Setup a virtualenv with ansible and windmill+dependencies
Containers to be added
Description of initially required playbook configuration
Deploy requirements such as Zookeeper and gearman
Use Windmill roles to create a single-node nodepool environment
nodepool-builder should be able to build a Xenial image
image uploaded to target cloud
node ready in nodepool
Have a Zuul merger and executor running
Start zuul scheduler and connect it to gerrit listening for changes on a single project of interest
Interact with zuul-web to see status
Trigger a single job on changes to monitored project
Be able to report results to upstream gerrit
The referenced 3rd Party CI documentation will need to be updated.
Highly dependent on solutions.
Initial spec reviews proposed a number of different alternatives. Whilst they are no mutually exclusive with Windmill, that will be our focus. The alternatives are presented below.
We could direct effort to updating the OpenStackCI puppet module described above to enable installation of zuulv3, nodepool and other requirements such as zookeeper.
This would take a significant amount of puppet expertise and involve
“detangling” some parts where the
system-config -> openstackci ->
puppet-* abstraction may have broken down slightly during
It (sort of) keeps the status-quo for people using puppet to deploy
We’re moving away from puppet, in general
We’ve moving away from using puppet to deploy nodepool/zuul in openstack infra, in specific
Who would actually do this?
Software Factory is an out-of-the-box open source solution that can be used for 3rd party CI.
can do 3rd party CI, but much more too
very active project
limited to CentOS deployments
Zuul provides a Zuul From Scratch document that describes how to configure a zuul and nodepool environment that can talk to gerrit. It does not provide automation such as puppet modules or ansible tasks.
We could show no preference to these deployment mechanisms and suggest people implement the broad installation instructions as they feel.
These documents would need to be enhanced to cover details on setting things up specifically for 3rd party on OpenStack’s Zuul instance.
Nothing to maintain
Leaves a lot of work in the hands of potential users
No possibility of CI/CD approach to ensure correctness
Something to deploy container images¶
Infra is moving to containerised services
As at September 2018, work in this area is just beginning with the pbrx container builds.
We could put 3rd Party CI direction setting on hold for a short period while this work bootstraps itself and the infra project gains some experience running containerised CI services. We could then offer this externally.
Would only target the container runtime, rather than a wider range of platforms of interest to 3rd parties.
Very new, so progress would likely come after infra have acquired better experience running containers.
Maintain modules, but no centralised driver¶
Specific modules such as
puppet-zookeeper and equivalent
ansible-role-* projects for
those that prefer ansible are provided. The canonical reference for
use of these modules is
system-config; which retains its status as
a non-generic “top-level” specific to OpenStack Infra deployment.
We could support individual deployment modules, however not attempt to maintain a generic driver for them.
This is effectively the current situation.
Unlikely anyone could reasonably create a working system out of this