Interoperability Guideline Testing

Testing against 2017.01 until present Capabilities

Tempest can be run standalone, or under a test runner such as refstack-client. We recommend running under refstack-client.

The test names of the capabilities are derived from a recent release of Tempest, from the time of capability approval. Keep in mind that Tempest is under active development, and tests may move. If you’re not seeing full coverage, please consider reverting back to a Tempest SHA that more closely matches the capability release date. The git SHA of Tempest that was known to be working at the time the Guideline was approved is listed in the Guideline JSON document itself (just search for “git-sha”). Please contact RefStack team at <openstack-discuss@lists.openstack.org> for assistance if needed. Use [interop] and/or [refstack] tags in the subject of the email.

It’s important to run a recent version of Tempest, as some bugs might have been fixed just recently. refstack-client uses Tempest from a specific commit (see setup_env script or README), which is intentionally older than master to avoid compatibility issues with older stable releases of OpenStack.

Network provisioning have been fixed. Some tests are still flagged due to outstanding bugs in the Tempest library, particularly tests that require SSH. We are working on correcting these bugs upstream. Please note that although some tests are flagged because of bugs, there is still an expectation that the capabilities covered by the tests are available.

In addition to testing required capabilities, we are also interested in collecting data on which API tests are being passed by production clouds. This information will be very useful in determining which capabilities will be used to define future releases. For that reason, we ask that you run all tests rather than just the required subset when submitting results to the OpenStack Foundation.

It is important to note that you MUST NOT modify the Tempest tests in any way. Modifying the tests means that Capability being tested is validated in a different way on your cloud than it is on other clouds, which voids any guarantee of interoperability. If you’re having problems passing all required tests, please contact RefStack team at <openstack-discuss@lists.openstack.org> for assistance or consider filing a request to have the tests flagged.

Please refer to HACKING for information on valid reasons to flag a test and how to file a flag request. Results from modified tests cannot be accepted as valid for trademark licensing purposes.