Defcore waiver for additional properties on Nova API response

Launchpad blueprint: https://blueprints.launchpad.net/refstack/+spec/refstack-waiver

Defcore waiver: https://review.openstack.org/#/c/333067/

Defcore committee approved above waiver which allows vendors who are using the Nova 2.0 API with additional properties to disable strict response checking when testing products for the OpenStack Powered program in 2016.

This spec defines the changes needed for refstack-client to optionally bypass Tempest strict validation.

Problem description

Vendors need an automated way to apply the waiver. The proposed method is to run Tempest from the RefStack client, identify tests that fail because of strict response checking, and rerun those tests with strict checking disabled.

APIs and test cases using the waiver must be clearly identified.

Proposed change

  1. Workflow

  • Vendor run Tempest suite as usual: via refstack-client test, ostestr, testr or with any other test runner. Some test cases failed due to additional properties in Nova response.

  • Vendor have the subunit test results file from the Tempest test execution.

  • Vendor have the Tempest configuration file.

  • Vendor rerun failed test cases by running refstack-client bypass-extras command. Command identifies failed test cases, disables Tempest strict validations, rerun test cases, and enables strict validations again.

$ refstack-client bypass-extras --subunit-file <results> --conf-file <tempest-conf-file>

  • Output of bypass-extras command is a zip bundle containing the following files:

    • tests_list - List of failed test cases due to additional properties.

    • patched_schemas - List of tempest schemas which value was set to True (to allow additional properties).

    • api_details - API call details from each failed test case (due to additional properties).

    • rerun_test_results - The subunit result file for the re-run test cases.

    • combined_test_results.json - The Refstack JSON file with the combined passed TCs from both initial and rerun subunit files.

2. Implement “bypass-extras” Refstack command: Assume Tempest test suite was run independently. Subunit test results and Tempest configuration files are available.

bypass-extras command is the helper tool for vendors to bypass the strict validation of additional properties in Tempest. Process steps and implementation details are explained on step 3.

$ refstack-client --help

usage: refstack-client [-h] <ARG> ...
...
    bypass-extras    Apply Defcore waiver to identify additional properties
                     on Nova API response. Re-runs failed test cases
                     without Tempest strict response validations.

$ refstack-client bypass-extras --help

usage: refstack-client [-h] <ARG> ...

To see help on specific argument, do:
refstack-client <ARG> -h waiver
   [-h] [-s | -v] [-y] [--url URL] [-k] [-i PRIV_KEY] file

optional arguments:
    -h, --help          Show this help message and exit
    -s, --silent        Suppress output except warnings and errors.
    -v, --verbose       Show verbose output.
    -y                  Assume Yes to all prompt queries
    --subunit-file      Path to subunit test result file.
    -c, --conf-file     Path of the Tempest configuration file to use.

  1. Flow for bypass-extras command.

Having as input a subunit test results file and a Tempest configuration file:

3.1 Find failed test cases and its details

Integrate code from find_additional_properties.py into Refstack-client to analyze subunit stream (from input results file). Find failed test cases due to additional properties in the response. Reconstruct the tempest schema causing the test case failure. Run subunit-describe-calls filter command to get test cases API call details.

Input: subunit-results

Output files:

  • tests_list - List of failed test cases due to additional properties.

  • patched_schemas - List of tempest schemas causing errors

  • api_details - API calls from each test case.

3.2 Patch Tempest:

Create patch for .tempest virtual environment which lives under refstack-client installation.

  • Modify tempest/lib/api_schema/response/compute/v2_1/__init__.py:

    • Import module where schema lives.

    • Set schema addtionalProperties key to True so that additional properties are accepted - bypass strict validation.

3.3 Rerun failed test cases using patched refstack-client .tempest environment

Use tests_list as withelist for ostestr in order to re-run failed test cases.

ostestr --serial -w test_list

Input: test_list and conf-file files.

Output: rerun_test_results subunit file

3.4 Remove Tempest patch

Regardless of previous steps outcome, unpatching Tempest step will be attempted.

Clean __init__.py by opening with access mode ‘w’

3.5 Create refstack JSON format files

Transform subunit result files - The one provided as input and the rerun test results - into a combined refstack JSON format.

Input: initial_results_file, rerun_test_results files.

Output: combined_test_results.json files.

3.6 Create zip bundle

Alternatives

  • Add additional property to Tempest config file

  • For Tempest patch Comment the validate_response call by looking into the service_client.py file for the corresponding method (search through code files).

  • Manual process

    Products applying for the OpenStack Powered Trademark in 2016 may request the waiver by submitting subunit data from their Tempest run that can be manually analyzed by the find_additional_properties.py script from the DefCore repository. This script will identify tests that failed because of additional properties. The vendor will then need to manually modify tempest-lib to remove additional checks on the impacted APIs.

Data model impact

None

REST API impact

None

Security impact

None

Notifications impact

None

Other end user impact

None

Performance Impact

None

Other deployer impact

None

Developer impact

None

Implementation

Assignee(s)

Primary assignee:

luz-cazares

Other contributors:

Chris Hoge

Work Items

  • Add bypass-extras command to refstack-client.

  • Integrate find_additional_properties code

  • Method to call subunit-describe-calls filter

  • Implement JSON schema gathering from test exception

  • Implement Tempest patch

  • Implement Tempest unpatch

  • Create zip bundle with file results

  • Add bypass-extras command usage documentation

Dependencies

None

Testing

Add unit testing for the new command, verify expected outcomes are met.

Documentation Impact

Add refstack-client bypass-extras usage information under refstack-client/ README.rst

References

None