============================================================= Defcore waiver for additional properties on Nova API response ============================================================= Launchpad blueprint: https://blueprints.launchpad.net/refstack/+spec/refstack-waiver Defcore waiver: https://review.openstack.org/#/c/333067/ Defcore committee approved above waiver which allows vendors who are using the Nova 2.0 API with additional properties to disable strict response checking when testing products for the OpenStack Powered program in 2016. This spec defines the changes needed for refstack-client to optionally bypass Tempest strict validation. Problem description =================== Vendors need an automated way to apply the waiver. The proposed method is to run Tempest from the RefStack client, identify tests that fail because of strict response checking, and rerun those tests with strict checking disabled. APIs and test cases using the waiver must be clearly identified. Proposed change =============== 1. Workflow - Vendor run Tempest suite as usual: via refstack-client test, ostestr, testr or with any other test runner. Some test cases failed due to additional properties in Nova response. - Vendor have the subunit test results file from the Tempest test execution. - Vendor have the Tempest configuration file. - Vendor rerun failed test cases by running ``refstack-client bypass-extras`` command. Command identifies failed test cases, disables Tempest strict validations, rerun test cases, and enables strict validations again. .. code-block:: bash $ refstack-client bypass-extras --subunit-file --conf-file - Output of bypass-extras command is a zip bundle containing the following files: - tests_list - List of failed test cases due to additional properties. - patched_schemas - List of tempest schemas which value was set to True (to allow additional properties). - api_details - API call details from each failed test case (due to additional properties). - rerun_test_results - The subunit result file for the re-run test cases. - combined_test_results.json - The Refstack JSON file with the combined passed TCs from both initial and rerun subunit files. 2. Implement "bypass-extras" Refstack command: Assume Tempest test suite was run independently. Subunit test results and Tempest configuration files are available. *bypass-extras* command is the helper tool for vendors to bypass the strict validation of additional properties in Tempest. Process steps and implementation details are explained on step 3. .. code-block:: bash $ refstack-client --help usage: refstack-client [-h] ... ... bypass-extras Apply Defcore waiver to identify additional properties on Nova API response. Re-runs failed test cases without Tempest strict response validations. $ refstack-client bypass-extras --help usage: refstack-client [-h] ... To see help on specific argument, do: refstack-client -h waiver [-h] [-s | -v] [-y] [--url URL] [-k] [-i PRIV_KEY] file optional arguments: -h, --help Show this help message and exit -s, --silent Suppress output except warnings and errors. -v, --verbose Show verbose output. -y Assume Yes to all prompt queries --subunit-file Path to subunit test result file. -c, --conf-file Path of the Tempest configuration file to use. 3. Flow for ``bypass-extras`` command. Having as input a subunit test results file and a Tempest configuration file: 3.1 Find failed test cases and its details Integrate code from find_additional_properties.py into Refstack-client to analyze subunit stream (from input results file). Find failed test cases due to additional properties in the response. Reconstruct the tempest schema causing the test case failure. Run subunit-describe-calls filter command to get test cases API call details. Input: subunit-results Output files: - tests_list - List of failed test cases due to additional properties. - patched_schemas - List of tempest schemas causing errors - api_details - API calls from each test case. 3.2 Patch Tempest: Create patch for .tempest virtual environment which lives under refstack-client installation. - Modify tempest/lib/api_schema/response/compute/v2_1/__init__.py: - Import module where schema lives. - Set schema addtionalProperties key to True so that additional properties are accepted - bypass strict validation. 3.3 Rerun failed test cases using patched refstack-client .tempest environment Use tests_list as withelist for ostestr in order to re-run failed test cases. .. code-block:: bash ostestr --serial -w test_list Input: test_list and conf-file files. Output: rerun_test_results subunit file 3.4 Remove Tempest patch Regardless of previous steps outcome, unpatching Tempest step will be attempted. Clean __init__.py by opening with access mode 'w' 3.5 Create refstack JSON format files Transform subunit result files - The one provided as input and the rerun test results - into a combined refstack JSON format. Input: initial_results_file, rerun_test_results files. Output: combined_test_results.json files. 3.6 Create zip bundle Alternatives ------------ - Add additional property to Tempest config file - For Tempest patch Comment the validate_response call by looking into the service_client.py file for the corresponding method (search through code files). - Manual process Products applying for the OpenStack Powered Trademark in 2016 may request the waiver by submitting subunit data from their Tempest run that can be manually analyzed by the `find_additional_properties.py` script from the DefCore repository. This script will identify tests that failed because of additional properties. The vendor will then need to manually modify tempest-lib to remove additional checks on the impacted APIs. Data model impact ----------------- None REST API impact --------------- None Security impact --------------- None Notifications impact -------------------- None Other end user impact --------------------- None Performance Impact ------------------ None Other deployer impact --------------------- None Developer impact ---------------- None Implementation ============== Assignee(s) ----------- Primary assignee: luz-cazares Other contributors: Chris Hoge Work Items ---------- - Add *bypass-extras* command to refstack-client. - Integrate find_additional_properties code - Method to call subunit-describe-calls filter - Implement JSON schema gathering from test exception - Implement Tempest patch - Implement Tempest unpatch - Create zip bundle with file results - Add *bypass-extras* command usage documentation Dependencies ============ None Testing ======= Add unit testing for the new command, verify expected outcomes are met. Documentation Impact ==================== Add refstack-client bypass-extras usage information under refstack-client/ README.rst References ========== None