Change default of expectedFailuresList parameter

[kstrenkova]

The expectedFailuresList parameter is used to define which tests are expected to fail so that they are not taken into consideration when marking the job's status. It is useful for specific scenarios, such as flaky tests, but we do not want every job to have this parameter by default. It should be used manually to make its intended use more explicit.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign abays for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/6551732e947b4f6faf05fa02a6c2e1eb

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 56m 50s
❌ podified-multinode-edpm-deployment-crc FAILURE in 1h 41m 10s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 30m 59s
✔️ cifmw-multinode-tempest SUCCESS in 1h 31m 22s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 8m 48s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 04s
✔️ build-push-container-cifmw-client SUCCESS in 17m 26s
✔️ cifmw-molecule-test_operator SUCCESS in 4m 24s

[frenzyfriday]

recheck

Unrelated failure
"Unhandled Error" err="k8s.io/client-go/tools/watch/informerwatcher.go:146: Failed to watch *unstructured.Unstructured: failed to list *unstructured.Unstructured: Get \"https://api.crc.testing:6443/apis/dataplane.openstack.org/v1beta1/namespaces/openstack/openstackdataplanedeployments?fieldSelector=metadata.name%3Dedpm-deployment&resourceVersion=100135\": dial tcp: lookup api.crc.testing on 199.204.44.24:53: no such host"

[softwarefactory-project-zuul]

Build failed (check pipeline). Post recheck (without leading slash)
to rerun all jobs. Make sure the failure cause has been resolved before
you rerun jobs.

https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/c2a07375d7934a7c8635d0a3aae39131

✔️ openstack-k8s-operators-content-provider SUCCESS in 1h 47m 52s
✔️ podified-multinode-edpm-deployment-crc SUCCESS in 1h 10m 37s
✔️ cifmw-crc-podified-edpm-baremetal SUCCESS in 1h 19m 48s
❌ cifmw-multinode-tempest FAILURE in 1h 30m 10s
✔️ noop SUCCESS in 0s
✔️ cifmw-pod-ansible-test SUCCESS in 9m 09s
✔️ cifmw-pod-pre-commit SUCCESS in 8m 14s
✔️ build-push-container-cifmw-client SUCCESS in 20m 59s
✔️ cifmw-molecule-test_operator SUCCESS in 4m 28s

[kstrenkova]

I think this output is not what we are looking for:
expectedFailuresList: | __omit_place_holder__860ed182641b95f03a71863857d53a312669d768
Maybe it is not possible to do through default(omit). The point was to not set expectedFailuresList automatically in jobs, so it should be omitted, but I am not sure how it could work with the | symbol.

[frenzyfriday]

I think this output is not what we are looking for: expectedFailuresList: | __omit_place_holder__860ed182641b95f03a71863857d53a312669d768 Maybe it is not possible to do through default(omit). The point was to not set expectedFailuresList automatically in jobs, so it should be omitted, but I am not sure how it could work with the | symbol.

Did you want something like this:

But I am wondering if jinja conditions work when defaults/main.yml is parsed. Apparently this passes the Yaml validators but I am not sure if ansible allows jinja conditions in defaults/main.yml

  kind: Tempest
  metadata:
    name: "{{ stage_vars_dict.cifmw_test_operator_tempest_name }}-{{ _stage_vars.name }}"
    namespace: "{{ cifmw_test_operator_namespace }}"
  spec:
    tempestRun:
      includeList: |
        {{ stage_vars_dict.cifmw_test_operator_tempest_include_list | default('') }}
        {%- if stage_vars_dict.cifmw_test_operator_tempest_expected_failures_list is defined -%}
      expectedFailuresList: |
        {{ stage_vars_dict.cifmw_test_operator_tempest_expected_failures_list  }}
        {%- endif -%}
      concurrency: "{{ cifmw_test_operator_concurrency }}"

[danpawlik]

@kstrenkova try @frenzyfriday solution, or tune it a littlebit if that will not pass:

  kind: Tempest
  metadata:
    name: "{{ stage_vars_dict.cifmw_test_operator_tempest_name }}-{{ _stage_vars.name }}"
    namespace: "{{ cifmw_test_operator_namespace }}"
  spec:
    tempestRun:
      includeList: |
        {{ stage_vars_dict.cifmw_test_operator_tempest_include_list | default('') }}
      {%- if stage_vars_dict.cifmw_test_operator_tempest_expected_failures_list is defined %}
      expectedFailuresList: |
        {{ stage_vars_dict.cifmw_test_operator_tempest_expected_failures_list  }}
      {%- endif %}
      concurrency: "{{ cifmw_test_operator_concurrency }}"