fix: add Etag field to request body to let Etag work

[mainred]

What type of PR is this?

/kind bug

What this PR does / why we need it:

This PR fill vmss and vmss vm Etag into request body before sending the put request, also this PR invalidated the vmss cache after updating the vmss to try to use the latest vmss mode with latest Etag before putting vmss. This logic has already been achieved in put vmss vm operations.

Tests have been done:
to let the EtagMiatch Error happen, I manually changed the code to let the first vmss update put with Etag "1", and the second one will pick the correct Etag, the same case will succeed. For the case of VMSS vm, I will let the put vmss vm fail by half by chaing the number on the first bulk put, and the second bulk puts succeed.
Reference: put vmss vmwith etagmimatch error mainred@5ef7fe9

• kusto query from put vmss vm EtagMismatch

• kusto query from put vmss EtagMismatch

I also tested add internal lb and delete internal lb for both cases above. For a detailed kusto query you may check

https://msazure.visualstudio.com/CloudNativeCompute/_wiki/wikis/personalplayground/746780/EtagMismatch-Error

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[k8s-ci-robot]

Hi @mainred. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[feiskyer]

/ok-to-test

[feiskyer]

LGTM for VMSS related changes, let's run some e2e tests to validate

[feiskyer]

@MartinForReal we'd also need to check whether other resources need similar ETAG fix (that could be done separately)

[mainred]

fixing the UT

[coveralls]

coverage: 76.62% (+0.02%) from 76.603%
when pulling 500b0af on mainred:crp-etag-track2-fix
into ec40d31 on kubernetes-sigs:master.

[MartinForReal]

superseded by #8117

[mainred]

@MartinForReal
I am not sure if you understand respecting the other's work.
I have a PR to add crp etag support in track2, you superseded my PR by saying my PR requires test.
and you superseded this PR by adding etag in putvm, but invalidating vmss vm cache is not covered in your PR but is essential.

I hope you can have a better test coverage of the your code change, since I am testing on my code heavily, but you superseded them.

secondly, the most prioritized work is to cut new cloud provider azure releases, which means your put vmss vm pr can be hold after your test credential provider track2 SDK,

cc @feiskyer

[feiskyer]

#8117 is not related to this change, it is addressing a different problem of vmss batch operations.

[feiskyer]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: feiskyer, mainred

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [feiskyer]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[feiskyer]

/kind bug

[MartinForReal]

@mainred please also update ensureBackendPoolDeletedFromNode
I'm not sure whether your test result is reproducible and related.
Could you please update the pr and get new test result?

[MartinForReal]

/hold

[MartinForReal]

/unhold

[mainred]

/test pull-cloud-provider-azure-e2e-ccm-capz

[mainred]

Could you please update the pr and get new test result?

Updated the PR description with the test scenarios and test result.
By the same, the test scenarios and test result has been put internal teams change after my test. I guess you did not see it.

[mainred]

Follow up:
need to add Etag in ensureBackendPoolDeletedFromNode, and add UT
I lost the Etag change in ensureBackendPoolDeletedFromNode because I forgot the copy the change from test branch