Add --poll-snapshot-period for periodic readiness requeue check when creating a VolumeSnapshotContent

[pwschuurman]

What type of PR is this?
/kind feature

What this PR does / why we need it:
Today, the csi-snapshotter runs with a reconciler pattern when processing VolumeSnapshotContent resources. This follows the familiar requeue, err pattern that is followed by other controller frameworks, like controller-runtime.

• Error Handling: When an error is encountered, VolumeSnapshotContent resources will be requeued with exponential decay. This behavior is desired, as presumably RPC calls to a CSI driver are due to congestion, and the request rate should be reduced.
• Requeue Processing: In the case of dynamic provisioning, the creation of a VolumeSnapshotContent may require some time until a snapshot is ready in the underlying storage system. This is where the requeue field is useful, as the reconciler can periodically attempt to check the status of a snapshot.

The problem with the current requeue implementation is that it relies on the same exponential backoff rate limiter queue as the error handling scenario. This is a problem, as it can lead to non-deterministic ready times, and can lead to significantly higher effective snapshot ready latencies for snapshots that take longer to process (eg: larger snapshots).

To fix this, the requeue logic should use a constant retry period, to allow for more reliable snapshot reconciliation. This PR adds a new command line argument --poll-snapshot-period to the csi-snapshotter, with a default of 10 seconds.

Which issue(s) this PR fixes:

Fixes #1282

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Requeue VolumeSnapshotContent resources based on --poll-snapshot-period in csi-snapshotter

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: pwschuurman
Once this PR has been reviewed and has the lgtm label, please assign jsafrane for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment