Serve metrics & pprof via CSIMetricsManager

[AndrewSirenko]

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change
/kind bug
/kind cleanup
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

/kind cleanup

What this PR does / why we need it:

We recently updated csi-lib-utils to serve additional registries via PR kubernetes-csi/csi-lib-utils#189 so that all sidecars can easily serve leader, work queue, process, and Go runtime metrics via Kubernetes legacymetrics.

This PR refactors the provisioner to

Tested:

See gist: https://gist.github.com/AndrewSirenko/2d3bcd1cbc47541cab4f4fae9561fec0

Built all sidecar, deployed alongside EBS CSI Driver v1.40. Saw new metrics when scraping each sidecar /metrics endpoint.

Also tested upgrade case: No errors when upgrading EBS CSI Driver via helm from revision with default sidecars to these custom sidecars.

Excercised dynamic provisioning, snapshot, resize, and modify workflows. No errors. Workqueue metrics increased.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

[k8s-ci-robot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: AndrewSirenko
Once this PR has been reviewed and has the lgtm label, please assign saad-ali for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment