Added support for https in expvar scraping

[cmetz100]

What does this PR do?

Adds support for https requests for scraping expvar. Currently I am allowing any certificate with no tls verification. This is most definitely a bad practice in general. For this specific use case it may be fine. After testing this I should be able to pass in the pem path used for tls validation if we dont want to allow any certificate.

Motivation

Error seen in this notebook

Update as of Jan 22:
The change to set trace agents debug endpoint to https has been reverted for now so this change is not needed yet.

Update as of Jan 29:
The change has landed. These errors will be present from now. I need to update expvar endpoint for quality gates in agent repo and finish this change and get a lading release out

Testing:
job_id:03321fcd-6cdb-45ee-b07d-0ca48e4848d0

• comparison image is nightly from jan 10 (was hitting api limit when trying to pull todays, jan 30)
• This version has the trace https change that was reapplied here
  -quality_gate_idle has expvar configured for https
  -quality_gate_all_features has expvar configured for http
  -quality_gate_logs has no expvar configured
  I expect to see the following:
• comparison + quality_gate_idle => no errors (logs)
• baseline + quality_gate_idle => cant get expvar endpoint since we are sending request to https port and 7.58 will expose over http (logs)
• comparison + quality_gate_all_features => Http request sent to an https server error (logs)
• baseline + quality_gate_all_features => no errors (logs)

SMPTNG-587

[GeorgeHahn]

The mentioned PR DataDog/datadog-agent#32355 looks like it configures expvar to not use HTTPS, am I missing other context?

[cmetz100]

The mentioned PR DataDog/datadog-agent#32355 looks like it configures expvar to not use HTTPS, am I missing other context?

This line I believe configures the expvar endpoint to be https. It turns out that the mentioned PR is most likely the root cause of the two other issues I have investigated in these two notebooks: 1 2. I also ssm'd into a machine to verify that expvars are only accessible over https not http

So in summary the next agent release will require us to configure the expvar path in lading.yaml to use https instead of http. However the previous releases all expose over http. In order to support the https request we either need to grab the cert it uses or allow no verification which is what i currently got going. Ideally we could keep the config in lading.yaml at http and automatically upgrade the request to https if the client can discern it is needed but that seems difficult as of now

[GeorgeHahn]

@cmetz100 Ah, I follow now. I misunderstood the changes in that PR, thanks for the explanation.

[scottopell]

Worth a small changelog entry I think, but fine as-is, up to you

[cmetz100]

Worth a small changelog entry I think, but fine as-is, up to you

Agreed, I will add that now