[Prometheus Input]: Expose metricbeat setting to disable SSL verification

[keiransteele-phocas]

Integration Name

Prometheus Input [prometheus_input]

Dataset Name

No response

Integration Version

0.7.1

Agent Version

8.17.0

OS Version and Architecture

Ubuntu Docker Container

User Goal

I'm testing out the Elastic Agent container as a sidecar to Keycloak deployed in AWS ECS Fargate. The Keycloak container has a /metrics endpoint with Prometheus metrics and I'm attempting to use the Prometheus Input integration to scrape these metrics.

Existing Features

Keycloak is configured to use https which forces the metrics endpoint to also be https, the SSL cert is self-signed. When configured with the https endpoint for metrics, the Prometheus Input integration is unable to connect. The documentation points to the metricbeat ssl configuration options but these are not available in the integration and as such I'm not able to disable SSL verification.

What did you see?

{
  "_index": ".ds-metrics-keycloak.metrics-default-2025.03.14-000001",
  "_id": "kIuskpUBHkgAVCzvLkaH",
  "_version": 1,
  "_source": {
    "agent": {
      "name": "ip-.us-west-2.compute.internal",
      "id": "5b0d0d90-7f24-4d10-b663-f4bbc7340d32",
      "type": "metricbeat",
      "ephemeral_id": "e360dc02-16b5-4889-b1ae-dfe4766f8c50",
      "version": "8.17.0"
    },
    "@timestamp": "2025-03-14T03:21:52.254Z",
    "ecs": {
      "version": "8.0.0"
    },
    "service": {
      "address": "https://localhost:9000/metrics",
      "type": "prometheus"
    },
    "data_stream": {
      "namespace": "default",
      "type": "metrics",
      "dataset": "keycloak.metrics"
    },
    "elastic_agent": {
      "id": "5b0d0d90-7f24-4d10-b663-f4bbc7340d32",
      "version": "8.17.0",
      "snapshot": false
    },
    "host": {
      "hostname": "ip-.us-west-2.compute.internal",
      "os": {
        "kernel": "5.10.234-225.895.amzn2.x86_64",
        "codename": "noble",
        "name": "Ubuntu",
        "family": "debian",
        "type": "linux",
        "version": "24.04.1 LTS (Noble Numbat)",
        "platform": "ubuntu"
      },
      "containerized": false,
      "ip": [
        "169.254.172.2",
        "fe80::30cf:5aff:fea9:d264",
        "",
        "fe80::487:f2ff:fe9a:b73d"
      ],
      "name": "ip-.us-west-2.compute.internal",
      "mac": [
        "06-87-F2-9A-B7-3D",
        "0A-58-A9-FE-AC-02"
      ],
      "architecture": "x86_64"
    },
    "metricset": {
      "period": 10000,
      "name": "collector"
    },
    "event": {
      "duration": 2834076,
      "agent_id_status": "verified",
      "ingested": "2025-03-14T03:21:52Z",
      "module": "prometheus",
      "dataset": "keycloak.metrics"
    },
    "error": {
      "message": "unable to decode response from prometheus endpoint: error making http request: Get \"https://localhost:9000/metrics\": x509: certificate signed by unknown authority"
    }
  }
}

Anything else?

No response

[strawgate]

The prometheus input integration has ssl.certificate_authorities can you provide the self-signed SSL cert here? Alternatively I think you could add the cert to the cert store in the container too

[keiransteele-phocas]

Hi @strawgate. Unfortunately I don't believe either of these options are feasible. The certificate is baked into the Keycloak container when it's built and it changes each time, ssl.certificate_authorities is a static value and would need to be updated manually each time. The certificate and key parameters which accept paths would be better but they're not exposed from Metricbeat either. Adding the certificate to the keystore would likely require rebuilding the Elastic Agent container to include the cert, we would need to maintain a new version of the agent container whenever a new Keycloak container is also built.

[devamanv]

@keiransteele-phocas There'a config option named ssl.verification_mode here. Currently, it's only available as a setting in Prometheus integration and not in Prometheus Input Integration. I have raised a PR to add that along with other SSL config options. It should be able to give you the functionality you're looking for.