Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[tmpnet] Add check for collection of logs and metrics to custom github action #3740

Open
wants to merge 4 commits into
base: tmpnet-managed-collectors
Choose a base branch
from

Conversation

maru-ava
Copy link
Contributor

@maru-ava maru-ava commented Feb 24, 2025

Why this should be merged

Adds checks to the run-monitored-tmpnet-cmd custom github action that verifies that logs were collected to loki and metrics were collected to prometheus for the job. A failing check will fail the job, ensuring that collection of logs and metrics doesn't silently fail to occur. This should also make reuse easier since any configuration issue will be more obvious.

How this works

  • add tmnetctl check-metrics to enable querying prometheus for metrics labeled for a github run
  • add tmnetctl check-logs to enable querying loki for logs labeled for a github run
  • call the new checks configured with GH_* env vars for a job at the end of run-monitored-tmpnet-cmd

How this was tested

  • Manually verified that tmpnetctl check-metrics passes for a network UUID for which metrics exist
  • Manually verified that tmpnetctl check-metrics fails for a non-existent network UUID
  • Checked that jobs for this PR pass and have metrics associated with them in grafana

Need to be documented in RELEASES.md?

N/A

TODO

@maru-ava maru-ava added the ci This focuses on changes to the CI process label Feb 24, 2025
@maru-ava maru-ava self-assigned this Feb 24, 2025
@maru-ava maru-ava force-pushed the tmpnet-check-metrics-collection branch 3 times, most recently from ab202c2 to fd27d90 Compare February 24, 2025 14:08
@maru-ava maru-ava force-pushed the tmpnet-check-metrics-collection branch 3 times, most recently from 3ec85e1 to 2e6404a Compare February 24, 2025 15:28
@maru-ava maru-ava force-pushed the tmpnet-managed-collectors branch from 91b64a7 to 4dce5a1 Compare February 24, 2025 15:34
@maru-ava maru-ava force-pushed the tmpnet-check-metrics-collection branch 5 times, most recently from 453b332 to e4243fc Compare February 24, 2025 17:20
@maru-ava maru-ava changed the title [tmpnet] Add check for metrics collection to monitoring action [tmpnet] Add check for collection of logs and metrics to custom github action Feb 24, 2025
@@ -388,6 +388,8 @@ github.com/jessevdk/go-flags v0.0.0-20141203071132-1679536dcc89/go.mod h1:4FA24M
github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY=
github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
github.com/jpillora/backoff v1.0.0 h1:uvFg412JmmHBHw7iwprIxkPMI+sGQ4kzOWsMeHnm2EA=
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is go.sum changed with no corresponding change?

Copy link
Contributor Author

@maru-ava maru-ava Feb 24, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's because of my use of the prometheus client. I didn't add any deps but the go mod job failed and this was the result of running 'go mod tidy'.

@maru-ava maru-ava force-pushed the tmpnet-managed-collectors branch from 4dce5a1 to 1a5222c Compare February 24, 2025 21:33
@maru-ava maru-ava force-pushed the tmpnet-check-metrics-collection branch 2 times, most recently from 424f7dc to 198b68f Compare February 24, 2025 22:31
@maru-ava maru-ava force-pushed the tmpnet-check-metrics-collection branch from 198b68f to 049efef Compare February 24, 2025 23:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci This focuses on changes to the CI process
Projects
Status: In Review 👀
Development

Successfully merging this pull request may close these issues.

2 participants