Make log-based metrics available with diagnostics tooling

[daniyelnnr]

What is the purpose of this pull request?

This PR introduces a modern metrics tooling layer using @vtex/diagnostics-nodejs (OpenTelemetry based) to replace the legacy MetricsAccumulator system. It provides a high-level API for recording metrics while maintaining full backward compatibility with existing code.

Key additions:

• New DiagnosticsMetrics class with simplified API (recordLatency, incrementCounter, setGauge)
• Attribute cardinality limiting (max 5 attributes) to prevent metrics explosion
• Migration of internal HTTP client, HTTP handler, and GraphQL metrics to new system
• Comprehensive test suite with minimal mocking approach (all tests passing)

Migration status:

• ✅ HTTP Agent statistics (gauges)
• ✅ Incoming request tracking (counters)
• ✅ HTTP Client metrics (latency + counters with attributes)
• ✅ HTTP Handler metrics (latency + counters with attributes)
• ✅ GraphQL field metrics (@metric directive)

Both legacy and new systems run in parallel - no breaking changes for external apps.

What problem is this solving?

Current limitations:

• Legacy MetricsAccumulator uses in-memory aggregation with snapshot-and-clear semantics
• Metrics exported via console.log requiring log parsing
• Client-side percentile calculation is memory-intensive
• No cardinality control leads to potential metrics explosion
• Dynamic metric names instead of attribute-based approach

Solution:

• OpenTelemetry-native metrics with proper SDK handling of aggregation
• Attribute-based metrics following OTel best practices (e.g., http_client_requests_total with status attribute)
• Built-in cardinality protection (5-attribute limit with warnings)
• Shared histogram (io_app_operation_duration_milliseconds) for all latencies
• Component differentiation via component attribute (http-client, http-handler, graphql)

How should this be manually tested?

Note: This cannot be fully tested locally as it requires the VTEX IO runtime in cloud environment.

Testing approach:

1. Deploy to test cluster first
2. Verify metrics exported to observability backend
3. Validate metric names match spec:
   • Histogram: io_app_operation_duration_milliseconds
   • Counters: http_client_requests_total, http_handler_requests_total, graphql_field_requests_total
   • Gauges: http_agent_sockets_current, http_server_requests_total
4. Check attributes are present: component, status, status_code, route_id, client_metric, cache_state
5. Verify legacy metrics still work (backward compatibility)
6. Test with external IO App (e.g., render-server) to ensure no breaking changes
7. Monitor for console warnings when global.diagnosticsMetrics unavailable

Validation checklist:

• Metrics appear in observability backend
• Attribute cardinality stays within limits
• Legacy MetricsAccumulator still functions
• No errors in service logs
• Performance overhead acceptable

Screenshots or example usage

New API usage:

// Recording latency (primary use case)
const start = process.hrtime()
// ... do work ...
global.diagnosticsMetrics.recordLatency(process.hrtime(start), {
  component: 'http-client',
  client_metric: 'checkout-api',
  status: 'success',
  cache_state: 'hit'
})

// Incrementing counters
global.diagnosticsMetrics.incrementCounter('http_client_requests_total', 1, {
  component: 'http-client',
  status: 'success',
  status_code: 200
})

// Setting gauge values
global.diagnosticsMetrics.setGauge('http_agent_sockets_current', 42, {})

Metric structure:

- io_app_operation_duration_milliseconds{component="http-client",client_metric="pages-api",status="success",cache_state="hit"}
- http_client_requests_total{component="http-client",status="success",status_code=200}
- http_handler_requests_total{component="http-handler",route_id="render",status="success",status_code=200}
- graphql_field_requests_total{component="graphql",field_name="products",status="success"}

Types of changes

• Bug fix (a non-breaking change which fixes an issue)
• New feature (a non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Requires change to documentation, which has been updated accordingly.

[arturpimentel]

Phew!! That was a long one 🤣

I admit that at first I tried to read everything, but by the 6th commit onwards I started to skim the test files. So don't count only on me in this one 😬

From what I gathered, this is thoroughly tested, which is awesome. The implementation files look good too. Nicely done!

[arturpimentel]

Out of curiosity: what happens if this initialization times out? Do we try again later?

[daniyelnnr]

In this case, if the initialization times out, there's no retry.
This rationale was chosen to align with what's already being done in the logs scope—but we can rethink this, no problem. If the timeout occurs, the metricsClient remains undefined, and this is reported in the console, allowing the application to continue normally without crashing. This approach allows telemetry failures to not impact availability, allowing requests to still be served by the application.

[arturpimentel]

But that also means that if telemetry is not available at startup, then we would need to restart the affected pods to re-enable it once it's back, right? I would consider a retry mechanism (maybe with some backoff) so we can ease our life when this situation happens.

[arturpimentel]

BTW this can also be a FUP

[arturpimentel]

Should those warnings be observed (not just being printed to stdout)?

[daniyelnnr]

I'm really not sure if this should be addressed; we ended up running into a problem of emitting metrics about missing metrics - I confess I'm not sure if this scope should be covered now. One way would be to treat this scenario as a follow-up and keep the behavior via stdout warnings for now. What do you think?

[arturpimentel]

Ok, that's good enough for me!