feat: add circuit breaker for upstream provider overload protection

[kacpersaw]

Summary

Implement circuit breakers for upstream AI providers that detect rate limiting and overload conditions (429/503/529 status codes) and temporarily stop sending requests when providers are overloaded.

This completes the overload protection story by adding the aibridge-specific component that couldn't be implemented as generic HTTP middleware in coderd (since it requires understanding upstream provider responses).

Key Features

• Per-endpoint circuit breakers - Separate circuit breakers for each endpoint within each provider (e.g., /v1/messages and /v1/chat/completions are isolated), preventing one endpoint's issues from affecting others
• Configurable parameters - Failure threshold, interval for clearing counts, timeout before half-open, max requests in half-open state
• Customizable failure detection - Pluggable IsFailure function to determine which status codes trigger the circuit
• Built on sony/gobreaker - Uses the battle-tested github.com/sony/gobreaker/v2 library
• Prometheus metrics - State gauge, trips counter, rejects counter with provider/endpoint labels
• Thread-safe - Uses sync.Map for concurrent access to per-endpoint breakers
• Opt-in per provider - Configured via CircuitBreakerConfig in provider config; nil means no protection

Circuit Breaker States

┌─────────┐  failures >= threshold  ┌─────────┐
│ CLOSED  │ ──────────────────────► │  OPEN   │
│(normal) │                         │(reject) │
└─────────┘                         └────┬────┘
     ▲                                   │
     │                          cooldown expires
     │                                   │
     │  successes >= threshold     ┌─────▼─────┐
     └──────────────────────────── │ HALF-OPEN │
                                   │ (testing) │
     ┌─────────────────────────────┤           │
     │         any failure         └───────────┘
     │
     ▼
┌─────────┐
│  OPEN   │
│(reject) │
└─────────┘

Status Codes That Trigger Circuit Breaker (Default)

Code	Description
429	Too Many Requests
503	Service Unavailable
529	Anthropic Overloaded

Other error codes (400, 401, 500, 502, etc.) do not trigger the circuit breaker since they indicate different issues that circuit breaking wouldn't help with. Custom IsFailure functions can be provided to change this behavior.

Default Configuration

Parameter	Default	Description
FailureThreshold	5	Consecutive failures needed to trip circuit
Interval	10s	Cyclic period for clearing internal counts in closed state
Timeout	30s	How long circuit stays open before transitioning to half-open
MaxRequests	3	Requests allowed through in half-open state

Prometheus Metrics

Metric	Labels	Description
circuit_breaker_state	provider, endpoint	Current state (0=closed, 0.5=half-open, 1=open)
circuit_breaker_trips_total	provider, endpoint	Total times circuit has tripped open
circuit_breaker_rejects_total	provider, endpoint	Requests rejected due to open circuit

Files Changed

• circuit_breaker.go - Core implementation: CircuitBreakerConfig, ProviderCircuitBreakers, CircuitBreakerMiddleware
• circuit_breaker_test.go - Unit tests for middleware, failure detection, state transitions
• circuit_breaker_integration_test.go - End-to-end tests with mock upstream for Anthropic and OpenAI providers
• bridge.go - Wire up circuit breakers per provider with metrics and logging callbacks
• provider.go - Add CircuitBreakerConfig() method to Provider interface
• provider_anthropic.go - Implement CircuitBreakerConfig() for Anthropic
• provider_openai.go - Implement CircuitBreakerConfig() for OpenAI
• config.go - Add CircuitBreaker field to ProviderConfig
• metrics.go - Add Prometheus metrics for circuit breaker state, trips, and rejects

Usage

// Enable circuit breaker for a provider
provider := aibridge.NewOpenAIProvider(aibridge.OpenAIConfig{
    BaseURL: "https://api.openai.com/v1/",
    Key:     "sk-...",
    CircuitBreaker: &aibridge.CircuitBreakerConfig{
        FailureThreshold: 5,
        Interval:         10 * time.Second,
        Timeout:          30 * time.Second,
        MaxRequests:      3,
    },
})

// Or use defaults
provider := aibridge.NewAnthropicProvider(aibridge.AnthropicConfig{
    CircuitBreaker: func() *aibridge.CircuitBreakerConfig {
        cfg := aibridge.DefaultCircuitBreakerConfig()
        return &cfg
    }(),
}, nil)

Testing

All tests pass:

go test -count=1 -short ./...
go vet ./...
go build ./...

Related

• Issue: feat: add overload protections to aibridged internal#1153
• Rate limiting/concurrency PR: feat(coderd): add overload protection with rate limiting and concurrency control coder#21161

[dannykopping]

It's looking better now, but still a few things which need addressing please.

[dannykopping]

Not used, right?

[kacpersaw]

It may be used like:

cbConfig := aibridge.DefaultCircuitBreakerConfig()
providers := []aibridge.Provider{
	aibridge.NewOpenAIProvider(aibridge.OpenAIConfig{
		BaseURL:        "https://api.openai.com",
		Key:            "test-key",
		CircuitBreaker: &cbConfig,
	}),
}

[dannykopping]

How do you imagine this will be configured?

[kacpersaw]

circuitBreakerConfig = &aibridge.CircuitBreakerConfig{
	FailureThreshold: 5,
	Interval:         10 * time.Second,
	Timeout:          30 * time.Second,
	MaxRequests:      3,
	IsFailure:        DefaultIsFailure,
}

providers := []aibridge.Provider{
	aibridge.NewOpenAIProvider(aibridge.OpenAIConfig{
		BaseURL:        coderAPI.DeploymentValues.AI.BridgeConfig.OpenAI.BaseURL.String(),
		Key:            coderAPI.DeploymentValues.AI.BridgeConfig.OpenAI.Key.String(),
		CircuitBreaker: circuitBreakerConfig,
	}),
}

[ssncferreira]

Overall looks like a nice implementation and good test coverage 👍
Added a few comments and suggestions

[ssncferreira]

LGTM 👍 Thanks for addressing the comments!

[dannykopping]

Almost there, but there are a few things we need to address first 👍

[dannykopping]

LGTM, one note about marshaling the errors in a way that will be more maintainable, but I'm fine for that to be a follow-up.

Thanks for your patience on this one 👍 good work @kacpersaw

[dannykopping]

Each provider has its own error type which can be marshaled the way they expect to return errors. We should use that.
See intercept/messages/base.go -> newErrorResponse() for example.