API Simplification of AIGW Proposals

[missBerg]

The purpose of this issue is to consolidate the conversations on the simplification on API Design into a single place and ensure we take action.

---

API Simplification

Scope: Resource renaming and consolidation

• API Simplification (Consider making AIServiceBackend an abstraction for Backends for AI Services #675): Strong community support for AIRoute/AIBackend consolidation
• Resource Complexity: Current hierarchy is too complex for good UX

Proposed Deliverables

• RFC for AIGatewayRoute → AIRoute & AIServiceBackend → AIBackend rename
• Plan to merge BackendSecurityPolicy into AIBackend
• Backward compatibility considerations
• Migration guidance

---

Summary of Recent Discussions from GitHub Issues

1. Resource Abstraction and Simplification (Issues #675, #764)

Current State:

• Complex resource hierarchy: GatewayClass -> Gateway -> AIGatewayRoute -> AIServiceBackend + n * Backend + n * BackendSecurityPolicy + n * BackendTLSPolicy
• Multiple separate CRDs requiring coordination
• Poor UX due to resource proliferation

Proposed Changes:

• Simplified hierarchy: GatewayClass -> Gateway -> AIRoute -> AIBackend
• Merge Backend, BackendSecurityPolicy, and BackendTLSPolicy into AIBackend
• Rename AIGatewayRoute to AIRoute and AIServiceBackend to AIBackend

Recent Community Feedback:

• @Xunzhuo: "+1 for this, would love to make API simplify" - highlights UX and frontend engineering concerns
• @arkodg: Suggests removing "Service" prefix as redundant
• @missBerg: Questions whether BackendSecurityPolicy should be promoted to Envoy Gateway upstream

Status: Active discussion with community poll proposed

2. Gateway API Extension Points (Issue #764)

Proposal:

• Replace custom CRDs with Gateway API extensions using:
  • Policy extensions
  • Filter extensions
  • BackendRef extensions
• Leverage Extension Server for translation

Arguments For:

• Better ecosystem integration
• Reuse existing Gateway API tools/workflows
• Easier adoption for existing Gateway API users

Arguments Against:

• Still requires separate CRD (AIProviderPolicy)
• Complex translation layer needed
• May not handle AI-specific requirements (priority, model names)

Recent Comments:

• @arkodg: "highlight the simplified interaction with EG extension usage"
• @yuzisun: "It is not as simple as a httproute reference anymore, we added the priority and model name to the backend refs"
• @mathetake: Add a priority to a backendRef kubernetes-sigs/gateway-api#1802 this is the route rule level priority issue in GWAPI upstream. Basically, it allows users to configure "priority depending on the model name" which is critical for egress fallback cases like us, and we have that priority at AIGatewayRoute's rule level. So, technically currently impossible to reuse HTTPRoute as-is.

---

[mathetake]

my 2c:

Due to the reason around the lack of HTTPRoute rule level priority in GWAPI upstream, I think the resource simplification seems be the way to go for now. While doing so, maybe working on adding more adaptive priority mechanism to envoyproxy/envoy (like adding an extension point to modify priority) should also happen IMO (but that requires relatively large amount of Envoy codebase task) to be able to allow AIGW API to be consumed solely as GWAPI extensions.

As for migration, we have two choices. 1. Simply rename and merge, breaking every existing users and rewrite CRD. This should be fine given that we are still v0.2 + API is v1alpha. We should get a consensus in the community meeting though. This comes with very few maintenance burden. 2. Introduce a brand new two CRD (AIRoute, AIBackend) into the existing API without modifying the existing ones while marking old ones deprecated. Document the migration path to the new one. This will result in very complex control plane code (i am just guessing), which I would rather want to avoid. Regardless of the path we choose, the new CRD (AIRoute, AIBackend) API can be discussed in a PR right now

[Xunzhuo]

+1 for option 1, we are still at early stage, so I think the API BREAKING CHANGES should be acceptable

[xiaolin593]

As early user and contributor, I can resonate problem statement and agree with Proposed Changes

Simplified hierarchy: GatewayClass -> Gateway -> AIRoute -> AIBackend
Merge Backend, BackendSecurityPolicy, and BackendTLSPolicy into AIBackend
Rename AIGatewayRoute to AIRoute and AIServiceBackend to AIBackend

To choices laid out in #784 (comment), in general I agree with option 1 but need to be mindful if there are already users use envoy-ai-gateway in production or closer to production . To be super safe, option 2 is better, not disruptive, and give users more time to migrate over.

[aabchoo]

@missBerg: Questions whether BackendSecurityPolicy should be promoted to Envoy Gateway upstream

+1 to promoting this upstream if it fits the timeline. AWS credentials may be shared with all aws backends which results in a lot of unnecessary repetition if backend is combined w/ bsp.

Simply rename and merge, breaking every existing users and rewrite CRD

If we do ultimately decide with this option, I hope that features would be back ported to pre-this-breaking change (up til a certain point) so users can benefit off new features while they migrate over.