[Discussion] Enable Contribution of Experimental Processing Extensions

[missBerg]

Description:
This Issue is for discussing how we can enable experimental data plane processing steps and control plane APIs

Relevant Links:
See notes from community meeting: https://docs.google.com/document/d/10e1sfsF-3G3Du5nBHGmLjXw5GVMqqCvFDqp_O65B0_w/edit?tab=t.0#heading=h.h7jtr1uxwvpi

cc @envoyproxy/ai-gateway-maintainers @envoyproxy/ai-gateway-assignable

[mathetake]

So I did a small refactoring of the Go packages before the v0.1.0rc release: #234. Based on that, the automatic model selection stuff proposed by @rootfs can live inside the experimental package https://github.com/envoyproxy/ai-gateway/blob/main/filterapi/x/x.go just like the custom router stuff (which requires the rebuild though).

I think the semantics caching (#30) will also fall into the same style of "calling out from the extproc"? cc @Krishanx92
Then let's start with the API design of "AI Gateway Semantics Processor" i guess ... which I believe will be a gRPC service

[rootfs]

Sounds good! I'll rebase to the latest code and create a PR.

[mathetake]

yeah the initial PR should be only focused on the API of the external gRPC sever - no other stuff needed yet !

[rootfs]

@mathetake it would be counterproductive to review an API protocol without a supporting PoC :D

[mathetake]

ok up to you - it would be helpful for reviews anyways!

[arkodg]

would be great to also see a PoC, which could be as simple as a kubectl patch cmd that enhances the generated EnvoyExtensionPolicy with a new extProc and documents this, and based off user needs, can be incorporated into a first class API

[rootfs]

@mathetake @arkodg #256 is updated. I also have a semantic cache PoC in the PR. You can see the demo here

[mathetake]

thanks @rootfs for the PR. unfortunately i still do not quite understand the value of the first class abstraction you are trying to introduce. Anything you demonstrated can be done by adding an additional extproc by yourself as @arkodg suggested above. Could you elaborate on what is not possible with the BYO extproc vs the first class API? in anyway I would love to see more users wanting to have it vs I don't think we can just approve it, merge and maintain given that all functionality can be achieved with BYO extproc

[rootfs]

@mathetake Adding BYO extproc will add latency in routing and API translation. If implemented as a BYO extproc, then each hop will add up additional network + compute + storage overhead, that won't scale well. Additionally, having multiple extproc will lose the optimization opportunities at the global level. For instance, when semantic cache is in place, it could be yet another backend in addition to upstream LLM target. Such integration will not be possible if individual BYO extproc are put together in isolation. Lastly, BYO extproc is a recipe to community divider not unifier.

[mathetake]

each hop will add up additional network + compute + storage overhea

wait, the additional network + compute etc is the same for your PoC isn't it?