Skip to content

Run ate-api-server in HA mode #538

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Haven Xia (HavenXia) merged 5 commits into from
Jul 28, 2026
Merged

Run ate-api-server in HA mode #538

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
da80e98
Prefactoring: Consolidate logic to port-forward to a service Pod.
juli4n Jul 25, 2026
f1456e2
Prefactoring: Delete `agent-secret` demo.
juli4n Jul 25, 2026
fd1b5c3
Prefactoring: Remove in-process state from mock volume plugin.
juli4n Jul 25, 2026
fa63c16
Prefactoring: Load-balance ate-api-server clients across replicas.
juli4n Jul 25, 2026
f89daf7
Run `ate-api-server` with 2 replicas.
juli4n Jul 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .ko.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ defaultPlatforms:

baseImageOverrides:
github.com/agent-substrate/substrate/demos/sandbox: alpine
github.com/agent-substrate/substrate/demos/agent-secret: alpine
# ateom-microvm needs glibc (for the fetched cloud-hypervisor binary) and mount/umount
# (to bind the image into the virtiofsd shared dir) — both in debian:stable-slim but
# not in the distroless static default.
Expand Down
3 changes: 1 addition & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ This demo highlights the core developer experience and "Agentic Infrastructure"
2. **State Persistence:** Persistent working memory (volatile RAM) and filesystem state preserved perfectly across hibernation cycles via full-state snapshots.
3. **Agent Swarm Multiplexing:** Demonstrates 30x+ oversubscription by "juggling" a large registry of stateful actors onto a small pool of shared physical pods.

To reproduce this demo in your own cluster, please refer to the detailed walkthroughs in the **[Counter Demo](demos/counter/README.md)** and **[Secret Agent Demo](demos/agent-secret/README.md)**.
To reproduce this demo in your own cluster, please refer to the detailed walkthrough in the **[Counter Demo](demos/counter/README.md)**.

For more videos and walkthroughs, visit our YouTube channel: **[agent-substrate](https://www.youtube.com/channel/UCN9PPqlTtVxlcpbQ-NWpfZQ)**.

Expand Down Expand Up @@ -199,7 +199,6 @@ We provide several sample applications demonstrating Agent Substrate's capabilit
1. **[Counter Demo](demos/counter/README.md)**: A stateful Go HTTP server demonstrating state preservation across suspends/resumes, and dynamic CRD routing.
2. **[Sandbox Demo (Antigravity)](demos/sandbox/README.md)**: A secure, sandboxed execution environment (running Alpine Linux) that allows arbitrary shell execution while preserving filesystem state across sessions.
3. **[Claude Code Multiplex](demos/claude-code-multiplex/README.md)**: Demonstrates oversubscribing physical hardware by multiplexing multiple Claude Code agents onto a limited pool of workers.
4. **[Secret Agent](demos/agent-secret/README.md)**: Highlights Substrate's "Zero-Idle" self-suspension and re-animation of volatile process memory.

### Documentation & Guides
* [API Configuration Guide](docs/api-guide.md): Detailed reference for configuring WorkerPools, ActorTemplates, Secrets, and Volumes.
Expand Down
14 changes: 14 additions & 0 deletions cmd/ateapi/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ import (
"golang.org/x/oauth2/google"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials"
"google.golang.org/grpc/keepalive"
"google.golang.org/grpc/reflection"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/rest"
Expand Down Expand Up @@ -195,6 +196,19 @@ func main() {
mux := grpc.NewServer(
grpc.Creds(serverCreds),
grpc.StatsHandler(otelgrpc.NewServerHandler()),
// Bounds every connection's lifetime so round_robin clients
// periodically re-resolve DNS and pick up replicas added since they
// last connected - without this, an existing connection never
// notices new replicas on its own (see https://github.com/grpc/grpc/issues/12295).
//
// TODO: Replace with a resolver that watches Endpoints/EndpointSlices
// directly and pushes address updates, instead of relying on forced
// reconnects to trigger DNS re-resolution. See
// https://github.com/sercand/kuberesolver.
grpc.KeepaliveParams(keepalive.ServerParameters{

@michaelasp Michael Aspinwall (michaelasp) Jul 27, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit concerning to me, effectively we are trading off session stability(1m keepalive) in order to obtain the endpoints. I see in the referenced issue that people get periodic unavailable errors with this approach. From the aforementioned issue:

The Go client has this resolver thing. You can import the package and Register a resolver builder or whatever. The client automatically picks it up, so I guess it's a global.

My coworker hacked something together to watch endpoints in the Kubernetes API and we have perfect load balancing when pods go in an out of service.

I think this is fine for now as we iterate, but seems like there's a way to pass endpoints into the resolver. We can probably make this a follow up bug, I don't mind hacking on this if wanted :)

@juli4n Julian Gutierrez Oschmann (juli4n) Jul 27, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we will eventually end up implementing our own resolver that watches endpoints. Let me add a TODO comment.

MaxConnectionAge: 1 * time.Minute,
MaxConnectionAgeGrace: maxRPCDeadline + time.Minute,
}),
grpc.ChainUnaryInterceptor(
ateapiauth.UnaryServerInterceptor(authCfg),
ateinterceptors.MaxDeadlineUnaryInterceptor(maxRPCDeadline),
Expand Down
2 changes: 1 addition & 1 deletion cmd/atenet/internal/router/cmd.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ func NewRouterCmd() *cobra.Command {
cmd.Flags().BoolVar(&cfg.Standalone, "standalone", false, "Run in standalone mode, bypassing creation of managed deployment and services in Kubernetes cluster")
cmd.Flags().StringVar(&cfg.Namespace, "namespace", "default", "Target operations namespace")
cmd.Flags().StringVar(&cfg.Kubeconfig, "kubeconfig", "", "Absolute path to the kubeconfig configuration file")
cmd.Flags().StringVar(&cfg.AteapiAddr, "ateapi-address", "api.ate-system.svc:443", "gRPC host address of the cluster ateapi Control instance")
cmd.Flags().StringVar(&cfg.AteapiAddr, "ateapi-address", "dns:///api.ate-system.svc:443", "gRPC dial target for the cluster ateapi Control instance.")
cmd.Flags().IntVar(&cfg.HttpPort, "port-http", 8080, "TCP port for workload traffic entering through the Envoy Router")
cmd.Flags().IntVar(&cfg.XdsPort, "port-xds", 18000, "TCP port listening for the xDS dynamic Envoy connections")
cmd.Flags().IntVar(&cfg.ExtprocPort, "port-extproc", 50051, "Listen port for the Envoy dynamic External Processing (ext_proc) server")
Expand Down
2 changes: 1 addition & 1 deletion cmd/benchmarking/boomer-glutton/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ import (

func main() {
var (
apiEndpoint = flag.String("api-endpoint", "api.ate-system.svc.cluster.local:443", "ateapi gRPC endpoint host:port.")
apiEndpoint = flag.String("api-endpoint", "dns:///api.ate-system.svc.cluster.local:443", "ateapi gRPC dial target.")
routerURL = flag.String("router-url", "http://atenet-router.ate-system.svc.cluster.local", "atenet HTTP router base URL (no trailing slash).")
atespace = flag.String("atespace", "benchmark", "Atespace every actor this worker creates lives in. Ensured (CreateAtespace, AlreadyExists is ok) at startup.")
promAddr = flag.String("prometheus-addr", ":8001", "Address for the Prometheus /metrics endpoint.")
Expand Down
92 changes: 0 additions & 92 deletions demos/agent-secret/README.md
View file
Open in desktop

This file was deleted.

76 changes: 0 additions & 76 deletions demos/agent-secret/agent-secret.yaml.tmpl
View file
Open in desktop

This file was deleted.

125 changes: 0 additions & 125 deletions demos/agent-secret/main.go
View file
Open in desktop

This file was deleted.

1 change: 0 additions & 1 deletion hack/install-ate.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,6 @@ ATE_DEMOS=()
source "${ROOT}"/hack/install-demo-counter.sh
source "${ROOT}"/hack/install-demo-sandbox.sh
source "${ROOT}"/hack/install-demo-claude-code-multiplex.sh
source "${ROOT}"/hack/install-demo-agent-secret.sh
source "${ROOT}"/hack/install-demo-multi-template.sh

# ANSI color codes for prettier output
Expand Down
Loading
Loading