Introduce support for sidecars in ECS

[CarmenAPuccio]

Description
ECS and Fargate support the concept of a sidecar for things like reverse proxies, logging, telemetry, etc... ECS also has a concept of container dependency in the task definition in order to control the startup for example as outlined here. While I realize this support does not exist today in Compose as outlined in compose-spec/compose-spec#65 I can easily imagine customers running on ECS would want this functionality.

Additional environment details (AWS ECS, Azure ACI, local, etc.):
AWS ECS

[ndeloof]

Compose-spec introduced support for depends_on: { condition: service_completed } which is an initial step toward init containers support, but the actual way to get them expressed within a compose application model is still to be discussed.

About user request for sidecars, my guess is that they would better adopt a declarative approach that let them opt-in for "service with telemetry" or "expose with reverse proxy", similar to istio on Kubernetes, vs having to explicitly configure a sidecar container, copy/pasting some complex container definition that they will then have to maintain.

About ECS support for dependencies and startup order, please note we are blocked by awslabs/goformation#61 to let user flag a service as Essential: false. Our own internal init container are using a terrible hack awaiting a better option.

[dalegaspi]

i need this in my life.

[ndeloof]

@dalegaspi could you please describe your use case, so we better understand how this can fit into the compose model, and how the compose file format could evolve to embrace this scenario?

[dalegaspi]

@ndeloof we have a service in ECS today deployed using docker-compose and it's basically a Spring Cloud Gateway working in conjunction with Envoy: the Envoy fronts the SCG and for all intents and purposes, they work as one and the Envoy works as a sidecar. Today, the docker-compose creates 2 task definitions with 2 services operating independently and the refer to each other "locally" through CloudMap. don't get me wrong, it works but there's unnecessary network hops...and the lack of explicit dependency of one service to the other is not much of a big deal to me. but as @CarmenAPuccio noted above...there are a couple of things that would make this better:

1. have some mechanism to indicate that one service is dependent on another (again, not a big deal for my use case but it's a nice to add feature)
2. (this in my mind is more important) the task definitions can have more than 1 containers (so essentially we are defining just 1 service)...the SCG and envoy containers will just launch in one service which makes it more efficient as there are less network hops, and the Envoy container acts like a true sidecar.

i have no strong opinion on how the above features should be implemented, but the more we adhere with the docker-compose way of doing things (as opposed to ECS extensions) would probably be the preferable option...probably.

[ndeloof]

thanks for the details.
so, basically, you're using a sidecar to add servicemesh feature to your deployment, which maybe could be supported using AWS AppMesh for better integration with the platform. For sure this could be expressed using explicit sidecar container definition, but you'll have to copy/paste this same container sidecar task definition for each service, and maintain it, while your actual intent is about higher-level routing/monitoring/resilience requirements that shoud be express as such, not by platform implementation details.

have some mechanism to indicate that one service is dependent
you can just use depends_on in your compose file

[dalegaspi]

yes there are service mesh features in use but we prefer portability and not be dependent on AWS-specific feature.

...while your actual intent is about higher-level routing/monitoring/resilience requirements that should be express as such

point taken...but i didn't really give details as to how the Envoy and SCG work together for us, but the "actual intent" a bit more than just routing/monitoring/resilience 😊...hence the decision to go with sidecar approach.

i'm confused i thought the depends_on doesn't work yet for docker-compose ECS? but good call, nonetheless, i will start using it.

[CarmenAPuccio]

A little more information here @ndeloof to think about. Dexter's use case is a common one and the way you implement AppMesh with an Envoy proxy is defined here. You are right that the actual intent in this case is about higher-level routing but unfortunately, the implementation details are key in this scenario as there is no mutating webhook admission controller concept in ECS/Fargate like there is in k8s so the user has to define it in their task definition for each and every service.

There are obviously many other examples where we need to inject a sidecar like in the case of logging with something like FluentBit. In this case, ECS/Fargate does make the implementation a little easier but the user is still left to define the "sidecar" in their task definition as seen here.

[ndeloof]

I totally agree there's technical challenges offering portability, but - typically - Compose model does define a logging section where I'd like I can opt-in for FluentBit or anything comparable, and not have to read AWS docs on the expected sidecar setup I need to apply for this to work.
Think about port mapping in plain docker: user says "container port 8080 must be exposed as 80", we don't as him to pass an iptable rule.
That's what I mean by "we need to capture the user intent in a higher-level construct within compose-spec".

[CarmenAPuccio]

Yep fair point... The idea is to make it easier so they don't have to go read the AWS docs I agree. With that said (and what got me going down this route honestly) is there are common scenarios where a user needs a sidecar (even without AWS specific details). For example, I was building a demo .NET app and trying to front the Kestrel web server with an NGINX reverse proxy as seen here.

[ndeloof]

sure, I'm not saying there's no use-case to require sidecars, just it seems to me most use-cases are actually implementation details for many modern infrastructure services (routing, security, monitoring, retry, short-circuit, logging, ...) which, from a user point of view, would better be expressed with higher-level constructs.

To use the "routing" use-case for illustration, istio or Traefik both allow use of annotations to define the routing. If we could specify such things in compose-spec, using X or Y implementation under the hood would become a platform detail.
most users.

@CarmenAPuccio about your .Net app sample: I agree a reverse proxy "does it better with HTTP" and most deployment will actually have such infrastructure components. This doesn't mean this should be explicitly modeled in your compose file. Typically, when deploying on AWS I'd probably opt for a platform provided reverse proxy service, the same way deploying to Kubernetes I'm setting ingress but never deploy an explicit nginx frontend container.

[mreferre]

More requests on the line of this: https://devops.stackexchange.com/questions/13802/docker-compose-to-ecs-two-services-in-one-task

[ndeloof]

@mreferre here again, I'd tend to ask "why do you need nginx as a (sidecar) frontend to your django service? Doesn't AWS offer something comparable directly implemented by the infrastructure (typically, using ALB rules to manage http->https redirect, 404, etc)
Or, to be more generic, "as a developer, why would I have to hard-code the HTTP infrastructure optimizations in my compose file". This should - worst scenario - be defined by an additional "infrastructure" file, but not tied to my service declaration.

[dalegaspi]

the reason i prefer to have this sidecar pattern option is so that i can view the two/more services as just one. less resources, less network hops. that should be a good enough reason, no?