Skip to content

fix: ECS Task needs memory and CPU overrides when invoked from Pipe #171

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 66 additions & 0 deletions examples/pipe-ecs-target/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# EventBridge Pipes ECS Target

Configuration in this directory creates EventBridge resource configuration including an ECS task target for a Pipe.

## Usage

To run this example you need to execute:

```bash
$ terraform init
$ terraform plan
$ terraform apply
```

Note that this example may create resources which cost money. Run `terraform destroy` when you don't need these resources.

<!-- BEGIN_TF_DOCS -->
## Requirements

| Name | Version |
|------|---------|
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 1.0 |
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 5.98 |
| <a name="requirement_random"></a> [random](#requirement\_random) | >= 3.0 |

## Providers

| Name | Version |
|------|---------|
| <a name="provider_aws"></a> [aws](#provider\_aws) | 5.100.0 |
| <a name="provider_random"></a> [random](#provider\_random) | 3.7.2 |

## Modules

| Name | Source | Version |
|------|--------|---------|
| <a name="module_ecs_cluster"></a> [ecs\_cluster](#module\_ecs\_cluster) | terraform-aws-modules/ecs/aws | ~> 5.0 |
| <a name="module_eventbridge"></a> [eventbridge](#module\_eventbridge) | ../../ | n/a |

## Resources

| Name | Type |
|------|------|
| [aws_iam_policy.eventbridge_pipes_ecs_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy) | resource |
| [aws_iam_role_policy_attachment.eventbridge_pipes_ecs_policy](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_role_policy_attachment) | resource |
| [aws_security_group_rule.default_egress_https](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/security_group_rule) | resource |
| [aws_sqs_queue.source](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue) | resource |
| [random_pet.this](https://registry.terraform.io/providers/hashicorp/random/latest/docs/resources/pet) | resource |
| [aws_caller_identity.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/caller_identity) | data source |
| [aws_region.current](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/region) | data source |
| [aws_security_group.default](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/security_group) | data source |
| [aws_subnets.default](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/subnets) | data source |
| [aws_vpc.default](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/vpc) | data source |

## Inputs

No inputs.

## Outputs

| Name | Description |
|------|-------------|
| <a name="output_eventbridge_bus_arn"></a> [eventbridge\_bus\_arn](#output\_eventbridge\_bus\_arn) | The EventBridge Bus ARN |
| <a name="output_eventbridge_rule_arns"></a> [eventbridge\_rule\_arns](#output\_eventbridge\_rule\_arns) | The EventBridge Rule ARNs |
| <a name="output_eventbridge_rule_ids"></a> [eventbridge\_rule\_ids](#output\_eventbridge\_rule\_ids) | The EventBridge Rule IDs |
<!-- END_TF_DOCS -->
196 changes: 196 additions & 0 deletions examples/pipe-ecs-target/main.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,196 @@
provider "aws" {
region = "eu-west-1"

# Make it faster by skipping something
skip_metadata_api_check = true
skip_region_validation = true
skip_credentials_validation = true
}

#############################################################
# Data sources to get VPC and default security group details
#############################################################
data "aws_vpc" "default" {
default = true
}

data "aws_security_group" "default" {
name = "default"
vpc_id = data.aws_vpc.default.id
}

data "aws_subnets" "default" {
filter {
name = "vpc-id"
values = [data.aws_vpc.default.id]
}
}

data "aws_caller_identity" "current" {}
data "aws_region" "current" {}

resource "aws_sqs_queue" "source" {
name = "${random_pet.this.id}-source"
}

####################
# Actual Eventbridge
####################
module "eventbridge" {
source = "../../"

# Schedules can only be created on default bus
create_bus = false

create_role = true
role_name = "ecs-eventbridge-${random_pet.this.id}"
attach_ecs_policy = true
ecs_target_arns = [
module.ecs_cluster.services["hello-world"].task_definition_arn,
"arn:aws:ecs:${data.aws_region.current.id}:${data.aws_caller_identity.current.account_id}:task/${random_pet.this.id}/*"
]

ecs_pass_role_resources = [module.ecs_cluster.services["hello-world"].task_exec_iam_role_arn]

pipes = {
test_ecs_pipe = {

attach_policies_for_integrations = true

source = aws_sqs_queue.source.arn
target = module.ecs_cluster.cluster_arn

attach_policies_for_integrations = true

target_parameters = {
ecs_task_parameters = {
assign_public_ip = "ENABLED"
task_count = 1
launch_type = "FARGATE"
task_definition_arn = module.ecs_cluster.services["hello-world"].task_definition_arn
container_name = "hello-world"

security_groups = [module.ecs_cluster.services["hello-world"].security_group_id]
subnets = data.aws_subnets.default.ids

memory = 512
cpu = 256

enable_ecs_managed_tags = true
}
}
}
}
}

resource "aws_iam_policy" "eventbridge_pipes_ecs_policy" {
name = "test-pipes-ecs-policy"
description = "Policy for EventBridge Pipes to run ECS tasks"

policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Effect = "Allow"
Action = [
"ecs:RunTask",
"ecs:TagResource"
]
Resource = [module.ecs_cluster.services["hello-world"].task_definition_arn]
},
{
Effect = "Allow"
Action = [
"iam:PassRole"
]
Resource = [
module.ecs_cluster.services["hello-world"].task_exec_iam_role_arn,
module.ecs_cluster.services["hello-world"].tasks_iam_role_arn
]
Condition = {
StringLike = {
"iam:PassedToService" = "ecs-tasks.amazonaws.com"
}
}
}
]
})
}

resource "aws_iam_role_policy_attachment" "eventbridge_pipes_ecs_policy" {
for_each = module.eventbridge.eventbridge_pipe_role_names

role = each.value
policy_arn = aws_iam_policy.eventbridge_pipes_ecs_policy.arn
}

######
# ECS
######

module "ecs_cluster" {
source = "terraform-aws-modules/ecs/aws"
version = "~> 5.0"

cluster_name = random_pet.this.id

fargate_capacity_providers = {
FARGATE = {
default_capacity_provider_strategy = {
weight = 100
}
}
FARGATE_SPOT = {
default_capacity_provider_strategy = {
weight = 100
}
}
}

services = {
hello-world = {
create_service = false
subnet_ids = data.aws_subnets.default.ids
desired_count = 1
deployment_maximum_percent = 100
deployment_minimum_healthy_percent = 0

security_group_rules = {
egress = {
type = "egress"
description = "container-pull-egress"
from_port = 443
to_port = 443
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}
}

container_definitions = {
hello-world = {
image = "public.ecr.aws/docker/library/hello-world:latest",
cpu = 256,
memory = 512
}
}
}
}
}

##################
# Extra resources
##################

resource "random_pet" "this" {
length = 2
}

resource "aws_security_group_rule" "default_egress_https" {
type = "egress"
from_port = 443
to_port = 443
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
security_group_id = data.aws_security_group.default.id
description = "Allow HTTPS outbound for ECR image pulls"
}
14 changes: 14 additions & 0 deletions examples/pipe-ecs-target/outputs.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
output "eventbridge_bus_arn" {
description = "The EventBridge Bus ARN"
value = module.eventbridge.eventbridge_bus_arn
}

output "eventbridge_rule_ids" {
description = "The EventBridge Rule IDs"
value = module.eventbridge.eventbridge_rule_ids
}

output "eventbridge_rule_arns" {
description = "The EventBridge Rule ARNs"
value = module.eventbridge.eventbridge_rule_arns
}
Empty file.
14 changes: 14 additions & 0 deletions examples/pipe-ecs-target/versions.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
terraform {
required_version = ">= 1.0"

required_providers {
aws = {
source = "hashicorp/aws"
version = ">= 5.98"
}
random = {
source = "hashicorp/random"
version = ">= 3.0"
}
}
}
2 changes: 2 additions & 0 deletions main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -907,6 +907,8 @@ resource "aws_pipes_pipe" "this" {
container_override {
command = try(ecs_task_parameters.value.command, [])
name = ecs_task_parameters.value.container_name
cpu = ecs_task_parameters.value.cpu
memory = ecs_task_parameters.value.memory

dynamic "environment" {
for_each = try(ecs_task_parameters.value.environment, [])
Expand Down