Skip to content

Commit bea0a62

Browse files
jimrightjimright
authored
Add network and hosts modules for AWS deployment (#1)
* Add network module for AWS * Add hosts module for AWS * Add top level readme Signed-off-by: Jim Enright <[email protected]>
1 parent ae1b4f9 commit bea0a62

File tree

20 files changed

+1059
-40
lines changed

20 files changed

+1059
-40
lines changed

‎README.md

Lines changed: 4 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,12 @@
1-
# Terraform Modules for Cloudera on Premise Community Edition Infrastructure Creation on <ENTER_INFRA_PROVIDER>
1+
# Terraform Modules for Cloudera on Premise Community Edition Infrastructure Creation on AWS
22

3-
This repository contains a number of Terraform modules for creation of infrastructure resources for Cloudera on premise clusters on <ENTER_INFRA_PROVIDER>. These modules support deployment of the Community Edition reference cluster.
3+
This repository contains a number of Terraform modules for creation of infrastructure resources for Cloudera on premise clusters on AWS. These modules support deployment of the Community Edition reference cluster.
44

55
## Modules
66

77
| Module name | Description |
88
| ----------- | ----------- |
9-
| [hosts](modules/hosts/README.md) | <ENTER_SHORT_DESCRIPTION> |
9+
| [network](modules/network/README.md) | Creates and manages AWS networking infrastructure with public/private subnets, NAT gateways, route tables, and security groups for Cloudera on premise deployments. |
10+
| [hosts](modules/hosts/README.md) | Provisions and manages AWS EC2 instances with flexible configuration for compute resources, storage volumes, and networking designed for Cloudera on premise deployments. |
1011

1112
Each module contains Terraform resource configuration and example variable definition files.
12-
13-
## Usage
14-
15-
The [TODO: CE Public Repo](https://github.com/cloudera-labs/community-edition) repository demonstrates how to use the modules together to deploy the infrastructure for a Cloudera on premise cluster.

‎modules/hosts/README.md

Lines changed: 40 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,44 +1,72 @@
11
<!-- BEGIN_TF_DOCS -->
2-
# Terraform Module for Hosts on ENTER\_INFRA\_PROVIDER
2+
# Terraform Module for AWS Hosts
33

4-
> [!IMPORTANT]
5-
> This readme is automation generated using the [`terraform-docs`](https://terraform-docs.io/) command with the static contenta taken from [doc\_fragments/header.md](doc\_fragments/header.md)
6-
7-
The `hosts` module contains resource files to provision and manage <ENTER\_INFRA\_PROVIDER> instances with flexible configuration options for compute resources, storage volumes, and networking. This module is designed for Cloudera on premise infrastructure deployments on <ENTER\_INFRA\_PROVIDER>.
4+
The `hosts` module contains resource files to provision and manage AWS EC2 instances with flexible configuration options for compute resources, storage volumes, and networking. This module is designed for Cloudera on premise infrastructure deployments on AWS IaaS but can be used for any AWS EC2 instance provisioning needs.
85

96
## Key Features
107

11-
<ENTER\_KEY\_FEATURES>
8+
- **Flexible Instance Provisioning**: Create single instances or multiple identical nodes with sequential naming
9+
- **Custom Storage Configuration**: Attach and manage additional EBS volumes with configurable size, type, and mount points
10+
- **Network Integration**: Place instances in specific subnets with optional public IP assignment
11+
- **Security Management**: Apply security groups to control access to instances
12+
- **Resource Tagging**: Add custom tags to all provisioned resources for better organization and cost management
13+
- **IMDSv2 Support**: Automatic detection and configuration of IMDSv2 tokens based on AMI capabilities
1214

1315
## Usage
1416

1517
The [examples](./examples) directory has example of using this module:
1618

17-
* `ex01-minimal_inputs` demonstrates how this module can be used to create a number of hosts.
19+
* `ex01-minimal_inputs` demonstrates how this module can be used to create a number of EC2 instances. The [terraform-aws-network](../terraform-aws-network/README.md) module is also used as part of this example.
1820

1921
The sample `terraform.tfvars.sample` describes the required inputs for the example.
2022

2123
## Requirements
2224

23-
No requirements.
25+
| Name | Version |
26+
|------|---------|
27+
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.13 |
28+
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | >= 4.60.0 |
2429

2530
## Providers
2631

27-
No providers.
32+
| Name | Version |
33+
|------|---------|
34+
| <a name="provider_aws"></a> [aws](#provider\_aws) | >= 4.60.0 |
2835

2936
## Modules
3037

3138
No modules.
3239

3340
## Resources
3441

35-
No resources.
42+
| Name | Type |
43+
|------|------|
44+
| [aws_ebs_volume.inventory](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ebs_volume) | resource |
45+
| [aws_instance.pvc_base](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/instance) | resource |
46+
| [aws_volume_attachment.inventory](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/volume_attachment) | resource |
47+
| [aws_ami.pvc_base](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/ami) | data source |
3648

3749
## Inputs
3850

39-
No inputs.
51+
| Name | Description | Type | Default | Required |
52+
|------|-------------|------|---------|:--------:|
53+
| <a name="input_image_id"></a> [image\_id](#input\_image\_id) | AMI image ID for the hosts | `string` | n/a | yes |
54+
| <a name="input_name"></a> [name](#input\_name) | Instance name. If 'quantity' is set, name will be <name>-NN. | `string` | n/a | yes |
55+
| <a name="input_security_groups"></a> [security\_groups](#input\_security\_groups) | List of security group IDs to attach to the instances | `list(string)` | n/a | yes |
56+
| <a name="input_ssh_key_pair"></a> [ssh\_key\_pair](#input\_ssh\_key\_pair) | SSH key pair name | `string` | n/a | yes |
57+
| <a name="input_subnet_ids"></a> [subnet\_ids](#input\_subnet\_ids) | List of subnet IDs to provision the instances | `list(string)` | n/a | yes |
58+
| <a name="input_instance_type"></a> [instance\_type](#input\_instance\_type) | Instance type name for the hosts | `string` | `"t2.micro"` | no |
59+
| <a name="input_offset"></a> [offset](#input\_offset) | Number offset for instance name | `number` | `0` | no |
60+
| <a name="input_public_ip"></a> [public\_ip](#input\_public\_ip) | Flag to assign public IP addresses to the hosts | `bool` | `false` | no |
61+
| <a name="input_quantity"></a> [quantity](#input\_quantity) | Number of instances. Defaults to a single instance without numbering (bare name). | `number` | `0` | no |
62+
| <a name="input_root_volume"></a> [root\_volume](#input\_root\_volume) | Root volume details | <pre>object({<br/> delete_on_termination = optional(bool, true)<br/> volume_size = optional(number, 100)<br/> volume_type = optional(string)<br/> })</pre> | `{}` | no |
63+
| <a name="input_tags"></a> [tags](#input\_tags) | Map of tags applied to all cloud-provider assets. | `map(any)` | `{}` | no |
64+
| <a name="input_volumes"></a> [volumes](#input\_volumes) | Additional storage volumes to attach to the hosts. Each volume is defined by a device name, mount point, size, type, and optional tags. | <pre>list(object({<br/> device_name = string<br/> mount = string<br/> volume_size = optional(number, 100)<br/> volume_type = optional(string, "gp2")<br/> tags = optional(map(string), {})<br/> }<br/> )<br/> )</pre> | `null` | no |
4065

4166
## Outputs
4267

43-
No outputs.
68+
| Name | Description |
69+
|------|-------------|
70+
| <a name="output_hosts"></a> [hosts](#output\_hosts) | Hosts |
71+
| <a name="output_storage_volumes"></a> [storage\_volumes](#output\_storage\_volumes) | Additional Storage Volumes |
4472
<!-- END_TF_DOCS -->

‎modules/hosts/doc_fragments/header.md

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,18 +1,20 @@
1-
# Terraform Module for Hosts on ENTER_INFRA_PROVIDER
1+
# Terraform Module for AWS Hosts
22

3-
> [!IMPORTANT]
4-
> This readme is automation generated using the [`terraform-docs`](https://terraform-docs.io/) command with the static contenta taken from [doc_fragments/header.md](doc_fragments/header.md)
5-
6-
The `hosts` module contains resource files to provision and manage <ENTER_INFRA_PROVIDER> instances with flexible configuration options for compute resources, storage volumes, and networking. This module is designed for Cloudera on premise infrastructure deployments on <ENTER_INFRA_PROVIDER>.
3+
The `hosts` module contains resource files to provision and manage AWS EC2 instances with flexible configuration options for compute resources, storage volumes, and networking. This module is designed for Cloudera on premise infrastructure deployments on AWS IaaS but can be used for any AWS EC2 instance provisioning needs.
74

85
## Key Features
96

10-
<ENTER_KEY_FEATURES>
7+
- **Flexible Instance Provisioning**: Create single instances or multiple identical nodes with sequential naming
8+
- **Custom Storage Configuration**: Attach and manage additional EBS volumes with configurable size, type, and mount points
9+
- **Network Integration**: Place instances in specific subnets with optional public IP assignment
10+
- **Security Management**: Apply security groups to control access to instances
11+
- **Resource Tagging**: Add custom tags to all provisioned resources for better organization and cost management
12+
- **IMDSv2 Support**: Automatic detection and configuration of IMDSv2 tokens based on AMI capabilities
1113

1214
## Usage
1315

1416
The [examples](./examples) directory has example of using this module:
1517

16-
* `ex01-minimal_inputs` demonstrates how this module can be used to create a number of hosts.
18+
* `ex01-minimal_inputs` demonstrates how this module can be used to create a number of EC2 instances. The [terraform-aws-network](../terraform-aws-network/README.md) module is also used as part of this example.
1719

1820
The sample `terraform.tfvars.sample` describes the required inputs for the example.

‎modules/hosts/examples/ex01-minimal_inputs/main.tf

Lines changed: 108 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -13,12 +13,117 @@
1313
# limitations under the License.
1414

1515
provider "aws" {
16-
region = var.aws_region
16+
region = var.region
17+
default_tags {
18+
tags = var.asset_tags
19+
}
1720
}
1821

22+
# ------- VPC -------
23+
24+
resource "aws_vpc" "ex01" {
25+
cidr_block = var.vpc_cidr
26+
tags = { Name = "${var.prefix}-vpc" }
27+
instance_tenancy = "default"
28+
enable_dns_support = true
29+
enable_dns_hostnames = true
30+
}
31+
32+
resource "aws_internet_gateway" "ex01" {
33+
vpc_id = aws_vpc.ex01.id
34+
tags = { Name = "${var.prefix}-igw" }
35+
}
36+
37+
resource "aws_vpc_dhcp_options" "ex01" {
38+
domain_name = "${var.prefix}.cldr.internal"
39+
domain_name_servers = ["AmazonProvidedDNS"]
40+
41+
tags = { Name = "${var.prefix}-vpc-dhcp-options" }
42+
}
43+
44+
resource "aws_vpc_dhcp_options_association" "ex01" {
45+
vpc_id = aws_vpc.ex01.id
46+
dhcp_options_id = aws_vpc_dhcp_options.ex01.id
47+
}
48+
49+
# ------- Network Module -------
50+
module "ex01_network" {
51+
source = "../../../network"
52+
53+
region = var.region
54+
prefix = var.prefix
55+
vpc_id = aws_vpc.ex01.id
56+
57+
}
58+
59+
resource "aws_security_group" "ssh" {
60+
vpc_id = aws_vpc.ex01.id
61+
name = "${var.prefix}-sg-ssh"
62+
description = "SSH traffic [${var.prefix}]"
63+
tags = { Name = "${var.prefix}-sg-ssh" }
64+
}
65+
66+
resource "aws_vpc_security_group_ingress_rule" "ssh" {
67+
for_each = toset(var.ingress_ssh_cidrs)
68+
69+
security_group_id = aws_security_group.ssh.id
70+
description = "SSH traffic from ${each.value}"
71+
cidr_ipv4 = each.value
72+
from_port = 22
73+
ip_protocol = "tcp"
74+
to_port = 22
75+
tags = { Name = "${var.prefix}-ssh-${index(var.ingress_ssh_cidrs, each.value)}" }
76+
}
77+
78+
# ------- Host Module -------
1979
module "ex01_hosts" {
20-
source = "../.."
80+
source = "../.."
81+
depends_on = [aws_key_pair.ex01, data.aws_ami.ex01]
2182

22-
# <ENTER_REQUIRED_INPUTS>
83+
name = "${var.prefix}-host"
84+
quantity = 2
85+
image_id = data.aws_ami.ex01.image_id
86+
instance_type = "t2.micro"
87+
subnet_ids = module.ex01_network.public_subnets[*].id # Will use the first public subnet
88+
ssh_key_pair = aws_key_pair.ex01.key_name
89+
security_groups = [
90+
module.ex01_network.intra_cluster_security_group.id,
91+
aws_security_group.ssh.id
92+
]
93+
public_ip = true
94+
}
2395

96+
# ------- AMI -------
97+
locals {
98+
ami_owners = ["amazon"]
99+
ami_filters = {
100+
name = ["al2023-ami-2023*"]
101+
architecture = ["x86_64"]
102+
}
24103
}
104+
105+
data "aws_ami" "ex01" {
106+
owners = local.ami_owners
107+
most_recent = true
108+
109+
dynamic "filter" {
110+
for_each = local.ami_filters
111+
112+
content {
113+
name = filter.key
114+
values = filter.value
115+
}
116+
}
117+
}
118+
119+
# ------- SSH -------
120+
121+
data "tls_public_key" "ex01" {
122+
private_key_openssh = file(var.ssh_private_key_file)
123+
}
124+
125+
resource "aws_key_pair" "ex01" {
126+
key_name = "${var.prefix}-key"
127+
public_key = trimspace(data.tls_public_key.ex01.public_key_openssh)
128+
}
129+

‎modules/hosts/examples/ex01-minimal_inputs/outputs.tf

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# Copyright 2025 Cloudera, Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
output "ssh_key_pair" {
16+
value = {
17+
name = aws_key_pair.ex01.key_name
18+
public_key = trimspace(data.tls_public_key.ex01.public_key_openssh)
19+
type = aws_key_pair.ex01.key_type
20+
fingerprint = aws_key_pair.ex01.fingerprint
21+
}
22+
description = "SSH public key"
23+
}
24+
25+
output "vpc" {
26+
value = aws_vpc.ex01.id
27+
description = "AWS VPC"
28+
}
29+
30+
output "nodes" {
31+
value = values(module.ex01_hosts)
32+
33+
description = "Node information including IDs, names, private and public IPs"
34+
}

‎modules/hosts/examples/ex01-minimal_inputs/terraform.tfvars.sample

Lines changed: 15 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,20 @@
1212
# See the License for the specific language governing permissions and
1313
# limitations under the License.
1414

15-
# <ENTER_REQUIRED_INPUT_VALUES>
16-
1715
# ------- Global Settings -------
18-
name_prefix = "<ENTER_VALUE>"
16+
prefix = "ex01"
17+
18+
region = "<ENTER_VALUE>" # Change this to specify Cloud Provider region, e.g. eu-west-1
19+
20+
asset_tags = {
21+
"Environment" = "Development"
22+
}
23+
24+
# ------- Network settings -------
25+
ingress_ssh_cidrs = [
26+
"<ENTER_CIDR>",
27+
"<ENTER_CIDR>"
28+
]
1929

20-
# ------- Cloud Settings -------
30+
# ------- SSH settings -------
31+
ssh_private_key_file = "<ENTER_SSH_PRIVATE_KEY_FILE_PATH>" # Path to the SSH

‎modules/hosts/examples/ex01-minimal_inputs/variables.tf

Lines changed: 39 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -12,12 +12,46 @@
1212
# See the License for the specific language governing permissions and
1313
# limitations under the License.
1414

15-
# <ENTER_REQUIRED_INPUT_VARIABLES>
15+
# ------- General and Provider Resources -------
1616

17-
# ------- Global settings -------
18-
variable "name_prefix" {
17+
variable "prefix" {
1918
type = string
20-
description = "Shorthand name to use when naming resources."
19+
description = "Deployment prefix for all cloud-provider assets"
20+
21+
validation {
22+
condition = length(var.prefix) < 8 || length(var.prefix) > 4
23+
error_message = "Valid length for prefix is between 4-7 characters."
24+
}
25+
}
26+
27+
variable "region" {
28+
type = string
29+
description = "AWS region"
2130
}
2231

23-
# ------- Cloud Settings -------
32+
variable "asset_tags" {
33+
type = map(string)
34+
default = {}
35+
description = "Map of tags applied to all cloud-provider assets"
36+
}
37+
38+
# ------- Network Resources -------
39+
variable "vpc_cidr" {
40+
type = string
41+
description = "VPC CIDR Block (primary)"
42+
default = "10.10.0.0/16"
43+
}
44+
45+
variable "ingress_ssh_cidrs" {
46+
type = list(string)
47+
description = "List of CIDRs to add to ingress SSH inbound rules."
48+
49+
default = []
50+
}
51+
52+
# ------- SSH Resources -------
53+
54+
variable "ssh_private_key_file" {
55+
type = string
56+
description = "Local SSH private key file"
57+
}

0 commit comments

Comments
 (0)