Skip to content

Add bucket for datalake #879

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
svevang wants to merge 2 commits into
base: main
Choose a base branch
from
+53 −2
Open

Add bucket for datalake #879

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5bb74fa
Add bucket for datalake
svevang Feb 24, 2026
c6ddc7c
Bucket is not versioned
svevang Feb 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 53 additions & 2 deletions spire/templates/apps/augury.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -240,7 +240,6 @@ Resources:
- s3:Get*
- s3:ListBucket
- s3:ListBucketMultipartUploads
- s3:ListBucketVersions
Effect: Allow
Resource: !Ref AdFilesS3BucketArn
Sid: AllowBucketActions
Expand All @@ -250,12 +249,32 @@ Resources:
- s3:ListMultipartUploadParts
- s3:PutObject
- s3:PutObjectAcl
- s3:PutObjectVersionAcl
Effect: Allow
Resource: !Sub ${AdFilesS3BucketArn}/*
Sid: AllowObjectActions
Version: "2012-10-17"
PolicyName: S3UploadAccess
- PolicyDocument:
Statement:
- Action:
- s3:Get*
- s3:ListBucket
- s3:ListBucketMultipartUploads
Effect: Allow
Resource: !GetAtt ForecastsDataLakeBucket.Arn
Sid: AllowDataLakeBucketActions
- Action:
- s3:AbortMultipartUpload
- s3:DeleteObject
- s3:Get*
- s3:ListMultipartUploadParts
- s3:PutObject
- s3:PutObjectAcl
Effect: Allow
Resource: !Sub ${ForecastsDataLakeBucket.Arn}/*
Sid: AllowDataLakeObjectActions
Version: "2012-10-17"
PolicyName: ForecastsDataLakeBucket
Tags:
- { Key: prx:meta:tagging-version, Value: "2021-04-07" }
- { Key: prx:cloudformation:stack-name, Value: !Ref AWS::StackName }
Expand Down Expand Up @@ -453,6 +472,10 @@ Resources:
Value: !If [IsProduction, Augury Production, Augury Staging]
- Name: BQ_DATASET
Value: !If [IsProduction, production, staging]
- Name: DATALAKE_URI
Value: !Sub s3://${ForecastsDataLakeBucket}
- Name: DATALAKE_AWS_REGION
Value: !Ref AWS::Region
- Name: SLOW_WORKER_RELAY_SQS_URL
Value: !Ref SlowWorkerRelaySqsQueue
- Name: SLACK_SNS_TOPIC
Expand Down Expand Up @@ -612,6 +635,10 @@ Resources:
Value: !If [IsProduction, Augury Production, Augury Staging]
- Name: BQ_DATASET
Value: !If [IsProduction, production, staging]
- Name: DATALAKE_URI
Value: !Sub s3://${ForecastsDataLakeBucket}
- Name: DATALAKE_AWS_REGION
Value: !Ref AWS::Region
- Name: SLOW_WORKER_RELAY_SQS_URL
Value: !Ref SlowWorkerRelaySqsQueue
- Name: SLACK_SNS_TOPIC
Expand Down Expand Up @@ -680,6 +707,26 @@ Resources:
- { Key: prx:dev:application, Value: Augury }
TaskRoleArn: !GetAtt TaskRole.Arn

ForecastsDataLakeBucket:
Type: AWS::S3::Bucket
DeletionPolicy: Retain
UpdateReplacePolicy: Retain
Properties:
PublicAccessBlockConfiguration:
BlockPublicAcls: true
BlockPublicPolicy: true
IgnorePublicAcls: true
RestrictPublicBuckets: true
Tags:
- { Key: prx:meta:tagging-version, Value: "2021-04-07" }
- { Key: prx:cloudformation:stack-name, Value: !Ref AWS::StackName }
- { Key: prx:cloudformation:stack-id, Value: !Ref AWS::StackId }
- { Key: prx:cloudformation:root-stack-name, Value: !Ref RootStackName }
- { Key: prx:cloudformation:root-stack-id, Value: !Ref RootStackId }
- { Key: prx:ops:environment, Value: !Ref EnvironmentType }
- { Key: prx:dev:family, Value: Dovetail }
- { Key: prx:dev:application, Value: Augury }

S3SigningUser:
Type: AWS::IAM::User
Properties:
Expand Down Expand Up @@ -1032,6 +1079,10 @@ Resources:
Value: !If [IsProduction, Augury Production, Augury Staging]
- Name: BQ_DATASET
Value: !If [IsProduction, production, staging]
- Name: DATALAKE_URI
Value: !Sub s3://${ForecastsDataLakeBucket}
- Name: DATALAKE_AWS_REGION
Value: !Ref AWS::Region
- Name: SLACK_SNS_TOPIC
Value: !Ref SlackMessageRelaySnsTopicArn
- Name: SLACK_CHANNEL_ID
Expand Down