This document describes how each AWS service is used and how to configure the integration.
The FastAPI application runs on Amazon ECS Fargate — a serverless container execution environment that removes the need to manage EC2 instances.
- The Fargate service runs multiple task replicas behind an Application Load Balancer (ALB)
The platform uses AWS CodeDeploy for zero-downtime blue/green deployments:
- A new task definition is registered with the updated container image
- CodeDeploy starts the new tasks in the idle target group; the Deploy workflow approves the shift once every one is healthy
- The canary shifts 10% of traffic, waits 15 minutes (why 15), then shifts the rest
- Two 5xx alarms, one per target group, watch the API through the canary and the hour after it: while either is in ALARM, CodeDeploy stops the deployment and rolls the API back by itself. They watch the API's target 5xx only, so a release that answers wrongly with a 2xx is not caught, and after that hour the Rollback workflow is the response to a bad release
To deploy a new version, run Actions → Deploy with a release tag
(deployment guide, section 3).
The API's revision moves through CodeDeploy, not through cdk deploy: the
service has a CODE_DEPLOY deployment controller, and ECS refuses a
task-definition change through UpdateService on such a service.
The ECS task reads configuration from AWS Secrets Manager at startup. The variables are listed in the Environment Variables Reference below.
Amazon Aurora PostgreSQL is the primary relational database, storing all experiments, feature flags, users, results, and audit data.
The API connects with the POSTGRES_* variables, not a DATABASE_URL:
POSTGRES_SERVER=aurora-cluster.cluster-xxxx.region.rds.amazonaws.com
POSTGRES_PORT=5432
POSTGRES_USER=username
POSTGRES_PASSWORD=password
POSTGRES_DB=experimentationIn prod the cluster has a writer and a reader; in the other environments it has one instance. The API sends every query to POSTGRES_SERVER, and routes none to the reader.
The API keeps a pool of 20 database connections, opens up to 20 more when they are all busy, and waits up to 30 seconds for a free one; no setting changes these numbers.
REDIS_HOST is the cluster's primary endpoint. REDIS_SSL is true because
in-transit encryption is on, and a plaintext client is refused.
REDIS_HOST=master.your-cluster.xxxxx.use1.cache.amazonaws.com
REDIS_PORT=6379
REDIS_SSL=trueWith CACHE_ENABLED on (it is off by default), GET /api/v1/experiments/{experiment_id} and its daily-results and segmented-results are cached in Redis for 3,600 seconds (an hour). A change to the experiment clears its cached detail; its cached daily and segmented results stay until they expire. GET /api/v1/results/{experiment_id} and GET /api/v1/experiments/{experiment_id}/results are cached in Redis whenever it is reachable, CACHE_ENABLED or not: for 5 minutes while the experiment is running, and for 24 hours otherwise. The API does not cache feature flags: it reads them from the database on every request, so a flag change is served on the next one.
The counters module (full profile) keeps real-time assignment, event and conversion counters
in Amazon DynamoDB, and increments them with atomic ADD update expressions so that
concurrent writers never contend on a row.
DynamoDBCounterService (modules/backend/app/services/dynamodb_counter_service.py) does the
increments. The API's real-time counter endpoints call it, and the bandit scheduler reads the
counters (backend/app/core/bandit_scheduler.py).
DYNAMODB_COUNTERS_TABLE names the table the API uses (default experiment-counters). On the
full profile the counters stack creates experiment-counters-<env>, and the Fargate stack sets
the variable on the API task to that same name (both come from counters_table_name in
infrastructure/cdk/stacks/names.py). It also sets AWS_DEFAULT_REGION to the stack's region,
because the counter service builds its DynamoDB client without naming a region, and gives the
task role dynamodb:Query and dynamodb:UpdateItem on that one table: the two calls the
service makes. The intent is that the /api/v1/counters routes and the bandit scheduler's
DynamoDB read reach the table once this is deployed; nothing writes counters automatically
yet, so the scheduler still takes its statistics from PostgreSQL while the table is empty.
The core profile creates no table and sets neither variable.
No Lambda function serves requests or processes events. The stacks deploy three:
| Function | Stack | What it does |
|---|---|---|
DatabaseAccessLambda |
experimentation-compute-<env> |
A placeholder. Its inline code returns 200 and does nothing |
AnalyticsLambda |
experimentation-analytics-<env> (full profile) |
A placeholder on the Kinesis stream. Its inline code returns 200 and processes nothing |
ETLTriggerLambda |
experimentation-glue-etl-<env> (full profile) |
Starts the Glue ETL jobs, from a daily EventBridge rule |
The repository has Lambda code for event processing and flag evaluation under
backend/lambda/, but no stack deploys it. SDKs call the API (/api/v1/tracking/*,
/api/v1/feature-flags/evaluate/*) for assignment and evaluation.
The CDK does not create a CloudFront distribution, and there is no S3 bucket
for frontend assets. The dashboard is an ECS service behind the same Application
Load Balancer as the API, which sends it every path except /api/*, /health,
/health/* and /metrics; see AWS CDK Deployment.
The split URL module's CloudFront edge router (beta) sends each visitor who opens a split URL
experiment's control page to their variant, as
Split URL testing describes: a CloudFront
distribution in front of your site, with a Lambda@Edge function that asks the API at viewer
request and sets the visitor-id cookie at viewer response. It is a CDK app of its own,
modules/infrastructure/split_url_edge/, with one stack, controlvar-split-url-edge, in
us-east-1, where AWS runs Lambda@Edge functions from.
Nothing is created unless you deploy it: cdk deploy --all of the platform's app does not,
and no workflow or installer step does. Deploying it creates a CloudFront distribution and a
Lambda@Edge function in us-east-1, which AWS bills. Every request for every path through the
distribution runs two Lambda@Edge invocations, one at viewer request and one at viewer
response, and the distribution caches nothing.
Before you deploy it:
- the account is bootstrapped for CDK in us-east-1;
- the CDK app's Python requirements are installed, as in AWS CDK Deployment;
- the API runs the full profile and is reachable from the internet over https, with a valid certificate;
- you have an API key for it: use an API key made for the router; changing it means redeploying the router;
- you manage the DNS of any alternate domain name you give the distribution.
Its settings are environment variables, read when you run cdk deploy from
modules/infrastructure/split_url_edge/:
| Setting | What it is |
|---|---|
SPLIT_URL_API_URL |
Required: the API's https URL, with no path |
SPLIT_URL_API_KEY |
Required: the router's API key |
SPLIT_URL_ORIGIN |
Required: the host name of your site's origin |
SPLIT_URL_COOKIE_NAME |
The visitor-id cookie's name (default controlvar_uid) |
SPLIT_URL_COOKIE_MAX_AGE_DAYS |
Its lifetime, from 1 to 400 days (default 30) |
SPLIT_URL_COOKIE_DOMAIN |
Its Domain attribute (none by default) |
SPLIT_URL_DOMAIN_NAMES |
Alternate domain names for the distribution, separated by commas; with the next |
SPLIT_URL_CERTIFICATE_ARN |
Their ACM certificate, in us-east-1; with the previous |
A setting that is missing or wrong stops the synth, with a message that names it. The function runs Python 3.12 on x86_64.
The router fails open: when the API does not answer in time or answers an error, the visitor
gets the page they asked for, with X-Split-URL: failed. The router gives the API 1 second to
send the routes table and 1.5 seconds to answer for a visitor, from the start of the call to
the last byte of the answer. Looking up the API's host name counts towards that time, but a
slow lookup is not cut short. If the router cannot get the routes table when the page comes
back, that visit gets no cookie. Each failure is one JSON line in
the router's log, with the HTTP status or a fixed word, such as
{"event": "split_url_failed", "phase": "viewer-request", "status": "429"}. The log is in
CloudWatch in the Region where the function ran, under the log group
/aws/lambda/us-east-1.<function name>.
X-Split-URL on a served page rests on a behaviour AWS does not document (that a header
added at viewer request is visible at viewer response). It is unverified until the router
has run on AWS: on 2026-11-02 it had not.
The CloudFront router sets the visitor-id cookie on entry pages without asking. A site that must have consent before it sets an identifying cookie uses a server-side integration instead, which makes no call and sets no cookie until the visitor consents.
On 2026-11-02 the router had been tested against simulated CloudFront events, and had not yet run on AWS.
The full profile's analytics stack (experimentation-analytics-<env>) creates a Kinesis Data
Stream that Firehose delivers to the S3 data lake, and an Amazon OpenSearch Service
domain. The resource names are generated by CloudFormation.
The API does not publish tracking events to the stream today: POST /api/v1/tracking/track
and /tracking/batch store events in PostgreSQL. Nothing indexes events into OpenSearch
either, because the Lambda on the stream is a placeholder (see
Lambda Functions).
All infrastructure is defined in infrastructure/cdk using AWS CDK v2, in Python.
How to bootstrap the account and deploy the stacks is in AWS CDK Deployment.
ALARM_EMAIL is the address every alarm emails, including the API rollback
alarms. It is required for staging and prod, and synth refuses to run
without it. Confirm the subscription it creates from the inbox; see
AWS CDK Deployment for
that and for the other required settings.
The stacks a deployment builds are listed in AWS CDK Deployment.
The CDK (infrastructure/cdk/stacks/fargate_service_stack.py) gives the API's ECS task role
these permissions:
- the
CloudWatchLogsFullAccessmanaged policy secretsmanager:GetSecretValueon the secrets the task readsssmmessages:CreateControlChannel,CreateDataChannel,OpenControlChannel,OpenDataChannel,logs:DescribeLogGroups,CreateLogStream,DescribeLogStreamsandPutLogEvents, which come with ECS Exec- full profile only:
dynamodb:Queryanddynamodb:UpdateItemonarn:aws:dynamodb:<region>:<account>:table/experiment-counters-<env>(see DynamoDB) - full profile only, for the
etlmodule's routes (see Glue):glue:StartJobRunandglue:GetJobRunonarn:aws:glue:<region>:<account>:job/experimentation-events-etl-<env>andjob/experimentation-metrics-etl-<env>glue:StartCrawlerandglue:GetCrawleroncrawler/experimentation-crawler-<env>glue:GetTableandglue:BatchCreatePartitiononcatalog,database/experimentation_<env>andtable/experimentation_<env>/*
There is no Kinesis, Cognito, Athena, S3 or other DynamoDB permission, and no other Glue action or Glue object.
On the full profile the Fargate stack tells the API the Glue names the glue-etl stack creates
(GLUE_ETL_JOB_NAME, GLUE_METRICS_JOB_NAME, GLUE_CRAWLER_NAME, GLUE_DATABASE, all from
glue_names in infrastructure/cdk/stacks/names.py) and grants the six calls above on those
objects alone. The routes accept only those names (see the ETL section of the API reference).
The Glue client is built without naming a region, so it takes the region from
AWS_DEFAULT_REGION. The client never reads AWS_REGION. The CDK sets AWS_DEFAULT_REGION on
the task to the stack's region, and the repository's docker-compose.yml sets it from
AWS_REGION. Anywhere else (Helm, a container you run yourself), set AWS_DEFAULT_REGION
to the region your Glue job and crawler are in. Without it the job, crawler and partitions
routes answer 500 and the API logs NoRegionError. On Helm, put it in api.extraEnv alongside the GLUE_* names. The chart has
no value of its own for it.
GLUE_EVENTS_TABLE (default raw_events) names the one catalog table POST /etl/partitions/add
accepts, and the CDK does not set it. It must be the table the crawler creates. The crawler is
given an S3 path and no table prefix, so it picks the table's name itself when it runs. Check
the name after the first crawl and set GLUE_EVENTS_TABLE to it; until then that route
answers 404.
| Variable | Required | Description |
|---|---|---|
POSTGRES_SERVER |
Yes | Aurora endpoint; with POSTGRES_PORT, POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB. DATABASE_URL is not read; in staging and production the container's start-up check refuses to start with it set |
REDIS_HOST |
Yes | ElastiCache primary endpoint (localhost if unset) |
REDIS_PORT |
No | ElastiCache port (default 6379) |
REDIS_SSL |
Yes, on AWS | true to connect over TLS (default false) |
SECRET_KEY |
Yes | Application secret key (min 32 chars) |
COGNITO_USER_POOL_ID |
Only for Cognito sign-in | AWS Cognito User Pool ID |
COGNITO_CLIENT_ID |
Only for Cognito sign-in | Cognito app client ID |
COGNITO_DOMAIN |
Only for sign-in through the Cognito Hosted UI | The Hosted UI's domain name, <prefix>.auth.<region>.amazoncognito.com, no scheme. A value that is not a domain name stops the API from starting |
AWS_REGION |
Yes | Primary AWS region, read by Cognito and the other settings that name it. The Glue and DynamoDB counter clients do not read it |
AWS_DEFAULT_REGION |
Full profile: yes, for the ETL routes | The region the Glue client (see Glue) and the DynamoDB counter client (see DynamoDB) use. The CDK sets it to the stack's region. Outside the CDK set it yourself (Helm: api.extraEnv); without it the job and crawler routes answer 500 |
GLUE_ETL_JOB_NAME, GLUE_METRICS_JOB_NAME, GLUE_CRAWLER_NAME, GLUE_DATABASE |
Full profile, for the ETL routes | The Glue job, crawler and database names the ETL routes accept. The CDK sets them to the glue-etl stack's -<env> names |
GLUE_EVENTS_TABLE |
Full profile, for POST /etl/partitions/add |
The catalog table that route accepts (default raw_events). Not set by the CDK: set it to the table the crawler creates (see Glue) |
DYNAMODB_COUNTERS_TABLE |
No | Full profile: the counters table, set by the CDK to experiment-counters-<env> (default experiment-counters; see DynamoDB) |
SLACK_BOT_TOKEN |
No | Slack bot token for alerting |
SENDGRID_API_KEY |
No | SendGrid API key for email alerts |
AUDIT_HMAC_KEY |
Full profile: yes | Key for HMAC-SHA256 audit event signing |
ANTHROPIC_API_KEY |
No | Claude API key for AI experiment design |