AWS

One Fargate task runs the image behind an ALB that admits only Cloudflare’s edge, and one proxied Cloudflare record is the public hostname. Everything here is infra/aws/ and .github/workflows/deploy-aws-prod.yml; nothing is shared with another deployment.

Topology

×

The task has no database, no bucket and no secret. Its weights are in the image and read offline. Bedrock is reached on the task role’s credentials, and only when a model variable is set; otherwise the LLM lanes report themselves unavailable and the role carries no Bedrock statement.

The task takes bytes inline and nothing else. INDX_URI_SCHEMES=none disables every URI source, so no caller can name a file on the task’s disk or a URL inside the VPC, and INDX_LOADER_FILE_ROOTS is closed to the samples directory as a second layer (ADR-0042). The playground’s URI tab is disabled on this host for the same reason. The public smoke asserts that resolvable.schemes is empty, that a file: URI is refused and that an inline upload still plans, and just test::bdd::hardened runs the same scenarios locally against a server started that way. Against the host itself, just test::bdd::deployed https://<hostname> runs every @deployed acceptance scenario (the deploy workflow runs it after the public smoke) and just frontend::e2e-deployed https://<hostname> drives the web app’s specs through a browser. One note for scripted clients: Cloudflare refuses Python’s default Python-urllib user agent with error 1010, while curl and httpx pass.

The stacks

Stack	Applied by	Holds
infra/aws/terraform-prod-bootstrap	an administrator, once	the state bucket, the lock table, the two IAM roles GitHub Actions assumes
infra/aws/terraform	the workflow’s deploy	VPC, ALB, ECR, ECS, IAM, logs, alarms
infra/aws/terraform-prod-dns	the workflow’s cutover	the one Cloudflare record

State for all three lives in the bucket and lock table the bootstrap stack creates for this project alone, under indx-everything-prod/aws/: bootstrap.tfstate, terraform.tfstate and dns.tfstate. The application itself has no bucket. just infra::terraform::validate formats, initializes without a backend and validates every stack, with no credentials; it is also the first job of every workflow run.

The perimeter is two rules in the core stack. The ALB’s security group admits Cloudflare’s published IPv4 ranges alone, and the HTTPS listener forwards only when the Host header is the certified hostname, so another Cloudflare zone cannot serve this ALB as its own. That is a perimeter and not authentication: the API is unauthenticated behind it. Cloudflare Access on the hostname is the gate an operator turns on before the cutover.

Bootstrap, once

Needs administrator credentials for account 697441478678 and the aws CLI. Every identifier the stack uses (region, account, project name, bucket, lock table, state prefix) is a variable with a validation pinning it to one value, so there is nothing to fill in.

A profile signed in with aws login holds a login_session and no keys, and the Terraform AWS provider does not read those sessions: it reports no credential source and falls through to the EC2 metadata endpoint. Hand it the session as environment variables instead:

eval "$(aws configure export-credentials --profile <admin profile> --format env)"

The bucket the bootstrap’s own state belongs in is one of the things it creates, so the first apply runs on local state and migrates it straight after. The stack declares the S3 backend, and init -backend=false would leave it unconfigured, so an override file swaps in the local backend for that one apply and is removed before the migration:

cd infra/aws/terraform-prod-bootstrap
printf 'terraform {\n  backend "local" {}\n}\n' > local_override.tf
terraform init
terraform apply
rm local_override.tf
terraform init -migrate-state -backend-config=backend.hcl.example
terraform output

The migration asks once to copy the local state into the bucket; answer yes, then delete the terraform.tfstate and terraform.tfstate.backup it leaves behind.

Note the two outputs github_deploy_role_arn and github_plan_role_arn.

The roles trust only the repository’s immutable GitHub OIDC subject, repo:INDXDev@209891251/indx-everything@1301159823:environment:<name>, so a workflow outside those Environments cannot assume them and a repository rename cannot transfer the trust. The plan role is read-only plus the state lock; the deploy role’s policy is infra/aws/terraform/ci-prod-deploy-policy.json, scoped to the resources this project’s stack names and denied mutation of anything tagged with another project.

The deploy policy is re-read on every terraform apply of this stack, so a change to that file (the Bedrock budget added one) is an administrator re-running the same commands.

GitHub, once

1. Create two Environments. production-plan with no reviewer and a deployment branch rule of main only. production with a required reviewer and the same branch rule. That reviewer is the approval for every mutation.
2. Set the repository variables and secrets:

Name	Kind	Value
PROD_AWS_ROLE_ARN	variable	the deploy role ARN
PROD_AWS_PLAN_ROLE_ARN	variable	the plan role ARN
PROD_DOMAIN_NAME	variable	the public hostname, under indx.jp
PROD_BEDROCK_ENABLED	variable, optional	true to turn the Bedrock lanes on; unset is off
PROD_BEDROCK_LLM_MODEL	variable, optional	e.g. global.anthropic.claude-sonnet-4-6
PROD_BEDROCK_EMBED_MODEL	variable, optional	e.g. cohere.embed-multilingual-v3
PROD_BEDROCK_EMBED_DIMENSION	variable, required with the model	the model’s vector size, e.g. 1024 for cohere.embed-multilingual-v3
PROD_BEDROCK_MONTHLY_BUDGET_USD	variable, optional	the monthly Bedrock spend at which calls are denied; 50 by default
PROD_CLOUDFLARE_API_TOKEN	secret	a token that edits DNS in the indx.jp zone

3. Enable Cloudflare Access for the hostname.

Region, account, project name, zone ID, bucket, lock table and state keys have defaults in the workflow, and scripts/aws/prod-preflight.sh refuses to run when any of them drifts from the value the roles were minted for.

The Bedrock switch

The API is public, and every call to a Bedrock-backed lane is billed to the account, so those lanes are off unless PROD_BEDROCK_ENABLED is true (ADR-0040). Off means the task carries no model variable, so llm, ner-llm and enrich-llm are not advertised and a request naming one is refused with a 422, generic-vlm and hosted-text report themselves unavailable, and the task role holds no Bedrock permission. A model named while the switch is off fails the Terraform plan rather than being ignored.

On, the stack adds a monthly AWS Budgets cost budget on Bedrock (PROD_BEDROCK_MONTHLY_BUDGET_USD) that notifies the alarm topic at 80 percent forecast and, at 100 percent actual, attaches a Deny bedrock:* policy to the task role. AWS refreshes budget data a few times a day, so the deny lands hours after the threshold; it stops a runaway bill, not a request. The deny stays until an operator resets the action:

aws budgets execute-budget-action --account-id 697441478678 \
  --budget-name indx-everything-prod-bedrock --action-id <id> --execution-type RESET_BUDGET_ACTION

Flipping the switch either way is a plan and a deploy, because the task definition changes. The public smoke asserts each Bedrock lane is present exactly when its model is set. A Cloudflare rate-limiting rule on the hostname is the second brake, and an operator setting.

Operating

Every operation is Deploy to AWS (production) on main, dispatched by hand. No push deploys.

action	Environment	What happens
plan	production-plan	A read-only plan of the core stack. Its summary is the artifact a deploy must name.
deploy	production	Builds the image into ECR if the tag is absent, captures the rollback point, re-plans and refuses when the plan differs from the approved run beyond the ECR bootstrap, applies, waits for healthy targets, runs the pre-cutover smoke, and, when the hostname already resolves, the public smoke, the @deployed scenarios and the web app’s specs. DNS is not changed.
cutover	production	Snapshots the current record, re-checks target health, plans the DNS stack, refuses anything but exactly one proxied record for the hostname, re-snapshots, applies, runs the public smoke.
rollback	production	Restores the task definition a deploy run captured, reconciles Terraform to the same image, and must be healthy within 600 seconds.

confirm must equal the hostname for every mutating action. deploy needs plan_run_id; rollback needs rollback_run_id. Every run starts with the static validation job, then the one job the action names. What each job does, in order:

plan

Runs in production-plan, so no reviewer, but only from main and only on the read-only role.

1. Assumes the plan role and runs prod-preflight.sh plan: the caller must be that exact role in that exact account, and the state bucket and lock table must exist.
2. Initializes the core stack against the S3 backend and plans it with the image reference <ecr>/indx-everything-prod-indx-everything:sha-<commit>.
3. Uploads core-plan-summary.json, the resource addresses and actions plus the commit, ref and repository it was planned from, as aws-prod-plan-summary-<run id>.

Note the run ID. It is the plan_run_id a deploy must cite, and it is valid for that commit only.

deploy

Dispatch with confirm=<hostname> and plan_run_id. The reviewer approves the production Environment, then:

1. Guards: the ref is main, confirm equals PROD_DOMAIN_NAME, plan_run_id is numeric. Downloads the approved plan summary from that run.
2. Assumes the deploy role and runs prod-preflight.sh deploy.
3. Applies only aws_ecr_repository.indx_everything, so the repository exists before the push.
4. Captures the rollback point: the service’s current task definition, written to rollback.json and uploaded as aws-prod-rollback-<run id>, kept 30 days. On the first deployment there is no cluster yet, and the step records that there is nothing to roll back to.
5. Frees runner disk, logs into ECR and GHCR, and ensures the weights image ghcr.io/indxdev/indx-everything-weights:w-<key> exists, publishing it only when absent.
6. Builds Dockerfile for linux/amd64 and pushes sha-<commit> to ECR, skipped when that tag already exists.
7. Plans the core stack again and diffs it against the approved summary. The provenance must be this repository, main and this commit, and the change set must be identical except for the ECR repository. Any other difference fails the run before anything is applied.
8. Applies that plan. The first apply also issues the ACM certificate and writes its validation record into Cloudflare, which is why the Cloudflare token is in scope here and not only at cutover.
9. Waits for the ECS service to be stable and the target group to report the task in service.
10. Runs prod-smoke.sh precutover. When the hostname already resolves, runs public too. DNS is not touched.

cutover

Dispatch with confirm=<hostname>. The reviewer approves, then:

1. Guards as for deploy, minus the plan run.
2. Runs prod-preflight.sh cutover, then snapshots the current Cloudflare record to cloudflare-before-cutover.json, uploaded as aws-prod-cutover-snapshot-<run id> and kept 90 days. The snapshot’s SHA-256 and the existing record ID feed the DNS stack.
3. Initializes both the core and the DNS state and re-runs the pre-cutover smoke against the live targets.
4. Plans the DNS stack. It reads the ALB name from the core state, imports the existing record when there is one, and prod-dns-plan-guard.sh refuses anything but exactly one proxied record for this hostname pointing at this ALB. The resource itself carries preconditions on confirm and on the snapshot hash, and prevent_destroy.
5. Snapshots the record again immediately before apply and fails when it changed since step 2.
6. Applies, then runs prod-smoke.sh public through Cloudflare.

rollback

Dispatch with confirm=<hostname> and rollback_run_id, the deploy run whose snapshot to restore. The reviewer approves, then:

1. Guards as for deploy, with rollback_run_id in place of the plan run. Downloads that run’s rollback.json.
2. Runs prod-preflight.sh rollback and restores the exact previous task definition on the service, outside Terraform.
3. Plans and applies the core stack with the previous image, so the state agrees with what runs, and waits for the service to be stable.
4. Runs the pre-cutover smoke, and the public one when the hostname resolves.

Every step after the guard shares one budget: the service must be healthy within 600 seconds of the run starting or the rollback fails. DNS is not touched.

The smokes

A runner cannot reach the ALB, since the security group admits Cloudflare alone. So scripts/aws/prod-smoke.sh precutover reads target health from AWS, and public drives the hostname through Cloudflare with the same capability assertion just infra::image::verify makes. Both scan the task’s log group for tracebacks since the run started. scripts/aws/test-prod-guards.sh exercises every refusal these scripts make, hermetically.

Gotchas

• A deploy is not a cutover. Flipping the Bedrock switch, or any other task definition change, is a plan and a deploy. The public record moves only on cutover, which typically runs once.
• Plan and deploy must be the same commit. A merge to main between the two fails the provenance check, so run plan again.
• The first deploy is special. There is no rollback snapshot, and the core apply creates the ACM validation record in Cloudflare.
• The budget brake is slow. With Bedrock on, the deny lands hours after the threshold, and it stays until an operator resets the action.
• Editing the deploy policy is a bootstrap re-apply, by an administrator, not by the workflow.