Staging rollout — step by step¶

Compare against today's state and the target state. This turned out much shorter than originally planned, once the real infrastructure was found.

1. Merge and apply the six PRs¶

Merge feature/ws2-alb-acm-certs (PR #26), attaches star_arep_ink to alb-apse2-stag-migrationv2 as an additional SNI cert, the one real cert gap that existed (*.arep.co already covered staging.arep.co).
Merge feature/ws2-alb-target-groups (PR #27), corrects all 11 backend Name-tag lookups (7 prod, applies to prod's own target groups) and adds 4 read-only data source lookups referencing migrationv2 and its 3 real target groups. No staging target groups are created.
Merge feature/ws2-alb-waf (PR #28), adds the missing second WAF association, attaching the existing Web ACL to migrationv2.
Merge feature/ws2-alb-listener-rules (PR #29), no staging-specific changes needed here at all, migrationv2's real listener rules already exist.
Merge feature/ws2-alb-cloudwatch (PR #30), adds 6 new alarms across migrationv2's 3 real target groups.

2. Start the actual backends¶

Confirm with whoever owns them that it's safe to start staging/campaign, staging/campaign-2, staging/audience-manager, and staging/deferred-collector, all four are currently stopped.

aws ec2 describe-instances \
  --filters "Name=tag:Name,Values=staging/campaign,staging/campaign-2,staging/audience-manager,staging/deferred-collector" \
  --query 'Reservations[].Instances[].{Name:Tags[?Key==`Name`]|[0].Value,State:State.Name,InstanceId:InstanceId}' \
  --output table

Start them.

aws ec2 start-instances --instance-ids <the 4 instance ids from step 6>
aws ec2 wait instance-running --instance-ids <same ids>

Confirm all 3 real target groups (tg-apse2-stag-campaign-am/cc, tg-apse2-stag-collector-sg) show healthy targets.

STAG_ALB_ARN=$(aws elbv2 describe-load-balancers \
  --names alb-apse2-stag-migrationv2 \
  --query 'LoadBalancers[0].LoadBalancerArn' --output text)

for TG_ARN in $(aws elbv2 describe-target-groups \
    --load-balancer-arn "$STAG_ALB_ARN" \
    --query 'TargetGroups[].TargetGroupArn' --output text); do
  NAME=$(aws elbv2 describe-target-groups --target-group-arns "$TG_ARN" \
    --query 'TargetGroups[0].TargetGroupName' --output text)
  echo "== $NAME =="
  aws elbv2 describe-target-health --target-group-arn "$TG_ARN" \
    --query 'TargetHealthDescriptions[].[Target.Id,TargetHealth.State]' --output table
done

3. Verify¶

Confirm a real response instead of the current 503. No DNS change is needed for this, staging.arep.co already points at migrationv2 today.

curl -sI https://staging.arep.co/
# expect: HTTP/2 200 (or whatever the backend legitimately returns),
# not 503

Confirm the Googlebot WAF rule fires correctly for staging.arep.co now that it's actually attached to the right ALB.

curl -s -o /dev/null -w "staging.arep.co Googlebot -> HTTP=%{http_code}\n" \
  -A 'Googlebot/2.1 (+http://www.google.com/bot.html)' \
  https://staging.arep.co/
# expect: 403

Confirm the new CloudWatch alarms are receiving real metric data.

aws cloudwatch describe-alarms \
  --query "MetricAlarms[?contains(AlarmName, 'stag')].[AlarmName,StateValue]" \
  --output table
# expect OK or INSUFFICIENT_DATA on first check, not stuck ALARM
# once the 3 target groups above are healthy

What this rollout does not include¶

No VPC peering, no DNS delegation, no new hosted zones, no manual cross-team NS record, none of that infrastructure was actually needed. The only real gap was that the existing, already-working ALB had incomplete cert and WAF coverage, and its target groups had stopped backends. Fixing those three things is the entire rollout.

alb-apse2-stag-server, the staging account's own ALB, is untouched by any of this and isn't part of this rollout at all.