Production rollout — step by step

Sequenced so nothing downstream is attempted before its prerequisite is actually true, not just merged. Compare against today's state and the target state.

Commands below use dynamic lookups (--query by name) instead of hardcoded ARNs/IDs, so they work regardless of when you run them.

1. Backend registration — done

Every target group's backend lookup previously searched for a Name tag that didn't match any real EC2 instance. All 7 prod filter values are now corrected and pushed, verified two independent ways before landing. (The 4 equivalent staging lookups were dropped entirely rather than fixed, see staging's target state, real target groups already existed.)

1. Run terraform plan and confirm all 7 prod backend lookups resolve to exactly one instance each.

terraform plan \
  -target='data.aws_instance.campaign_1' \
  -target='data.aws_instance.campaign_2' \
  -target='data.aws_instance.campaign_3' \
  -target='data.aws_instance.audience_manager_1' \
  -target='data.aws_instance.audience_manager_2' \
  -target='data.aws_instance.collector' \
  -target='data.aws_instance.automated_testing'

Each data source should resolve with no error. A Your query returned no results or multiple results error means a Name tag filter is still wrong.

2. Land the baseline and certs

1. Merge and apply the ALB baseline change (idle_timeout, TLS policy) — PR #25. Lowest-risk change in the whole rollout, no data sources, no conditionals.

ALB_ARN=$(aws elbv2 describe-load-balancers \
  --names alb-apse2-prod-server \
  --query 'LoadBalancers[0].LoadBalancerArn' --output text)

aws elbv2 describe-load-balancer-attributes \
  --load-balancer-arn "$ALB_ARN" \
  --query "Attributes[?Key=='idle_timeout.timeout_seconds'].Value" \
  --output text
# expect: 120

aws elbv2 describe-listeners --load-balancer-arn "$ALB_ARN" \
  --query 'Listeners[?Protocol==`HTTPS`].SslPolicy' --output text
# expect: ELBSecurityPolicy-TLS13-1-2-2021-06

1. Merge and apply the ACM certificate + DNS validation resources — PR #26. Wait for all three certs to reach ISSUED (typically 15–60 minutes) before proceeding.

aws acm list-certificates \
  --query "CertificateSummaryList[?DomainName=='arep.co' || DomainName=='arep.cc' || DomainName=='arep.ink'].{Domain:DomainName,Status:Status}" \
  --output table
# every row must read ISSUED before step 7

3. Build the ALB configuration

1. Merge and apply target groups and backend attachments (now unblocked by step 1) — PR #27. Confirm every target group shows healthy for all its registered targets.

ALB_ARN=$(aws elbv2 describe-load-balancers \
  --names alb-apse2-prod-server \
  --query 'LoadBalancers[0].LoadBalancerArn' --output text)

for TG_ARN in $(aws elbv2 describe-target-groups \
    --load-balancer-arn "$ALB_ARN" \
    --query 'TargetGroups[].TargetGroupArn' --output text); do
  NAME=$(aws elbv2 describe-target-groups --target-group-arns "$TG_ARN" \
    --query 'TargetGroups[0].TargetGroupName' --output text)
  echo "== $NAME =="
  aws elbv2 describe-target-health --target-group-arn "$TG_ARN" \
    --query 'TargetHealthDescriptions[].[Target.Id,TargetHealth.State,TargetHealth.Reason]' \
    --output table
done

tg-apse2-prod-monitor is expected to show unhealthy — monitor-backend is confirmed dead (see Issue Resolved), not a rollout blocker.

1. Merge and apply the WAF Web ACL (IP denylist + Googlebot rule, scoped only to automated.arep.co) — PR #28.

aws wafv2 get-web-acl-for-resource --resource-arn "$ALB_ARN" \
  --query 'WebACL.{Name:Name,Id:Id}'

1. Merge and apply listener rules and SNI certificate attachments — PR #29. Includes the exact-path / redirect to https://audiencerepublic.com on arep.co (nginx gives exact matches priority over its prefix catch-all; ALB doesn't, so this needed its own rule) and the 4 routes automated.arep.co's live nginx config has that the original rule set didn't replicate (/privacy, /terms, /share-analytics/, /maintenance) — its 2 static-asset path patterns needed no new rule, they already resolve to the same target group the default rule forwards to.

LISTENER_ARN=$(aws elbv2 describe-listeners --load-balancer-arn "$ALB_ARN" \
  --query 'Listeners[?Protocol==`HTTPS`].ListenerArn' --output text)

aws elbv2 describe-rules --listener-arn "$LISTENER_ARN" \
  --query 'Rules[].[Priority,Conditions[0].Values[0]]' \
  --output table
# confirm priorities are gap-free / not colliding, and priority 9
# (arep.co exact-root redirect) is present ahead of everything else

1. Merge and apply CloudWatch alarms (per-target-group health, ALB-level 5xx/latency) — PR #30.

aws cloudwatch describe-alarms \
  --query "MetricAlarms[?contains(AlarmName, 'prod')].[AlarmName,StateValue]" \
  --output table

tg_prod_monitor's 2 alarms are expected to already be in ALARM state (same dead backend as step 5) — everything else should be OK or INSUFFICIENT_DATA until real traffic starts flowing.

4. Shadow test, no DNS changes yet

1. Get the ALB's own IP addresses and run the full hostname/path smoke-test matrix against it directly, for every path rule and every hostname. The listener's default action stays a 503 fixed-response until this passes, zero live-traffic exposure during this phase.

ALB_DNS=$(aws elbv2 describe-load-balancers \
  --names alb-apse2-prod-server \
  --query 'LoadBalancers[0].DNSName' --output text)
ALB_IP=$(dig +short "$ALB_DNS" | head -1)

for path in "/api/v1/cn/health" "/api/v1/am/health" "/api/v1/sg/health" \
            "/api/v1/ingest/whatever" "/share-analytics/whatever" \
            "/css/main.css" "/maintenance" "/privacy" "/terms" "/support" "/"; do
  printf "%-45s " "arep.co${path}"
  curl -s -o /dev/null -w "HTTP=%{http_code}\n" --max-time 5 \
    --resolve "arep.co:443:${ALB_IP}" "https://arep.co${path}"
done

for host in arep.cc arep.ink automated.arep.co; do
  printf "%-45s " "${host}/"
  curl -s -o /dev/null -w "HTTP=%{http_code}\n" --max-time 5 \
    --resolve "${host}:443:${ALB_IP}" "https://${host}/"
done

1. Probe the WAF rules directly (spoofed Googlebot UA against in-scope and out-of-scope hostnames, IP denylist test).

   # Googlebot must 403 on automated.arep.co (in-scope):
   curl -s -o /dev/null -w "automated.arep.co Googlebot -> HTTP=%{http_code}\n" \
     -A 'Googlebot/2.1 (+http://www.google.com/bot.html)' \
     --resolve "automated.arep.co:443:${ALB_IP}" "https://automated.arep.co/"
   
   # Googlebot must NOT 403 on out-of-scope hostnames:
   for host in arep.co arep.cc arep.ink; do
     curl -s -o /dev/null -w "${host} Googlebot -> HTTP=%{http_code}\n" \
       -A 'Googlebot/2.1 (+http://www.google.com/bot.html)' \
       --resolve "${host}:443:${ALB_IP}" "https://${host}/"
   done
   

2. Confirm TLS 1.2 and 1.3 handshakes succeed and TLS 1.1 is rejected.

   for v in tls1_3 tls1_2; do
     echo "== $v =="
     openssl s_client -connect "${ALB_IP}:443" -servername arep.co -$v \
       </dev/null 2>&1 | grep -E "Protocol|Cipher"
   done
   
   echo "== tls1_1 (must fail) =="
   openssl s_client -connect "${ALB_IP}:443" -servername arep.co -tls1_1 \
     </dev/null 2>&1 | tail -3
   

5. Cut over, lowest blast radius first

1. Lower all six hostnames' DNS TTLs to 60 seconds, 24 hours ahead of the first cutover.

   for host in arep.ink arep.cc staging.arep.co staging.arep.ink automated.arep.co arep.co; do
     printf "%-22s " "${host}"
     dig +short +ttl "${host}" @8.8.8.8 | head -1
   done
   # all should print 60 once the TTL change has propagated
   

2. arep.ink → single-record DNS swap to the ALB. Watch 30 minutes.

   ZONE_ID=$(aws route53 list-hosted-zones-by-name --dns-name arep.ink \
     --query 'HostedZones[0].Id' --output text)
   
   aws route53 change-resource-record-sets --hosted-zone-id "$ZONE_ID" \
     --change-batch '{"Changes":[{"Action":"UPSERT","ResourceRecordSet":
       {"Name":"arep.ink","Type":"A",
        "AliasTarget":{"HostedZoneId":"'"$(aws elbv2 describe-load-balancers --names alb-apse2-prod-server --query 'LoadBalancers[0].CanonicalHostedZoneId' --output text)"'",
                       "DNSName":"'"$ALB_DNS"'",
                       "EvaluateTargetHealth":true}}}]}'
   
   # watch: dig +short arep.ink @8.8.8.8   (should return ALB IPs)
   #        curl -sI https://arep.ink/     (should return 200)
   

3. arep.cc → same shape, substituting the arep.cc zone id and hostname. Watch 30 minutes.

4. automated.arep.co → same shape, after re-confirming the Googlebot WAF rule still fires correctly for this specific hostname (rerun step 10's automated.arep.co probe).

5. arep.co apex → weighted cutover, not a single-record swap: 5% ALB / 95% gateway1, watch 15–30 min, step to 50/50, watch, step to 100/0. Paired execution, two people, one driving.

   ZONE_ID=$(aws route53 list-hosted-zones-by-name --dns-name arep.co \
     --query 'HostedZones[0].Id' --output text)
   
   # Step to any weight split by changing gateway1/alb Weight values below.
   aws route53 change-resource-record-sets --hosted-zone-id "$ZONE_ID" \
     --change-batch '{"Changes":[
       {"Action":"UPSERT","ResourceRecordSet":
         {"Name":"arep.co","Type":"A","SetIdentifier":"gateway1","Weight":95,"TTL":60,
          "ResourceRecords":[{"Value":"52.62.81.96"}]}},
       {"Action":"UPSERT","ResourceRecordSet":
         {"Name":"arep.co","Type":"A","SetIdentifier":"alb","Weight":5,
          "AliasTarget":{"HostedZoneId":"'"$(aws elbv2 describe-load-balancers --names alb-apse2-prod-server --query 'LoadBalancers[0].CanonicalHostedZoneId' --output text)"'",
                         "DNSName":"'"$ALB_DNS"'",
                         "EvaluateTargetHealth":true}}}
     ]}'
   
   # watch each step:
   aws cloudwatch get-metric-statistics --namespace AWS/ApplicationELB \
     --metric-name HTTPCode_ELB_5XX_Count --statistics Sum --period 300 \
     --start-time "$(date -u -v-15M +%Y-%m-%dT%H:%M:%S)" \
     --end-time "$(date -u +%Y-%m-%dT%H:%M:%S)" \
     --dimensions Name=LoadBalancer,Value="$(aws elbv2 describe-load-balancers --names alb-apse2-prod-server --query 'LoadBalancers[0].LoadBalancerArn' --output text | cut -d/ -f2-)"
   

   Rollback (if 5xx/latency exceeds baseline at any step) — zero the ALB weight, never UPSERT a plain unweighted record, which would destroy the SetIdentifier pair:

   aws route53 change-resource-record-sets --hosted-zone-id "$ZONE_ID" \
     --change-batch '{"Changes":[
       {"Action":"UPSERT","ResourceRecordSet":
         {"Name":"arep.co","Type":"A","SetIdentifier":"gateway1","Weight":100,"TTL":60,
          "ResourceRecords":[{"Value":"52.62.81.96"}]}},
       {"Action":"UPSERT","ResourceRecordSet":
         {"Name":"arep.co","Type":"A","SetIdentifier":"alb","Weight":0,
          "AliasTarget":{"HostedZoneId":"'"$(aws elbv2 describe-load-balancers --names alb-apse2-prod-server --query 'LoadBalancers[0].CanonicalHostedZoneId' --output text)"'",
                         "DNSName":"'"$ALB_DNS"'",
                         "EvaluateTargetHealth":true}}}
     ]}'
   

6. Decommission

1. 48-hour observation window with gateways still running, nginx still active, watching dashboards only.

2. Confirm zero real traffic on gateway access logs.

   ssh -i ~/.ssh/ar_ssh ts-deploy@<gateway1-ip> \
     'sudo tail -1000 /home/ts-deploy/nginx/log/*access*.log | wc -l'
   # expect: 0, or only health-check traffic
   

3. Terminate gateway-short-url (already has zero traffic today, doesn't need to wait for this point, but formal termination happens here).

4. Decide gateway1/gateway2's fate before proceeding, downsize or fully retire. This depends on confirming whether app.audiencerepublic.com genuinely no longer needs them (a separate audit suggests it doesn't, since it's served by Netlify), confirming what happens to <promoter>.portal.arep.co, which isn't part of this rollout and currently depends on gateway1 existing in some form (and separately needs its expired TLS cert renewed — see Issue Resolved), and confirming /zoomverify/'s migration to S3+CloudFront, since it's still served directly from gateway disk today.