Production rollback

Rollback isn't one procedure. Which command is correct depends on which Route 53 routing policy is live for that hostname at the moment you're rolling back, and cutover itself changes that policy, so the answer is different before cutover starts, during it, and if something needs reversing afterward.

What each hostname uses today, before any cutover

Hostname	Current routing policy	Current target(s)
arep.co	Failover	Primary → gateway1, Secondary → gateway2
arep.cc	Weighted	200 → gateway2, 0 → 52.64.21.46 (dead EIP, dangling health check)
arep.ink	Simple	Single A record → gateway2
automated.arep.co	Weighted, single entry	200 → 52.64.21.46 (dead EIP, dangling health check)

None of these are the shape cutover introduces. Cutover replaces whichever policy is live today with a new Weighted pair, SetIdentifier: "gateway" and SetIdentifier: "alb", so the gradual traffic shift (5% → 50% → 100%) has something to shift between. Once that new weighted pair exists, it's what rollback has to target, not the original Failover/Simple/Weighted shape from the table above.

Rollback once cutover has introduced the new weighted pair

# Zero the ALB weight, restore the gateway to 100. Both records survive, untouched structurally.
aws route53 change-resource-record-sets \
  --hosted-zone-id <zone id> \
  --change-batch '{"Changes":[
    {"Action":"UPSERT","ResourceRecordSet":
      {"Name":"<hostname>","Type":"A","SetIdentifier":"gateway","Weight":100,"TTL":60,
       "ResourceRecords":[{"Value":"<gateway IP>"}]}},
    {"Action":"UPSERT","ResourceRecordSet":
      {"Name":"<hostname>","Type":"A","SetIdentifier":"alb","Weight":0,
       "AliasTarget":{"HostedZoneId":"<ALB hosted zone>",
                      "DNSName":"<alb dns name>",
                      "EvaluateTargetHealth":true}}}
  ]}'

Never UPSERT a single plain record over a weighted pair

Once cutover has created the two-record weighted pair, replacing it with one plain, unweighted A record deletes both SetIdentifier entries. The weighted structure is gone, and re-enabling the ALB later means rebuilding the pair from scratch, not just nudging a number, exactly the wrong thing to be figuring out mid-incident. Always UPSERT both records back with adjusted weights, never delete, never replace with a single record.

Apex arep.co specifically

The apex uses the same weighted-pair rollback shape as every other hostname once cutover has begun, the only difference is that today, before cutover, it's on Failover routing, not Weighted. Cutover itself is what converts it to a weighted pair. There's nothing to "roll back" about the Failover policy specifically, that policy simply stops being used the moment cutover starts.

When to actually pull the trigger

Decide the threshold before starting, not during. If HTTPCode_Target_5XX_Count or TargetResponseTime p99 exceeds the pre-cutover baseline materially at any watch checkpoint, roll back immediately rather than waiting to see if it self-resolves. Investigation happens after, off the critical path.