Most on-call runbooks are lies. They were written eighteen months ago by an engineer who has since left, they reference a load balancer that was replaced during the last migration, and the one command that actually mattered was never written down because the person who knew it just typed it from memory every time. We all know this. We still page people at 3am and point them at a wiki page nobody has opened since the last audit. I spent the last few weeks attacking this from a different angle. Instead of trying to keep a library of static runbooks fresh, I started generating them on demand from the alert payload itself. The alert already contains almost everything you need. A CloudWatch alarm tells you the metric, the namespace, the exact dimensions, the threshold, the evaluation window, the datapoints that breached, the account and the region. A PagerDuty incident carries the service, the escalation policy, the priority and the full trigger log entry. A Datadog monitor gives you the query, the scope and the tags. That is dense, structured, high signal context. It is exactly the kind of input a language model reasons over well. The pipeline is boring in the best way. Alert fires. Webhook or SNS topic hands the JSON to a small script. The script normalises the payload into a common shape, merges it with a static environment context file that describes your org, your clusters, your escalation tiers and your hard rules, and sends the whole thing to Claude with a strict output contract. Ninety seconds later the responder has a document with a summary, immediate actions, a ranked table of diagnostic commands, escalation criteria, three hypothesised root causes with remediation and rollback for each, and a pre-filled post-mortem template. The part that surprised me was the diagnostic ordering. I asked the model to sort commands by information gain per second rather than by category, and to state which hypothesis each command discriminates between. That single instruction turned a generic checklist into something that reads like a decision tree. For a checkout API 5xx alarm it opened with the target group health check and the deployment history, not with CPU graphs, because CPU is almost never the discriminator on a sudden error rate spike.

Last Tuesday I was about to merge a 400-line Terraform PR into our staging account. Two engineers had already approved it. Plan looked clean. 41 to add, 6 to change, 0 to destroy. Before I hit merge, I pasted the diff into Claude with one prompt: "You are a senior AWS platform engineer doing a pre-merge Terraform review. Review the HCL below for security misconfigurations, cost impact, and state-management risks. For each finding give me: severity, the exact resource address, why it matters, and the corrected HCL. Do not comment on style." Thirty seconds later I had three findings that both human reviewers missed. FINDING 1 - CRITICAL Our aws_security_group for the RDS instance had an ingress rule with cidr_blocks = ["10.0.0.0/8"] on port 5432. Looks private, right? Except our VPC is 10.0.0.0/16 and we're peered with two other 10.x VPCs including the one our contractors use. Claude flagged it and rewrote the rule to reference the app tier's security group ID instead of a CIDR. Blast radius went from "three VPCs" to "one security group." FINDING 2 - HIGH (cost) Someone had set an aws_db_instance to db.r6g.2xlarge with multi_az = true for a service handling roughly 40 queries a second. Claude did the math out loud: about $1,050/month in us-east-1 before storage and I/O, and the workload profile fits a db.r6g.large. It also pointed out storage_type was left at gp2 instead of gp3, which is both slower and more expensive at that size. Combined annualized delta: just under $9,600. FINDING 3 - CRITICAL (state) This is the one that scared me. The PR renamed a module from module.network to module.vpc_core. No moved block. Terraform plan in CI ran against a stale workspace and didn't show it. In our real state, that rename would have destroyed and recreated the NAT gateways and every route table attached to them. Claude caught it purely from reading the diff and wrote the moved blocks for me. Here's the part that matters: none of these are exotic. They're the same three categories that show up in almost every Terraform review — overly broad network rules, oversized instances nobody revisits, and refactors that quietly move state.

Last month a client's AWS bill jumped $4,300 in nine days. AWS Cost Anomaly Detection caught it on day two and sent a perfectly correct, perfectly useless email: "Anomaly detected. Service: Amazon Elastic Compute Cloud. Impact: $312." Nobody opened it. It looked like the other 40 cost emails that month. Here is the thing about AWS Cost Anomaly Detection: the detection part works. It uses a machine learning model on your historical spend, it's free, and it will find the spike. What it will not do is tell you WHY. It gives you a service name, a region, a usage type, and a dollar figure. Then it stops, and a human has to go do 45 minutes of archaeology across CloudTrail, Auto Scaling activity, and the tagging report. That gap is where the Claude API fits. Not as a detector - AWS already detects. As a translator and a triage layer. The pattern is three steps and it takes about 200 lines of Python: 1. boto3 client for ce (Cost Explorer is global, pin it to us-east-1). Call get_cost_and_usage with DAILY granularity, 30-day window, grouped by SERVICE. Then call get_anomalies to pull what AWS already flagged. 2. Format that into a compact text block - anomaly window, actual vs expected spend, root cause dimensions, and the 14-day daily series per service. 3. Send it to Claude with a system prompt that says "you are a senior FinOps engineer writing for an on-call engineer who needs to act in 15 minutes." What comes back on the real bill I mentioned: "NAT Gateway data processing charges in eu-west-1 rose from $18/day to $340/day starting July 22, while EC2 compute stayed flat. Flat compute plus rising NAT egress means traffic pattern change, not scale-up. Most likely cause (high confidence): a workload started routing S3 or ECR traffic through the NAT Gateway instead of a VPC endpoint. Verify with: aws ec2 describe-vpc-endpoints. Action: add a Gateway endpoint for S3, estimated saving $9,600/year." That's the difference. The alert said EC2 is expensive. Claude said you're missing a VPC endpoint.

A friend of mine runs platform engineering at a mid-size fintech. In March they got an alert that a production S3 bucket had been emptied. That was the moment they found out they'd been compromised. When they pulled the CloudTrail logs and walked backwards, the actual break-in had happened seventy-one hours earlier. Here's what those three days looked like. Day one, an access key belonging to a contractor who'd left eight months earlier gets used from a residential IP in a country the company doesn't operate in. It makes exactly four API calls, all of them read-only. ListRoles, ListAttachedRolePolicies, GetRole, SimulatePrincipalPolicy. Nothing fires, because nothing should fire on a read. Day two, the same key calls iam:PassRole and lambda:CreateFunction, eleven minutes apart. Individually both of those are things their deploy pipeline does forty times a day. Day three, the Lambda gets invoked once, it assumes a role with AdministratorAccess, mints a new access key, and then the fun starts. Total time from first suspicious call to full admin: about seventy-two hours of extremely patient, extremely quiet work. Their SIEM saw every single one of those events. It just never saw a reason to care about any of them. That's the part that keeps me up. The logs were perfect. The detection was the failure. The reason static rules lose this fight is that privilege escalation isn't an action, it's a sentence made out of actions, and the grammar is enormous. You can write a rule for iam:CreateUser followed by iam:AttachUserPolicy followed by iam:CreateAccessKey, and you should. But then the attacker uses iam:CreatePolicyVersion plus iam:SetDefaultPolicyVersion to quietly rewrite a policy that's already attached to something they control, and your rule never fires. So you write another rule. Then they use iam:UpdateAssumeRolePolicy to add themselves to a role's trust policy and just assume it. Another rule. Then iam:PassRole with ec2:RunInstances and they read the credentials off the instance metadata service. Another rule. There are somewhere north of thirty documented escalation paths in AWS and the number of two-and-three step orderings across them runs into the thousands. You will always be writing the rule for the technique they used last time.

Welcome. If you just joined, here's the 3-minute version of how to use this place. 1. Classroom → pick one lesson. Every lesson is a real build (Terraform drift agent, K8s cost optimizer, AWS anomaly detector) with working code, not slides. Start with whichever problem you actually have this week. 2. Comment on this post with what you're working on right now — cost overruns, drift, K8s chaos, IAM sprawl, anything. I read every one and it shapes what gets built next. 3. Feed gets a new real-world project weekly. Free tier = tips, tutorials, Q&A. Paid tier (Classroom) = full labs, scripts, and templates you keep. This only works if it's not a monologue — so tell me what's breaking in your stack. What are you fighting with right now?