There's a stage between "built" and "live" that I used to skip, and skipping it taught me the same lesson more than once: an automation can produce plausible output for days before it produces a noticeably wrong one. So new systems run in shadow first. They do the full job — sort, draft, prepare — but nothing they produce reaches anyone. I compare their output against what I did by hand, on the same inputs, for a set number of runs. Three things come out of the shadow period that I can't get any other way: - The real error rate, not the demo error rate. Demos are chosen; real inputs are not. - The *shape* of the mistakes. Random slips I can guard. A consistent misunderstanding means the instructions are wrong, not the model. - A feel for which outputs need a hard look and which are safe to skim — which is what makes the eventual review step fast instead of theatrical. Promotion out of shadow is boring on purpose: a fixed number of clean runs, not a vibe. Boring is what trustworthy looks like from the inside. Do you have a trial stage for new automations, or do yours go straight to live and earn trust in public?

Here's a pattern I see in almost every small business I talk workflows with: the first response is fast, the service is good — and then silence. The quote that never got chased. The "let me think about it" that nobody revisited. Not because anyone's lazy, but because follow-up lives in somebody's memory, and memory is a terrible system. The version I build is deliberately boring: 1. Every open conversation gets a state: waiting on us, waiting on them, or closed. If it's not written down somewhere a machine can read, it doesn't exist. 2. "Waiting on them" past a set number of days triggers a drafted nudge — referencing the actual conversation, not a generic "just checking in." 3. A human approves the nudge before it sends. Tone matters too much here for full automation; a pushy follow-up is worse than none. 4. Two nudges with no reply closes the loop politely and moves on. Chasing forever is how you train people to ignore you. The technology here is trivial. What makes it work is the state list being complete — which means the unglamorous work of making sure every conversation actually gets recorded. Where does follow-up break down in your business — remembering, deciding what to say, or just finding the time?

A quick war story from building inbox systems: the drafts were good. Tone right, structure right, customers helped. Except for one recurring disaster class — prices. A draft would quote a price from an old email thread the system had read, the human reviewer would be moving fast, and out went a number the business hadn’t charged in a year. The root cause wasn’t the AI. It was that the business’s prices lived in eleven places — a website, a PDF, someone’s memory, old threads — and the system grazed on all of them. The fix was boring and total: one canonical list. Prices, hours, policies, service areas — one document, owned by the business, updated in exactly one place. The drafting system reads only that document. Old threads became history, not sources. When a price changes, it changes in one spot, and every future draft is right. Review got faster too. When a reviewer knows the numbers could only have come from the canonical list, they stop re-verifying every figure and spend their attention on tone and fit — the things review is actually for. If your business’s key facts live in more than one place, which copy would your systems (or your team) quote — and is it the current one?

When people demo inbox AI, they show the reply. Nobody demos the sort — but in my experience the sort is where the value (and the risk) actually lives. Before any drafting happens in the systems I build, every message gets classified: booking, pricing, support problem, complaint, sales pitch, or unknown. That one label decides everything downstream — which approved answers are even eligible, how fast it needs a human, and whether it should be drafted at all. Three things I learned getting the sort right: 1. Complaints never get drafts first. They get flagged for a human immediately. A fast, templated-feeling response to an unhappy customer is worse than a slower human one. 2. "Unknown" is a real category, not a failure. Forcing every message into a known bucket is how a refund demand ends up answered with a price list. A system that can say "I don't know what this is" is safer than a confident one. 3. The mis-sorts are the roadmap. Every week I look at what landed in the wrong pile. Fixing one recurring mis-sort improves every future message of that type, forever. The reply is the visible 20%. If you're building in this space, how are you handling classification — and what does your system do with the messages it can't place?

Confession from my own pipeline: the first version automated almost everything, including two things it never should have touched — picking the hook and doing the final watch-through. The hook problem showed up first. Generated hooks were technically fine: clear, on-topic, grammatically clean. They were also all the same shape, and they read like it. A hook is a judgment call about what a specific viewer is curious about right now; the machine was averaging every hook it had ever seen, and average is the opposite of curious. Hooks went back to being mine. They take ten minutes. They're the ten minutes that decide whether the other hours matter. The final watch-through was worse. I tried replacing it with automated checks — loudness, timing, caption presence, all green — and shipped videos with a mismatched visual sitting on screen for six seconds and a narration pause in exactly the wrong place. Every automated check passed. The video was still wrong in ways only eyes catch. So the pipeline now has two explicitly manual gates: the hook at the front, the watch-through at the end. Everything between them is fair game for automation. What have you deliberately taken back from automation after trying it — and what told you it wasn't working?