My take: the concern is valid, but "therefore stop testing" is the wrong conclusion. The criticism has a legitimate core. The testing industry has been slow to reckon with acquisition costs. If your test "wins" on conversion rate but the winning experience makes your CAC worse, you might have won nothing — that's a real blind spot. But every single change you introduce might affect your CACs; stopping to measure their impact is not a solution. What's actually going on in most cases: paid media and on-site testing run on completely different clocks. Meta sees a 5% CAC shift and someone acts that day. A valid on-site test usually needs weeks of traffic. They're measuring different things in different places, and when paid wobbles during a test window, it's easy to blame the test. But there's something many people miss: if your numbers hurt a little while the test runs, but you walk away knowing something you'll use for the next 6, 12, 18 months... that's not a loss. That's the cost of learning something real. A temporary CAC spike that teaches you the right price point is a bargain. What I'm curious to hear from you: - Have you seen any true correlation between running tests and CAC spikes? What did you do about it? - Has this changed how you structure your program at all? - How are you handling this conversation with stakeholders? Our gut says the CAC impact is probably overstated... but that's not always how it reads in the room. How do you navigate that?

@Dave Diederen came on Live with Intelligems a few weeks back. He runs Syntralabs, a CRO and experimentation agency that's run a serious volume of tests across a lot of different Shopify stores. That's an enormous amount of pattern recognition to build up, store by store, test by test. So when he decided to launch his own brand, the plan should have been simple: apply everything he already knows. Instead he froze. He described it as information overload, feeling like a newbie discovering CRO again, stuck rebuilding the same page over and over because he kept imagining what a stranger would think looking at it. What got him moving again was lowering the bar on purpose. Early on, he told himself, the ad and the creative do most of the work. The site doesn't have to be the best-converting page ever built, it has to be good enough to not get in the way while the real signal comes from the ads. So he shipped it as is and let the data, not his own judgment, tell him what to fix next. There's a nuance worth sitting with here. His edge wasn't wasted, it just showed up somewhere else. All those reps paid off in the label, the copy, knowing which small change usually wins. The site was the one place his expertise worked against him, because he could see every way it might be wrong before a single visitor ever landed on it. The full interview is attached below if you want to watch the whole conversation. Curious if this lands with anyone here. Has more experience ever made it harder for you to ship something of your own, rather than easier? What's the "done, not perfect" moment that actually got you unstuck? And where does your own expertise turn into a disadvantage instead of an edge?

Linda Bustos came on Live with Intelligems recently. She runs ecomideas.com, a database of over a thousand of the boldest, weirdest, "I can't believe they shipped this" design patterns in ecommerce. The kind of thing you screenshot and drop in a Slack channel. Here's the idea worth putting in front of this group. A lot of the boldest patterns she's archived don't last. The animated PDP gets stripped back to text. The stepped cart cross-sell gets reverted to a simple PDP upsell. And when we see that as testers, we fill in the story ourselves: it lost the test, so they killed it. But how would we actually know? Linda's point is that conversion is usually not the reason. Sometimes the creative team turned over and nobody understood the feature. Sometimes it's a performance or accessibility cost. Sometimes a replatform is coming and nobody wants to rebuild it. Sometimes engagement was low and it quietly got dropped. The feature disappearing tells you almost nothing about whether it worked. That cuts both ways. When a competitor ships something bold, you can't assume it's winning either. Which is her whole case for treating other brands as inspiration, not instruction. A pattern that's a founder-endorsed winner for a tight hero-product catalog, like True Classic switching product types right on the PDP, might fall apart on a 30,000-SKU store. Same idea, different catalog, opposite result. So the skill isn't spotting the clever idea. It's knowing which one is worth a test slot on your store, and testing it instead of reading a competitor's roadmap off their live site. The full interview is attached below if you want to watch the whole conversation. Curious how this community handles it. When you see a competitor kill a feature you liked, do you read anything into it or ignore it? How do you decide a hot idea from another brand earns the test slot on your own site? And has a pattern ever crushed it for a brand you admired and then flopped when you tried it?

Nobody in this community needs convincing that conversion rate isn't the whole story. Profit per visitor over CVR, we mostly get it. When we had @Luka Nikolić on Live with Intelligems recently, the interesting part wasn't the pitch for it, it was how much time he spent on why it's so hard to actually run a program this way. His answer came down to data and financial literacy. Getting profit into a test means getting COGS out of a client, and that's a harder conversation than it sounds, especially when they aren't used to sharing it. He's had to fight for those uploads. An agency can run a retainer for months, tweaking buttons, and never touch the number that actually decides whether the business is healthier, because nobody ever put the real financials in front of them. His pitch was to close the laptop on the AI tools for a bit and go read the boring spreadsheets instead. Return rates, supplier terms, how customers actually segment. The unglamorous stuff practitioners tend to skip is usually where the profit lever is hiding. His line for it: "revenue is vanity, profit is sanity." The full interview is attached below if you want to watch the whole conversation. Curious where this community lands. For those tracking profit per visitor, how did you get COGS out of a client who didn't want to hand it over? Have you ever shipped a test that won on conversion rate and later realized it hurt margin? And is there a "boring spreadsheet" you've gone into that ended up reshaping what you tested next?