switchcasestudios.com
The productivity gains are real, measurable, and conditional — and quietly dependent on a human who knows what they're doing at the controls.
The same headline keeps coming around, just with the noun swapped out. AI replaces developers. AI replaces analysts. AI replaces your whole marketing team. It makes for a great tweet and a terrible strategy.
The good news: we no longer have to argue from vibes. There's a solid stack of real studies — randomized trials, field experiments, large-scale telemetry — and they all rhyme. Start with the study that kicked the field off.
Erik Brynjolfsson and colleagues looked at 5,179 customer support agents at a Fortune 500 software firm as they rolled out a generative-AI assistant. Agents with the tool resolved 14% more issues per hour on average. Customers were happier, agents quit less, and resolution quality held steady. Brynjolfsson's own reaction is the telling part — he'd spent years studying IT rollouts where a 1–2% productivity gain was cause for celebration. Fourteen percent isn't an improvement, it's a regime change.
Then there's the BCG / Harvard "jagged frontier" experiment — 758 management consultants, randomized, given real consulting tasks. For work that sat inside what the AI was good at, consultants using GPT-4 finished 25% faster, completed 12% more tasks, and produced noticeably higher-quality output. These aren't entry-level temps — BCG admits roughly 1% of applicants — and the AI still moved the needle hard.
So far this is the slide every vendor shows you. The next section has the one they don't.
In 2025, the research nonprofit METR ran a proper randomized controlled trial — the clinical-drug-trial kind — on 16 experienced open-source developers working through 246 real tasks on codebases they'd lived in for an average of five years.
Going in, the developers predicted AI would make them 24% faster. After they finished, they reported feeling about 20% faster. The actual result:
Let that gap sink in, because it's the single most important finding in this entire space. It's not just that AI didn't help these particular developers — it's that they had no idea. The acceleration was a feeling, not a fact. Generate code, watch it appear instantly, feel fast — but the clock doesn't lie.
Before anyone screenshots that chart with a "SEE? AI IS USELESS" caption — slow down. The METR result isn't the opposite of the Brynjolfsson result. It's the same lesson from a different angle.
These were senior engineers on codebases they knew cold, holding work to a high standard — exactly the population that gained the least in every other study. Brynjolfsson found the 14% average masked a 34% jump for novices and basically nothing for experts. The BCG consultants only won on tasks inside the AI's frontier; push them outside it and the AI made them 19% less likely to get the right answer. METR's developers spent a big chunk of their "saved" time cleaning up generated code that didn't meet their standards.
AI's value depends entirely on who's holding it and what you point it at. That's not a knock on the tools — that's the definition of a power tool.
— SWITCHCASE STUDIOS · AI PRACTICEIf AI were actually a replacement, we'd be shipping its output straight to production. We are emphatically not doing that.
None of this means the tools are bad. The workflow that actually works has a human in the loop by design — not as a bottleneck, but as the part of the system that supplies judgment, taste, context, and a sense of what "done" means. The AI drafts; you decide. That's the whole game.
This is also why the "just let it run" agentic fantasy is dangerous. An agent can burn through real money grinding on a problem a human would've abandoned after the second try, because the agent doesn't know when it's stuck. Knowing when to stop is judgment. Judgment is still ours.
The BCG researchers put labels on people who use AI well, and they've stuck. Centaurs split the work cleanly — human does the parts humans are good at, AI handles the rest, with a clear boundary. Cyborgs blend continuously, handing micro-tasks back and forth in a tight loop. Both beat the people who either ignored the AI or trusted it blindly. The consistent losers were the ones who outsourced their judgment.
AI shines on boilerplate, first drafts, unfamiliar territory, and high-volume repetitive tasks. It struggles on deep, context-heavy work in domains you already know cold. Know which task you're on before you reach for the tool.
The feeling of being faster is not evidence of being faster. If it matters, measure it. The most dangerous person on your team is the one who's certain the AI doubled their output and has never once checked.
The gains live downstream of human verification. Skip the review and you're not saving time — you're just moving the cost to whoever debugs it in production at 2 a.m. (Probably also you.)
The Brynjolfsson study's most striking finding: AI worked like a knowledge equalizer — it handed novices the patterns of the best workers and pulled them up the experience curve fast. Used that way, it's a teacher. Used as a crutch, it's a way to never learn anything.
AI isn't coming for your job. A person who's genuinely good with AI might be — and the difference between those two sentences is the entire point.
Accelerator, not replacement. Power tool, not new hire. Boring framing — but it's the one the data keeps backing up.
The productivity gains are real. So is the productivity placebo. So is the 1.7× defect rate on unreviewed AI code. What unifies all of it is that the human is still the critical variable — in skill, in judgment, and in the ability to know which tool to reach for, when, and on what.