Are AI Tools Actually Making Customer Success Teams More Productive?
In fifth grade, I walked into band class, sat down behind an alto saxophone I wanted nothing to do with, and decided I’d had enough: the soggy sliver of wood pressing against my lip and pretending that was normal, dragging cork grease across the mouthpiece like some kind of woodwind lipstick, and what do you mean mold can grow in there?! I told my teacher that if I couldn’t switch to percussion, I was done. No negotiation. No compromise. A full fifth-grade ultimatum from a kid who had apparently decided, at ten years old, that this was the hill he was prepared to die on.
My teacher let me switch.
Twenty-five years later I’m still back there: not the best drummer, never claimed to be, but it’s fun to still pick up the sticks and give it a fill. And after twenty-five years, you learn a few things. Every drummer eventually learns the same lesson, usually the hard way: the click track doesn’t lie, but your body absolutely will.
Here’s why. Adrenaline does something specific to a sense of time. Get a room of people clapping along, get the energy of a show building, get locked into a groove that feels incredible, and your internal clock starts speeding up without telling you. You don’t feel like you’re rushing. You feel like you’re finally in the pocket. Tighter, more dialed in, playing the best you’ve played all night. Meanwhile the tempo has crept from 120 to 128, the bass player is chasing you, the vocalist is running out of breath trying to keep up with phrasing that was written for a slower song, and nobody on stage consciously decided any of this happened.
It’s such a well-known phenomenon that it doesn’t even need a clever name. Every drummer just calls it “rushing.” And the brutal part is that rushing feels like the opposite of a mistake while it’s happening. It feels like momentum. It feels like the band’s finally clicking. You only find out the truth when you play back the recording against a click track and watch the tempo line creep upward across four minutes, completely independent of how good the take felt in the room.
I’ve been thinking about that a lot lately, watching the Customer Success industry talk about AI.
TL;DR: CS teams are reporting that AI makes them feel faster and more strategic, but independent research suggests the perception gap between how AI assistance feels and what it actually produces can be significant. This piece argues that without a defined metric, a baseline, and someone accountable for reading the data, CS orgs are rushing the tempo without knowing it.
Are CS Teams Actually Getting Faster With AI Or Does It Just Feel That Way?
Let’s be honest about where we are. Most CS orgs adopted some flavor of AI in the last eighteen months: a copilot here, a sentiment scoring tool there, an agent that drafts your QBR deck while you’re still finding your coffee. And almost universally, leadership is reporting the same thing back up the chain: we feel faster. We feel more strategic. Our CSMs feel like they have more bandwidth.
Feel. Feel. Feel.
I want to introduce you to a study that has nothing to do with Customer Success on the surface, and everything to do with it underneath. Researchers at METR (an independent group that evaluates AI model capabilities, with no CS platform to sell you and no AI vendor logo to defend) ran a study in early 2025 on experienced software developers using AI coding assistants on real work in their own codebases. They measured actual completion time. They also asked the developers, before and after, how much faster they thought the AI was making them.
The AI-assisted work took about 19% longer. The developers walked away convinced they’d been roughly 20% faster.
Read that again, because it’s not a rounding error and it’s not a fluke. It’s a flat-out perception gap between what happened and what people believed happened, in a population of smart, technical professionals doing work they know well. They weren’t lying. They weren’t dumb. They were playing without a click in their ears, feeling locked in while quietly rushing the tempo the whole time.
METR tried to run a follow-up study in late 2025 to see if things had changed. They ran into a problem: so many developers had become convinced that AI makes them faster that a significant share simply refused to participate in any study that required them to work without it, even in exchange for $50 an hour to work on projects of their own choosing. The ones who did participate were selective about which tasks they submitted, specifically avoiding tasks where they knew AI would give them a big edge. METR concluded their newer data was too contaminated by this selection effect to be reliable, and they’re redesigning the study. Make of that what you will.
But tell me, with a straight face, that Customer Success, despite spending a decade proudly admitting it still struggles to calculate Time to Value correctly, is somehow immune to the same illusion.
What Data Should CS Teams Actually Be Measuring?
I keep seeing the same prediction dressed up in different vendor decks this year: the average CSM will have X% more bandwidth thanks to AI handling the repeatable work. I don’t doubt that AI is handling repeatable work. What I doubt, what I want someone, anyone, to show their math on, is the leap from “AI is doing tasks” to “humans are now better.”
Where’s the recording we can play back against the click? Show me the before-and-after on actual outcomes: time-to-resolution on escalations, accuracy of churn calls, renewal forecast variance, anything measured against a baseline instead of self-reported on a survey. Because right now what I mostly see is the CS equivalent of asking the band how the set felt right after they walked off stage, riding the adrenaline, and writing down “band reports feeling tight.”
This isn’t a new problem for us, by the way, we’ve been here before. We’ve watched teams declare a customer “healthy” because a CSM’s gut said so, with no regression analysis behind it. We’ve watched orgs report Time to Value based on login counts instead of actual customer-defined outcomes, because the proxy was easier to measure than the truth. The instinct to substitute a comfortable feeling for an inconvenient measurement isn’t an AI problem. AI just turned up the energy in the room and took away everyone’s earpiece.
And the research backs up just how widespread the drift actually is. MIT’s NANDA initiative found that the overwhelming majority of enterprise AI deployments, somewhere in the neighborhood of 95%, failed to produce any measurable return. Even Bain, who has seen enough CS transformations to know what works, hedges its own optimism: AI only converts wasted hours into real value if the organization sets a clear, quantifiable target for what it’s actually trying to achieve. Not “be more strategic.” Not “feel more efficient.” A number. A click you can actually count against.
How Should CS Teams Actually Approach AI Productivity?
So here’s the resolution, and it’s not “stop using AI.” I’m not your reactionary uncle. AI is a perfectly good band to be in. The problem isn’t the music, it’s that we’re playing the whole set on adrenaline with no click in our ears and calling that tight.
Set the tempo before you count it in. Before your team adopts any AI tool, decide in writing, ahead of time what the actual measurable outcome is supposed to be. Faster first-response time? Show me the baseline and the after. Higher renewal accuracy? Show me the variance, not the vibe. If you can’t name the metric before you start, you don’t have a productivity initiative, you have a feeling you’re hoping gets validated later.
Listen back constantly, not annually. Tempo drifts. So does perception. The teams getting real value out of AI right now, meaning the successful cases researchers have actually been able to document rather than project, are the ones running tight, disciplined pilots on two or three high-impact processes instead of bolting AI onto everything at once and hoping the averages work out in their favor.
Put someone on the headphones whose entire job is listening to the click, not playing the set. This is where I think a CS Analyst earns their keep more than ever. Not by advocating for AI adoption, but by objectively measuring what changed and what didn’t. Their job is to care less about how the performance feels and more about whether the performance improved. Somebody has to be willing to say, “We’re rushing,” when the rest of the band thinks they’re playing the best set of the night.
And when the data and the feeling disagree, believe the data. Every single time. That’s the whole lesson of playing to a click, and it’s the whole lesson of every churn surprise that ever blindsided a “healthy” account. Your gut is not malicious. It’s just not built to track tempo under adrenaline, and right now, with AI moving this fast, the whole industry is playing a set that keeps speeding up without anyone agreeing to it.
We don’t need to break up the band. We just need to stop confusing how the set feels with what the recording actually shows.
One More Thing
I’ll admit I didn’t always feel this way about click tracks (and I mean that literally).
One of the last shows my band ever played, my lead singer pulled me aside afterward and told me I needed to start playing to a click because I’d been rushing, and the rest of the band had spent the whole set chasing me. My initial reaction was exactly what you’d expect: pure, unfiltered offense. I was the drummer. Keeping time was my entire job. The implication that I couldn’t do it stung in a way I wasn’t prepared for.
Then he asked me who my favorite drummer was.
Neil Peart. Without hesitation.
“You think he doesn’t play to a click track?”
I went silent. And then I agreed.
That’s the thing about rushing. It doesn’t feel like a flaw while it’s happening. It feels like confidence. It feels like command. It took someone I respected, asking me one question about someone I admired, to make me see that accepting the click wasn’t an admission that I couldn’t keep time. It was an acknowledgment that even the best need an objective reference point: something outside their own perception to verify what they already believe to be true.
I think CS teams are in that same moment right now. Nobody’s saying you can’t keep time. We’re just asking you to prove it.
Here’s a little treat of a long haired, rock and roll Justin…no earpiece and likely rushing.
Until next time.
Before You Ask (FAQs)
Is AI actually making CS teams more productive?
The honest answer is: we don’t know yet, because most teams aren’t measuring it. Independent research on developers using AI tools found a significant gap between perceived and actual productivity: they felt roughly 20% faster while being measurably 19% slower. CS teams face the same risk without baseline measurement in place before adoption.
What data should CS teams measure to prove AI productivity gains?
Define the metric before adopting the tool. Time-to-resolution on escalations, churn call accuracy, and renewal forecast variance are all solid starting points (anything with a pre-AI baseline you can compare against). Self-reported bandwidth gains don’t count.
Why are most enterprise AI deployments failing to show ROI?
MIT’s NANDA initiative found that roughly 95% of enterprise AI deployments failed to produce measurable return. The consistent culprit: organizations didn’t set a clear, quantifiable target before deployment. AI only converts time savings into real value when you know exactly what outcome you’re optimizing for.
What is the CS Analyst’s role in AI adoption?
Not to drive adoption but to measure it. The CS Analyst’s value in an AI-heavy environment is being the person whose only loyalty is to whether the numbers actually moved, independent of how good everyone feels about the tool. Somebody has to be willing to say “we’re rushing” while the rest of the band insists it’s never felt better.
What is “rushing” and why does it apply to AI in Customer Success?
Rushing is a drumming phenomenon where a player speeds up the tempo without realizing it. It feels like momentum, but the recording proves otherwise. The METR study on AI-assisted developers showed the same effect in knowledge work: people felt faster while being measurably slower. CS teams adopting AI without baseline metrics risk the same illusion at scale.