The Holdout Column

Call Your Customers

The personal stories of a dozen real users will often beat the aggregated data of thousands — because they tell you what to even measure.

Jun 14, 2026 · 3 min read

The dashboard that knew everything except why

I have a reputation as a data guy, so this will sound strange coming from me: most teams are drowning in data and starving for understanding. The dashboard tells you that 38% of new users never come back after week one. It will not tell you why. You can slice that number twelve ways and still be guessing.

So before I model anything, I call people. Ten, maybe fifteen real customers. On the phone, not a survey. It is the least scalable thing I do all quarter, and usually the highest-leverage. The data tells you what is happening. Humans tell you why. Confuse the two and you will optimise the wrong number with great precision.

Start with CLV, then go pull the right people

Who you call matters as much as the calling, so I don’t dial at random. I start from customer lifetime value and pull the top income deciles — the people who already vote with their wallet. Then I build data-driven portraits of them: what segment, what device, what cadence, what they bought first. I look hard at the main entry point and the very first action they took, because the first session quietly decides most of the relationship.

Then I run a look-alike study — who in the wider base resembles these best customers but hasn’t converted into them yet. That gap is the whole game. Now my call list isn’t «some users». It’s «the high-value pattern, and the people who almost match it». Same recipe, two ends.

Data tells you what is happening. Humans tell you why. Confuse the two and you’ll optimise the wrong number with great precision.

NPS, one open question, and a small promo code

The calls don’t scale, so I pair them with something that does. I send an NPS survey, but the score is the boring part — the useful part is one open question: what nearly stopped you, and what made you stay? Open text, their words, not my checkboxes. To get a real response rate, I incentivise it: a small promo code costs almost nothing and roughly triples replies. People answer honestly when you’ve already given them something.

Now two streams meet in the middle. A dozen deep conversations that generate sharp, specific hypotheses. And a few hundred open answers that tell me whether each hypothesis is one person’s quirk or a pattern worth betting on.

They aren’t rivals. They take turns.

Here is the part people get backwards. They treat qualitative and quantitative as opponents — soft stories versus hard numbers, pick a team. They’re not opponents. They take turns. The calls generate the hypotheses. The data tests them. You cannot A/B test an idea you never had, and the best ideas almost never fall out of a spreadsheet. They fall out of someone on the phone saying the one sentence you didn’t expect.

So yes — build the funnel, watch the cohorts, respect the numbers. But once a quarter, put fifteen names on a list and call them. The aggregate will tell you the market is moving. A single honest user will tell you which way — and that’s the part you can act on Monday.

This is the thinking. The Diagnostic is where I point it at your stack.