From punch cards to AI: Structure first, then speed
Based on my talk “Fast, Confident and Wrong” at Data & Analytics Congres 2026, Utrecht.
In short
- AI in consumer research goes wrong when there’s no structure underneath it: the answers are fast, confident and wrong.
- Keep anything exact, like significance testing, as rules. Let specialist AI interpret. Keep people on the final call.
- It’s a marathon, not a sprint: every step of structure pays off on its own.
What the 1890 census teaches us about AI
Our first AI outputs on Product Hub were fast, confident and wrong. The fix wasn’t a better model. It was structure.
That lesson is 130 years old.
A counting problem that sounds familiar.
In 1880, the US ran its census, as it does every ten years. Counting it by hand took seven to eight years. The population was growing fast, and experts estimated the 1890 count would take 13.
The next count would start before the last one was finished.
Washington, 1880: seven years to count by hand. From my DAC ’26 talk.
Sound familiar? We have more data than we can read. AI promised to fix that. Most teams got faster output and less certainty: confident answers nobody can trace back to a definition, a rule or a source.
The census was saved by a machine. But the machine wasn’t the clever part.
The card was the schema.
Herman Hollerith had worked on the 1880 census and wanted to automate it. What he built first wasn’t a machine. It was a card.
One card per person. One hole position per fact, decided before a single card was punched: sex, age band, place of birth. That’s where messy handwriting became structured data. Every answer had to fit a hole that already existed. No free text. No ambiguity.
Then the machine could run. Pins dropped through the holes into little cups of mercury, closing a circuit and moving a dial one by one. No hole, no contact, no count.
Same card, same counts, every time. That’s deterministic: same input, same output. Keep that word in mind. It’s the whole story.
The result: 63 million people counted, with results out 18 months sooner than in 1880.
The machine got the credit. The structure did the work.
The sorting was the algorithm.
The same circuits also popped open one of 26 lids on a sorting box. The machine decided where each card went; a person carried it there. That unlocked something hand-counting never could: asking new questions of the same data. In 1880, a new question meant going back to the paper forms. With cards, it meant running a deck you already had.
Collect once, ask many questions. That’s still the whole point of a database. Hollerith’s company went on to become IBM, and the principle stayed the same: one hole, one meaning.
Three details matter more than the speed:
- The machine never guessed. A card that didn’t fit went to a reject pocket, for a person.
- Checks came before counting. Within a decade, the census was throwing out incomplete or inconsistent cards before anything got counted.
- The users made it better. The smart way to sort numbers, last digit first, most likely came from the operators, not from Hollerith’s patent.
From machines to software: the rules moved into code
For the next 70 years, the rules didn’t change much. Where they lived did. Computers moved them into code: the card layout became a database schema, the sorting box became a query.
Every job automation done followed the same pattern. Switchboard operators, until an automatic switch turned a call into a rule. Lift operators, once a stop button and an alarm made automatic lifts trustworthy. Even “computer” was a job title before it was a machine.
Define the task. Automate the rule. Move people up.
All of it deterministic. A program does exactly what it’s told, the same way every time. You can audit it, test it and trust it.
But it can only do what someone already wrote down.
Why AI gives fast, confident and wrong answers
AI is the first automation that isn’t deterministic. A rule gives you the same answer every time. A model doesn’t. That’s what lets it do what rules never could: read a chart in context, write commentary, find the themes in a thousand open answers. It’s also why it can be confidently wrong.
So there are now three ways to automate a step, and the skill is picking the right one:
Three ways to automate: deterministic rules, generic AI and specialist AI.
- Deterministic: rules. Same input, same output. Auditable and exact, but it can’t produce anything new. Use it for significance, KPIs, action standards.
- Generic AI: a general model and a prompt. Fast and fluent on almost anything, but it doesn’t know your context, goals or rules.
- Specialist AI: one narrow job, with your data and rules as context, benchmarked against your experts. Use it for interpretation, commentary and synthesis.
*Prompt engineering and skills help. They don’t replace the context.
Most teams only use the middle one. That’s where fast, confident and wrong comes from.
It happened to us. Twice, in my case. Before Product Hub, I led an early attempt at an AI knowledge base at MMR. It failed because there was nothing structured underneath it. Then, when we first tried AI on Product Hub, our Product Intelligence Ecosystem for consumer and sensory testing, we gave a general model our data. The outputs took longer to read than they were worth: fluent, confident, and stuck at 80% done, with us patching holes.
The model didn’t know the product, the objective, which differences were significant, or how our researchers read a result. The problem sat underneath the model, not in it.
Even the model makers say so. Anthropic’s guidance on reducing hallucinations reads like our playbook: restrict the model to the information you give it, ground and verify its claims, refine its outputs step by step, and have people validate anything critical. The techniques reduce hallucinations; they don’t remove them. Accuracy isn’t a model setting. It’s something you build around the model.
One rule has held up since: the numbers never come from the model. Rules calculate. AI explains. And a model never signs off a go/no-go.
AI now works on both sides of the card. It helps punch it: we’re building AI that tags products and preps the product information for each study.* And it helps read it: interpreting results, writing commentary, triangulating across studies. Either way, a person checks, because a wrong hole looks exactly like a right one.
*Still in build. I’ll share how it goes.
What happens to insights people
The role doesn’t shrink. It moves up. Like it did for the human computers: the arithmetic went, the judgement stayed.
AI takes the digging: reading every table, drafting the commentary, spotting every difference across every study. What it can’t do is decide what matters.
Four jobs stay with people:
- Set sharp objectives. The system is only as good as the question it’s given.
- Decide what data to collect. Today’s data choices are tomorrow’s advantage.
- Know the standards. Which data points will actually shape the decision, and what counts as a win.
Make the call. Rules decide what’s real. AI explains what it means. People make the go/no-go.
“A computer can never be held accountable, therefore a computer must never make a management decision.”
— IBM, 1979
The company that grew out of Hollerith’s punch cards wrote that nearly 50 years ago. It still holds. Setting the benchmark, deciding what data gets collected and served to the AI, making the final call: all still human. AI makes those calls faster, better informed and more confident, because every answer traces back to data and rules we chose.
And one more job: making the AI better. Like the sorting trick, the improvements come from the people using it every day: experts who know what good looks like, champions, and client partners willing to build it with you.
How to get accurate AI outputs in consumer research
Good outputs come from three things underneath the model: the right data, mapped-out rules, and outputs benchmarked against your experts.
What sits under every step: data, rules, written judgement and specialist AI.
Start with the intern test. Could a smart new intern do this step with only what you’d give the AI? Is “good” written down anywhere? Would you spot a wrong answer? If not, the AI can’t do it either. AI-ready means intern-ready.
1. Serve the right data, structured the same way every time
It starts before the project. Every study runs on one of our standard solutions, from prototype screening to blind and branded benchmarking, so studies can be compared from day one. Then the project is our punch card. Every project carries the same structure:
- Methodology & design: method, design, sample, markets, questionnaires with clear question types.
- Project information: client, category, objectives, action standards.
- Product information & metadata: what each product is, prototype or in-market, its benchmark, its tags.
It has to be the same every time. If every project looks a little different, the AI has to work out the structure first. It fails, drifts off and answers a different question, or burns a ton of tokens working out what a consistent structure would have told it for free.
Structure is cheaper than tokens.
And the right data isn’t all the data. The platform pushes each AI skill the right slice at the right moment; the model doesn’t go looking. Because consumer and sensory results for the same products sit in one database, that slice can include hundreds of past studies, structured across 20+ parameters where our norms used to run on two or three.
2. Map out the rules, and write down the judgement
Map one piece of work end to end and mark every step: rules, judgement or waste. Most teams find the waste before the AI opportunity.
Anything that has to be exact stays as rules. For us that’s significance testing, only ever within product groups. Even our new charts work this way: AI draws them from tables the rules already calculated.
Then write down what your experts carry in their heads: what counts as a win, how to read a result, which benchmark applies. That’s the semantic layer, the intern’s briefing pack. Build it once, use it on every study.
3. Benchmark the outputs, and own the benchmarks
A benchmark turns “it looks good” into a number you can check.
- Benchmark on existing work. We compared our Reporting Assistant, which writes deck commentary, pair by pair against decks our researchers had finalized. It’s now used on every project and has cut delivery time significantly.
- Build new benchmarks where none exist, and let the people who’ll rely on the tool decide when it’s good enough.
- Prove before you build. Our AI questionnaire builder started as a proof of concept with one applied engineer. It’s now in beta.
- Version everything and re-run the benchmark on every change. If the scores drop, it doesn’t ship.
Do that, and the model stops being the risky part. You can point the same pipelines at any model and keep whichever wins.*
Anyone can rent the same model. Nobody can rent your data, your rules or your benchmarks.
*Your learning and your data should stay yours.
Structure first, then speed
Hollerith didn’t start with the machine. He started with the card.
It’s a marathon, not a sprint. And every step of structure pays off on its own.
So before you blame the model, fix what sits underneath it:
- Structure your data the same way every time.
- Keep anything exact as rules, and write down the judgement.
- Benchmark the AI against your experts, and own the benchmarks.
- Keep people on the decisions that matter.
Then ask: what data do you have that nobody else can buy, and what do your experts know that’s never been written down? Where those two meet is your moat.
Own the stack for what matters. Build the right things.
Want to see how this works in practice? Get in touch with the Product Hub team.
Alex Dobromir is Lead Product Manager at Product Hub by MMR Research and a Greenbook Future List 2026 honoree. He speaks regularly at IIEX, Quirk’s and Eurosense on making AI-assisted insight trustworthy.
Sources
- The Hollerith Machine, US Census Bureau
- Herman Hollerith’s Tabulating Machine, Smithsonian Magazine
- Algorithms to Live By, Brian Christian and Tom Griffiths (2016)
- Reduce hallucinations, Anthropic
- From projects to programs: fixing the hidden flaws in modern product testing, Product Hub by MMR at Quirk’s Chicago 2026
- Automated product testing solutions, Product Hub by MMR