Key Takeaways
- AI research agents are strong at synthesis and structurally locked out of acquisition. The wall is access, not intelligence
- I ran the test. An AI agent answering "why do people quit melatonin" cited two hospital systems and two supplement brands, quoted zero customers, and missed the single biggest reason, which accounts for about 24% of quit mentions in a 9,900-comment corpus
- Deep research agents keep link validity above 94% and source relevance above 80%, but land at only 39% to 77% factual accuracy, and accuracy falls roughly 42% as they do more digging (Onweller and colleagues, 2026)
- Reddit's robots.txt disallows every crawler, and bulk access costs $0.24 per 1,000 API calls. Cloudflare made AI-crawler blocking the default for new domains in July 2025, on a network carrying 20% of the web's traffic
- Asking a model to imagine your customers instead is worse. Synthetic answers sat 14.5 percentage points off real ones in January 2026, and the bias runs toward well-known brands
Short version: AI does one half of research very well and cannot do the other half at all. It can read, sort, compare and summarise faster than any human. It cannot go and get the thing that needs reading, at least not the kind of thing voice of customer work runs on.
That distinction gets lost because both halves get called "research." When someone says an AI agent did their research, they usually mean it read what's already been published and wrote it up. Which is genuinely useful. It's also not what voice of customer is.
AI research agents perform two separable tasks: acquiring source material and synthesizing it. Current agents are strong at synthesis and constrained at acquisition, because the raw customer conversations that voice of customer research depends on sit behind authenticated APIs, rate limits and crawler blocks rather than on the indexable open web.
What deep research is genuinely good at
Let me be fair to the thing before I take it apart. If your question can be answered from published material, deep research is remarkable, and it does in fifteen minutes what used to take a competent analyst a week.
Ask it what the compression boot market looks like, or which claims regulators have gone after in the supplement category, or how three competitors describe their return policy. It will read forty sources and hand you something structured and mostly right. I use it constantly for exactly that.
Even there, the accuracy isn't what the confident formatting suggests. Onweller and colleagues tested source attribution across frontier deep research agents in May 2026 and found the citations look far better than they are. Link validity stayed above 94% and relevance above 80%, while actual factual accuracy landed between 39% and 77% (Cited but Not Verified, arXiv, May 2026).
The uncomfortable part of that study is the trend. Fact-checking performance fell by roughly 42% as the agents scaled from 2 tool calls to 150. The harder the agent works, the less its citations hold up, which is the opposite of how research is supposed to behave.
So even on its home turf, the output needs checking. But that's a quality problem, and quality problems get better. What follows is a different kind of problem.
I ran the test on a question I already had the answer to
Here's the useful thing about publishing studies. I have real answers sitting around, so I can check an AI answer against one instead of arguing about it in the abstract.
I asked an AI agent with full web access a question I'd already answered from 9,900 comments: why do people stop taking melatonin. My version is public, in the melatonin study. The agent came back in about a minute with five reasons, ordered, written cleanly, with six sources attached.
Two of those sources were hospital systems, one was a trade magazine, two were supplement brands writing about their own category, and one was a personal blog. Not one customer was quoted. Not one proportion was given. Here's how its answer lines up against the actual corpus.
| What the AI answer said | What 9,900 comments say |
|---|---|
| Ineffectiveness (listed first) | "It never did anything" ranks 4th, about 10% |
| Tolerance and receptor desensitization | "Stopped working" ranks 6th, about 7% |
| Side effects: vivid dreams, next-day drowsiness | Grogginess about 22% and dreams about 20%, the two largest side-effect reasons, neither separated nor ranked |
| Cost of ongoing supplementation | Doesn't appear in the top seven |
| Wanting to test natural sleep | Doesn't appear in the top seven |
| Not mentioned | "It's a hormone, I'll get dependent" is the single biggest reason, about 24% |
Look at what happened rather than at whether the answer was "wrong." Nothing the agent said was false. Every item on its list is a real thing that real people do. It just had no way to know how often, in what order, or in whose words.
And the top reason was invisible to it. People quit melatonin mostly because they've decided it's a hormone and they'll stop producing their own. That belief lives in comment sections. Clinics publish articles titled "is melatonin addictive," which answers the question rather than counting the people asking it.
In a controlled comparison, an AI research agent answering "why do people stop taking melatonin" produced five plausible reasons from six published sources, none of which quoted a customer or reported a proportion. The largest real reason, a belief that supplemental melatonin suppresses natural production, accounted for roughly 24% of coded quit mentions across 9,900 comments and did not appear in the AI answer at all.
Why it happened, and why it isn't about intelligence
The agent didn't fail at reasoning. It reasoned fine over the material it could reach. The material it could reach was articles, because articles are what the open web will serve to a bot in 2026.
Reddit's robots.txt is User-agent: * / Disallow: /, a blanket refusal that's been in place since the June 2024 update (MERJ, July 2024). Getting comments in bulk means the Data API, which is free only under 100 queries per minute and only for non-commercial use, and otherwise costs 24 cents per 1,000 requests (TechTarget, July 2023).
None of that is a prompt. It's an OAuth client, a pagination loop, a rate limiter, a retry policy for the 429s, a dedupe pass, and a budget. I know because I maintain one, and the boring parts of it break more often than the interesting parts.
Here's the part that made me laugh while writing this. Researching this post meant fetching about a dozen pages. Five came back refused. OpenAI's own page describing deep research returned a 403. So did Reddit's help centre, Reddit's corporate blog, and TechCrunch.
An AI agent, blocked from reading the public web, while researching a post about AI being blocked from reading the public web. Of course the numbers I needed were all in the pages that said no.
Bigger models won't close this gap
This is the bit I'd push back on if someone said the post ages badly. The wall isn't cognitive, so raising the ceiling on reasoning doesn't move it.
On 1 July 2025, Cloudflare flipped the default. New domains now block AI crawlers unless the owner opts in, rather than the other way round, and publishers who'd rather charge than block get a pay-per-crawl option (MIT Technology Review, July 2025). Cloudflare manages traffic for 20% of the web, so that default carries some weight (Cloudflare, July 2025).
The direction of travel is one way. Comment data went from free to licensed, and licensed means a contract, not a clever prompt.
There's an arithmetic problem underneath it too. A 9,900-comment corpus is somewhere around 600,000 tokens before you've done a single thing with it. That's not impossible to process, but it isn't something a chat window does while you wait, and it certainly isn't something an agent quietly gathers mid-answer.
So the honest forecast is that models keep getting better at the reading and stay locked out of the gathering, because the people who own the gathering now have a business reason to keep them out. If you want the wider argument about why the source of your data matters more than the method, I made it in why surveys fail DTC brands.
The shortcut everyone tries, and why it's worse
Once people hit the wall, the next idea is always the same. Skip the collection. Ask the model to be the customer. Describe your buyer, ask what she thinks about your price, and get an answer instantly.
This has been tested properly, and the results are not encouraging. Verasight ran 52 questions past 2,000 real US adults and against LLM-generated synthetic respondents in January 2026. The mean absolute error across 305 response options was 14.5 percentage points, and their conclusion was blunt: synthetic samples fail to replicate human response patterns (Verasight, Synthetic Sampling Report IV, January 2026).
It got worse where it matters most to a DTC brand. On personal behaviours and lifestyle preferences, the categories without a tidy demographic predictor, performance dropped hardest. Health care was the worst category tested at 23.4 points of error. Politics was the best, which tells you what these models actually learned.
The brand-level finding is the one I'd tape to the wall. Kaiser and colleagues compared synthetic and real respondents on brand questions and found synthetic answers systematically more positive, off by an average of 1.2 points on a 7-point scale, with the largest bias toward well-known brands (NIM Marketing Intelligence Review, 2026).
Read that in DTC terms. A simulated customer will flatter the incumbent you're trying to take share from and undersell the challenger you're trying to build. It's biased in precisely the direction that makes a small brand give up on a position worth fighting for. And the responses cluster tighter than real ones, so the disagreement you most need to find, the thing your best customers say that nobody else does, gets averaged away.
Synthetic respondents are not a substitute for collected customer data. Verasight measured a 14.5 percentage point mean absolute error between LLM-generated and human responses across 52 questions in January 2026. Kaiser and colleagues found synthetic brand ratings ran 1.2 points high on a 7-point scale, with the strongest positive bias toward well-known brands and materially less response variation than real participants.
Where AI does belong in this work
None of this makes me an AI sceptic. It makes me specific about which step it's for, and the step is after the corpus exists.
Once ten thousand real comments are sitting in a file, the job changes from getting to reading, and reading at that volume is miserable by hand. That's where a model earns its keep: first-pass sorting, grouping things that say the same thing differently, surfacing the comment you'd have missed at hour six. I still decide what the categories are, I still count, and I still open every quote before it goes in a report, because a quote that turns out not to exist is worse than no quote at all.
Think of the pipeline in three parts. Gathering is engineering, and it's the part that's closed. Reading is judgment, and it's the part that's yours. In between sits a large, dull, mechanical sorting job, and that's the part AI genuinely transformed. The mistake is assuming that because the middle got solved, the ends did too. If you want the full method end to end, the complete guide to VOC research for DTC brands walks it, and how to use Reddit for DTC research covers the platform this argument is mostly about.
Two questions that tell you which kind you have
Before you send a question to an AI agent, ask whether anyone has already published the answer. If the thing you want to know appears in articles, reports, filings or product pages, deep research will do a good job and you should use it. That's most competitive and category work.
Then ask whether you need to know how many. If your next decision depends on proportion, on which complaint is the common one rather than merely a real one, no amount of synthesis gets you there. Articles don't carry counts of what customers say, because their authors weren't counting either.
When both answers point the wrong way, when the thing you need is unpublished and needs weighing, that's when somebody has to go and collect it. That's the whole job, and it's less glamorous than it sounds.
Got a question no article has answered?
Insightios collects the actual conversations from Reddit, YouTube, forums and reviews where your market talks, counts what repeats, and hands you a report with proportions and real quotes attached. Flat fee, fixed turnaround.
Frequently asked questions
Can ChatGPT or deep research do voice of customer research?
It can summarise what has been written about your category, which is not the same thing. Voice of customer needs the raw comments, and those sit behind APIs and robots.txt rules an AI agent cannot pass. Deep research reads articles about your customers. It does not read your customers.
Why can't an AI agent just scrape 10,000 Reddit comments?
Reddit's robots.txt disallows every crawler, and bulk access runs through the Data API, which is free only under 100 queries per minute for non-commercial use and costs $0.24 per 1,000 calls otherwise. Collecting a corpus means authenticated, paginated, rate-limited requests. That is code, not a prompt.
Will better AI models fix this?
Not on its own, because the barrier is contractual rather than cognitive. Cloudflare made AI-crawler blocking the default for new domains in July 2025, on a network carrying 20% of the web's traffic, and platforms now sell data access rather than give it away. A smarter model still gets a 403.
Can AI simulate customers instead of collecting real ones?
It can, and the output is unreliable. Verasight measured a 14.5 percentage point mean absolute error between synthetic and real responses across 52 questions in January 2026. Kaiser and colleagues found synthetic brand ratings ran 1.2 points high on a 7-point scale, with the largest bias toward well-known brands.
Where does AI actually help in voice of customer research?
After the corpus exists. Once you have thousands of real comments in front of you, AI is genuinely useful for first-pass sorting, clustering and drafting, with a human deciding the coding scheme and checking every quote. The split is simple: gathering is engineering, reading is judgment, and the middle is where AI earns its place.
What to do next
Take whatever question you were about to hand to an AI agent and run it through the two tests above. Most questions pass, and you should absolutely use the agent for those, because it's faster than you and cheaper than anyone you'd hire.
For the ones that don't pass, the honest position is that someone still has to go and get the data. Maybe that's you on a Saturday, which I've argued elsewhere is often the right call. Maybe it's me. Either way the answer arrives the same way it always has, which is by reading what people actually wrote, in the places they actually wrote it. Once you have it, turning it into copy is the next problem, and that's using customer language in your DTC marketing.
Sources
- Onweller, H., Lumer, E., Huber, A., Ramchandani, P., Subbiah, V. K., & Feld, C. (2026). Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents. arXiv, submitted May 7, 2026. Link Retrieved August 2026.
- Verasight. (2026). Synthetic Sampling Report IV: Can Large Language Models Replicate Survey Data Across Topics? (2,000 US adults, 52 questions, 305 response options), January 13, 2026. Link Retrieved August 2026.
- Kaiser, C., Kaiser, J., Schallner, R., Manewitsch, V., & Rau, L. (2026). Leaving Insight to Digital Twins? Promise, Progress and Limits of Synthetic Respondents. NIM Marketing Intelligence Review, 18(1), 48-53. Link Retrieved August 2026.
- MERJ. (2024). Investigating Reddit's robots.txt Cloaking Strategy (analysis of the June 25, 2024 update), July 4, 2024. Link Retrieved August 2026.
- TechTarget. (2023). Reddit pricing: API charge explained (24 cents per 1,000 API requests; free under 100 queries per minute via OAuth), July 11, 2023. Link Retrieved August 2026.
- MIT Technology Review. (2025). Cloudflare will now block AI bots from crawling its clients' websites by default, July 1, 2025. Link Retrieved August 2026.
- Cloudflare. (2025). Cloudflare Just Changed How AI Crawlers Scrape the Internet-at-Large (press release; "helping to manage and protect traffic for 20% of the web"), July 1, 2025. Link Retrieved August 2026.
- Insightios. (2026). Why People Quit Melatonin: Dreams, Grogginess and the Hormone Worry (9,900 Reddit comments). Link