Synthetic surveys: separating AI hypotheses from customer evidence

A product team wants to evaluate an app idea before conducting interviews. It is tempting to ask AI to impersonate hundreds of prospective customers, put the answers into a spreadsheet and choose the most popular feature. The spreadsheet resembles research: it has segments, percentages and quotations. But the origin of its answers determines which decisions it can support.

A synthetic respondent is a model answering as an assigned persona. Generating more answers increases the amount of generated material. It does not, by itself, establish that actual customers think or behave in the same way. Product teams need an explicit boundary between generated hypotheses and observations or answers obtained from people.

What Pew Research Center tested

On September 30, 2026, Pew published a methodological experiment comparing American Trends Panel participants with AI personas based on those participants. The model received characteristics of real panelists and information from an earlier survey, then answered questions from three surveys conducted in the first half of 2026. The principal findings in these chapters concern Claude Opus 4.6 with the experiment's selected settings.

In the chapter on certainty and factual knowledge, Pew reports that human respondents selected an explicit “not sure” option on opinion questions 16% of the time, compared with 4% for synthetic respondents. AI also answered factual-knowledge questions correctly more often. The generated answers reflected information available to the model and its inclination to give a definite answer, rather than accurately reproducing people's knowledge and uncertainty.

This was an experiment about public opinion in the United States. Its percentages cannot be transferred to your customers, another country or every model. It was not a direct study of B2B customer interviews either. Its relevance to product work is a specific caution: a plausible persona can conceal uncertainty that a team needs to investigate with actual users.

An average view can hide important exceptions

In the separate chapter on diversity of public opinion, Pew describes narrower answer distributions. For some questions, the model clustered people around middle categories; for others, it assigned almost everyone the same answer. A result pointing in a similar direction to a human poll did not establish matching diversity.

In an application, exceptions can matter more than an average preference. One person works from a phone, another passes information to a colleague and a third lacks authority to approve a request. If synthetic personas converge on a convenient default journey, the team may miss conditions under which the process fails. That is a design risk arising from the difference between hypotheses and observations, rather than a product effect measured by Pew.

When planning a customer product, examine these boundaries: who starts an action, who finishes it, what information each person has and what happens when a request is rejected. A repeated AI answer is not evidence that exceptions do not exist. It can suggest which exceptions should be investigated.

Where generated answers are useful

Use AI during preparation to propose alternative question wording, identify specialist jargon and suggest competing explanations or scenarios. A model might hypothesize that users do not understand a request status. Turn that into a research task: ask a customer to find the status and explain the next action.

Save the initial hypothesis before an interview. Record the observations that would support it, evidence that would contradict it and the decision depending on the result. If the hypothesis concerns why people abandon a process, asking whether they like a screen is insufficient. Discuss their last actual experience or observe them completing a task.

Keep synthetic text, human answers, product events and team interpretations separate in the working evidence table. Add origin, date and context. For AI material, retain the model and prompt where organizational rules permit. Personal information and confidential customer records need an established handling process before being shared with an external service.

Moving to checks with real people

Imagine a team building a service-request portal. It can start with a complete journey, as described in planning a first portal release. Ask how customers submitted their last request, where they waited for a reply and whom they contacted when the status was unclear. Then introduce a prototype and observe a specific action.

Recruit people facing different working conditions, rather than relying only on the easiest volunteers. Distinguish new from experienced users and request initiators from approvers. A small set of observations can identify journey problems without establishing a precise percentage for the whole customer base. Recruitment and sample size should fit the research question.

For mobile purchasing, a mobile friction audit supplies another kind of evidence: actual devices and actions rather than a fictional persona's explanation. After launch, measuring repeated onboarding value continues the investigation through return visits to a useful task, alongside first-impression feedback.

What the decision record should contain

Before deciding the scope of a custom application, write down what came from people, what appears in observed behaviour, what AI suggested and what remains untested. For a disputed feature, identify the next validation method and the cost of choosing incorrectly. That record is more useful than a large synthetic poll without an explanation of where its data came from.

Generated answers can enrich preparation when the team treats them as material for better questions. Establishing a customer need requires evidence connected to the customer. AI's useful role is to help the team notice what it does not yet know and organize a way to investigate the gap.

Sources

• Pew: How synthetic respondents express certainty and factual knowledge, September 30, 2026.
• Pew: How well synthetic polls capture the diversity of public opinion, September 30, 2026.