Last year, I did a quick, small-scale study (N = 76) to assess demand for health insurance in Nigeria. Fast forward to some months ago, I stumbled upon a paper testing whether synthetic data generated by LLM-generated “participants” could approximate how real people might have responded to a survey questionnaire.
That intrigued me, so I did the same thing with my study.
I fed my health insurance demand survey questionnaire to Claude and asked it to generate responses that 100 Nigerians might have provided. After that, I aggregated the responses from the synthetic “Nigerians” and compared it with the 76 real Nigerians I had surveyed.
Some quick observations:
When health insurance was really cheap (price = ₦0–₦100/month; $0–$0.00067), demand from aggregate synthetic “Nigerians” was higher than that of the aggregated real Nigerians. I had interviewed some of the real Nigerians last year who told me they would be suspicious of the quality of any insurance plan available at such a low price. Claude hadn’t taken that into account when generating its synthetic data.
When health insurance was really expensive (the maximum price in the survey was ₦560,000/month; $373.33), demand from aggregate synthetic “Nigerians” was 0%, while demand from my real participants hovered between 12–16%. The news media is rife with reports on the poverty levels of low-to-middle income countries (LMICs) and that likely informed the patterns of responses in Claude’s output. Yet, based on my survey data, there are people who were either wealthy enough to afford health insurance at that price, or simply valued it so much that they were willing to sacrifice other things for its sake.
Bottom line: I don’t know what things would look like in 50 years, but for now, I think it’s safe to bet that LLMs will continue to struggle to model non-WEIRD behavioral and decision-making patterns — and opportunities abound for those who continue to do research in this space.
Originally posted on Substack Notes, November 2025.