2025-llm-agents-grounded-in-self-reports-arxiv2411.10109-latest
The study explores whether large language models (LLMs) can serve as general‑purpose simulators of individual humans by grounding generative agents in participants’ self‑reports rather than outcome‑specific training data. A stratified U.S. sample of 1,052 adults completed (i) a two‑hour AI‑driven, semi‑structured voice interview (American Voices Project protocol) and (ii) structured surveys (General Social Survey core items and the 44‑item Big Five inventory), plus five incentivized economic games and five replicated experiments. Three agent families were built: interview‑only, survey‑only, and combined interview‑plus‑survey agents, each augmented with four sets of domain‑expert reflections (psychologist, behavioral economist, political scientist, demographer). Agents predict held‑out responses on a second wave (two‑week test‑retest) using a chain‑of‑thought prompting pipeline (GPT‑4o). Evaluation metrics include categorical accuracy, mean absolute error (MAE) for numerical items (normalized to 0‑1 based on historical GSS ranges), Pearson correlation, and a normalized‑accuracy score that scales predictions against participants’ internal consistency. Results show interview‑only, survey‑only, and combined agents recover 82‑86 % of participants’ test‑retest consistency on the GSS, outperform demographic‑only (≈74 %) and persona‑based baselines, and achieve comparable performance on Big‑Five traits. Economic‑game predictions show no significant differences across agent types. In five replicated experiments, agents reproduce effect‑size directions and magnitudes with high correlation (r≈0.9) to human replications. Bias analysis using Demographic Parity Difference (DPD) demonstrates that self‑report‑grounded agents reduce prediction gaps across race, ideology, and gender relative to demographic baselines. Mechanistic ablations reveal two key drivers: (1) direct retrieval of verbatim answers from interview transcripts and (2) inferential reasoning from surrounding narrative content; performance degrades modestly when outcome‑aligned questions are removed, but interview agents remain superior even after extensive removal. Additional robustness checks include survey‑only agents, summary agents (bullet‑pointed transcript dictionaries), random‑lesion interview agents (progressive removal of 0‑80 % of utterances), and maximal agents (all data sources combined), showing diminishing returns beyond interview data. Model‑tier experiments with 50 agents across GPT‑4o, GPT‑4o‑mini, GPT‑5, GPT‑4.1, and reasoning‑focused variants show modest accuracy gains for higher‑tier models and a trade‑off between cost and performance for mini models. Fine‑tuning GPT‑4o on 500 agents does not improve performance over the standard prompting pipeline. Extensive regression tables document how demographic variables (age, census division, political ideology, party, education, race, ethnicity, gender, income, neighborhood, sexual orientation) predict GSS, Big‑Five, and economic‑game outcomes, confirming expected patterns (e.g., liberal ideology correlates with higher GSS liberal items). Supplementary construct‑level analyses provide MAE, correlation, and replication‑ratio metrics for individual GSS items, personality dimensions, and game outcomes. The paper concludes that rich self‑reports provide a practical, scalable foundation for person‑level simulation with LLMs, while noting limitations (English‑only U.S. sample, single model family, unknown population‑level correlational fidelity) and proposing extensions to other languages, models, and larger‑scale simulations.
Topics
Motivation and Research Gap
The paper argues that traditional machine‑learning models for human behavior rely on outcome‑specific labeled data, limiting scalability and generality. It proposes grounding large‑language‑model agents in rich self‑reports (interviews and surveys) as a universal, person‑level simulation framework.
Data Collection Infrastructure
Describes the AI‑driven, semi‑structured two‑hour voice interview protocol (American Voices Project) and the administration of structured surveys (GSS core items, BFI‑44), five incentivized economic games, and five replicated experiments on a stratified U.S. sample of 1,052 adults.
Predictive Performance, Bias Mitigation, and Construct‑Level Results
Reports that interview‑only, survey‑only, and combined agents recover 82‑86 % of participants’ test‑retest consistency, outperform demographic and persona baselines, and achieve comparable Big‑Five predictions. Bias analysis using Demographic Parity Difference shows reduced gaps across race, ideology, and gender. Supplementary tables provide MAE, correlation, and replication‑ratio for individual GSS items, personality dimensions, and game outcomes.
Ablation and Robustness Analyses
Mechanistic ablations isolate two drivers: verbatim answer retrieval from interview transcripts and inferential reasoning from surrounding narrative. Additional robustness checks compare survey‑only, summary (bullet‑point) agents, random‑lesion interview agents (0‑80 % utterance removal), and maximal agents (all data sources), revealing diminishing returns beyond interview data.
Model Tier Comparisons and Fine‑Tuning Effects
Evaluates 50 agents across GPT‑4o, GPT‑4o‑mini, GPT‑5, GPT‑4.1, and reasoning‑focused variants, showing modest accuracy gains for higher‑tier models and a cost‑performance trade‑off for mini models. Fine‑tuning GPT‑4o on 500 agents does not improve performance over the standard prompting pipeline.