Kurate.org
Rankings
Back to Rankings

Designing Behavioral Profiles for Language Model Simulation of Out-of-Distribution Human Behavior

Robin Na, Kehang Zhu, Logan Cross, Michiel A. Bakker

Oct 31, 2026arXiv:2611.00661v1
cs.AIecon.GN
Share
Scorecard· 16/16
6.5/10 impact

Timely, carefully designed OOD evaluation of persona/profile design for LLM simulation with useful cautionary findings, but modest sample sizes, model-specific feature selection and a confounded persona comparison limit its reach.

Abstract

Large language models (LLMs) are increasingly used to simulate human behavior, but which descriptions of people improve prediction when experimental designs change remains unclear. We develop a framework that draws candidate behavioral features from the literature and constructs profiles from earlier human experiments. A profile-generation LLM assigns feature values from observed actions, messages, and surveys; a separate LLM simulates behavior using these profiles. By comparing simulated and observed outcomes in the learning data, we select profile contents through forward feature selection, adding features while they improve prediction. We then evaluate the selected contents without modification against outcomes from new participants under new designs. Across 616 public-goods games, 144 multi-party bargaining games, and 1,480 generalized 11-20 money-request designs, leading one-feature profiles predict human outcome distributions better than no profile or general-purpose personas. Profiles with more features provide no clear additional benefit, and features' relative predictive value largely persists across learning and validation. Using profiles from one game family to simulate another produces uneven gains and can make predictions less accurate than simulations without profiles. The framework makes the choice of behavioral information a testable part of simulation design, helping researchers assess which descriptions remain useful as experimental conditions change.

AI Impact Assessments

(1 model)

Core Contribution

The paper treats the content of an LLM simulator's behavioral profile as a design variable that can be tested. Candidate features come from the behavioral-economics literature (57 in total across three game families). A profile-generation LLM scores each participant's behavioral record on these features, citing evidence and a certainty rating, and leaves a feature out when the record gives no qualifying evidence. Forward feature selection then picks profile contents using learning data. The selected profiles are evaluated unchanged on new participants under new designs: 250 public-goods games, 48 bargaining games and 884 generalized 11–20 designs.

The headline results:

  • A leading one-feature profile beats no profile and three general-purpose persona libraries (Nemotron, Twin-2K-500, ANES) in all three families. Normalized alignment is 65.5 vs −1.0 in public goods, 77.5 vs 40.5 in bargaining, and 73.5 vs 12.2 in 11–20.
  • Larger profiles give no clear additional benefit.
  • The ranking of individual features is stable from learning to validation (Spearman ρ of 0.77–0.89).
  • Cross-family transfer is uneven and sometimes harmful. A 11–20 profile applied to public goods scores −68.

    • The "less is more" finding and the explicit out-of-distribution framing are the main contributions.

    Methodological Rigor

    Strengths:

  • Learning and validation use different participants and non-overlapping designs.
  • Candidate features were fixed before simulation outcomes were examined.
  • The study used model snapshots whose knowledge cutoffs predate the datasets' public release.
  • Evaluation compares full distributions via Wasserstein distance, anchored between an uninformed reference and a human-resampling benchmark.
  • It uses paired bootstrap comparisons at the level of the independent unit (games, cohorts, designs).
  • It includes generic and length-matched controls, calibration of personas, model-swap robustness checks, and candid reporting of negative results.

    • Weaknesses:

  • The learning comparisons are small. The public-goods panel has 48 games, and the bargaining third-feature tests use 2–7 cohorts. Selection is therefore noisy.
  • Bargaining validation has only 24 cohorts.
  • Each simulation is a single draw with no repeated sampling.
  • The persona comparison is confounded. The behavioral profiles come from the same experimental population and task, whereas the personas describe different people and have nothing to do with the task. The authors acknowledge this. The comparison shows that task-specific behavioral information beats generic demographics, which is expected, rather than that the selection procedure is better.
  • The key gain comes from the single leading feature, and the real selection effect is thin. In public goods, the leading feature ranks 1st in learning but 5th in validation. Its gain over the complete profile is not significant in two of the three families.
  • Different feature sets are selected under other generator/simulator pairs, so the specific "leading features" are not robust. The paper admits this.
  • The 11–20 benefit appears only with pooled distributions. At the within-design level, no-profile simulation does better (Table A13), which weakens the claim of improved prediction.
  • In the alternative bargaining split (G.2), the complete profile beat the compact one.
  • Construct validity of the LLM-assigned feature values is not established.
  • There is no comparison against full raw behavioral records, which is an obvious competitor.

    • Potential Impact

    This is useful methodological guidance for researchers building LLM-based social-science simulations, in computational social science, experimental economics and mechanism design. Its practical messages are to validate profile contents instead of assuming that richer personas are better, to expect compact profiles to suffice, and to expect cross-domain transfer to be fragile. The warning that profiles can make predictions worse than no profile is valuable. The interpretable, evidence-grounded features also offer a route to hypothesis generation for later human experiments. The reach is limited by dependence on a specific simulator and generator (GPT-5-mini/5.1), three game families, population-level targets only, and no demographic analysis. The specific features found are unlikely to become standard. The framework and the cautionary findings are the lasting parts.

    Timeliness

    Timeliness is high. LLM-based simulation of human subjects is growing quickly and drawing methodological criticism over validation, out-of-distribution generalization, and persona design. The paper addresses these directly and uses a held-out-design protocol that is stricter than most prior work.

    Strengths and Limitations

    Strengths:

  • It is honest about null and negative results.
  • It has a clear OOD evaluation design with a human-benchmark-normalized metric.
  • It uses recent, contamination-aware datasets.
  • Its appendices are extensive.
  • It reports sensitivity analyses.

    • Limitations:

  • Evidence is modest in sample size and effect robustness.
  • The framework is somewhat ad hoc (a 0.70 correlation redundancy rule, strict "improve in every comparison" stopping that mechanically yields two features).
  • The "no clear benefit from more features" conclusion rests on non-significant intervals, so it is absence of evidence rather than evidence of equivalence.
  • The generalization claim is tested in only three narrow strategic-game settings.
  • Reproducibility depends on proprietary APIs, and the codebooks are in supplementary files.

    • Overall this is a solid, timely and careful applied-methods paper with useful cautionary findings. It is not a paradigm shift, and its specific conclusions are tied to particular models and tasks.

    Rating:6.5/ 10
    Significance 6.5Rigor 6.5Novelty 6Clarity 7

    Generated Nov 3, 2026

    Comparison History (0)

    No comparisons yet.