Can Large Language Models Revolutionize Survey Research? Experiments with Disaster Preparedness Responses
Yan Wang, Ziyi Guo, Christopher McCarty
Abstract
Survey research faces mounting structural challenges: declining response rates, sample bias, block-wise missingness among at-risk respondents, and AI-assisted fraudulent completions in online panels. Large language models (LLMs) have been proposed as a remedy, yet rigorous evaluations across the full survey workflow remain scarce, particularly in disaster contexts where data quality matters most. We present and evaluate a five-stage framework for LLM integration covering questionnaire design, sample selection, pilot testing, missing-data imputation, and post-collection analysis, using the 2024 Hurricane Milton preparedness survey of Florida residents (n=946) as a shared empirical testbed. We introduce a Protection Motivation Theory (PMT)-constrained co-occurrence knowledge graph and develop seven LLM configurations spanning zero-shot inference, retrieval-augmented baselines, and novel theory-informed variants. Our proposed Anchored Marginal Theory-Informed LLM (A-TLM) outperforms all three classical imputation baselines (IPW/MI, MICE+PMM, missForest) on RMSE under disaster-relevant block-wise MNAR conditions (S4 RMSE 1.439 vs. 1.496 for the next-best), while achieving near-zero signed bias (-0.121) where the random-forest imputer produces the largest absolute bias (-0.631). Organizing retrieval around PMT causal structure and integrating all evidence in a single model call outperforms unstructured retrieval and staged sequential inference (MAE 0.993 vs. 1.097 for standard RAG). We document that near-zero aggregate bias can mask opposing subgroup errors and propose subgroup-stratified bias auditing as a reporting standard. A retrieval-constrained knowledge-graph chatbot demonstrates that hallucination is architecturally manageable through grounded refusal.
AI Impact Assessments
(1 model)Scientific Impact Assessment
1. Core Contribution
This paper proposes a five-stage framework for integrating LLMs into the survey research workflow: questionnaire design, sample selection, pilot testing, missing-data imputation, and post-collection analysis. The central empirical contribution is the Anchored Marginal Theory-Informed LLM (A-TLM), which organizes retrieval-augmented generation around Protection Motivation Theory (PMT) causal structure via a constrained co-occurrence knowledge graph. The paper uses the 2024 Hurricane Milton preparedness survey (n=946) as a unified testbed across all stages.
The key innovation is the theory-constrained retrieval architecture: rather than using flat embedding similarity for RAG, the authors organize evidence according to PMT's causal cascade and integrate all evidence in a single model call. This outperforms both unstructured retrieval and staged sequential inference. The paper also contributes the methodological insight that near-zero aggregate imputation bias can mask opposing subgroup errors, proposing subgroup-stratified bias auditing as a reporting standard.
2. Methodological Rigor
The experimental design is generally sound but has notable limitations that the authors acknowledge. The controlled experiments (Stages 3 and 4) are well-structured with a deterministic train/validation split, four progressively demanding missingness mechanisms (MCAR through block-wise MNAR), and comparison against three established classical baselines (IPW/MI, MICE+PMM, missForest). The component ablation in Table 8 effectively isolates the contribution of peer examples and vulnerability cues.
However, several concerns temper confidence in the results:
3. Potential Impact
The paper addresses a genuine and worsening crisis in survey methodology. The statistics cited—response rates declining 15-30 percentage points, AI agents passing attention checks at 99.8%, usable completion rates falling to 10% in some panels—paint a dire picture. The proposed framework has several practical applications:
The subgroup-stratified bias auditing recommendation is perhaps the most broadly impactful contribution—it's a simple, implementable standard that could improve transparency across any LLM-augmented survey workflow.
4. Timeliness & Relevance
This paper is highly timely. The convergence of declining survey quality, proliferating AI-generated fraudulent responses, and rapid LLM capability improvement creates an urgent need for systematic evaluation of LLM integration in survey science. The disaster context amplifies this urgency given climate-driven increases in extreme weather events. The paper also speaks to the broader debate about "silicon sampling" and AI surrogates in social science, providing a more nuanced position than either uncritical adoption or wholesale rejection.
5. Strengths & Limitations
Key Strengths:
Notable Weaknesses:
Overall Assessment
This is a competent, well-structured paper that makes a meaningful contribution to a timely problem. The five-stage framework provides a useful organizational scaffold, and the theory-constrained retrieval architecture is a genuine methodological contribution. However, the empirical evidence is somewhat thin—small validation samples, single dataset, single LLM, and modest performance margins limit the strength of claims. The paper's greatest value may be in its conceptual framing and its subgroup-bias decomposition insight rather than in the specific numerical improvements reported.
Generated May 20, 2026
Comparison History (24)
Benchmarks evaluating large language models, particularly in complex areas like multi-turn emotional intelligence, typically achieve widespread adoption and high citation counts across the broad AI and NLP communities. While Paper 1 offers rigorous methodological advancements for survey research, its impact is largely concentrated within computational social science. Paper 2 introduces a fundamental evaluation tool with broader relevance to human-computer interaction, model alignment, and conversational AI development.
Paper 1 likely has higher scientific impact due to its broader, timely relevance to LLM agents and web retrieval, a clearly novel problem formulation (state-gated retrieval), and a reusable benchmark with diagnostic error taxonomy that can drive progress across many agent systems and applications. Benchmarks often become community reference points, enabling comparable evaluation and follow-on methods work. Paper 2 is rigorous and practically important for survey methodology, but its empirical scope is narrower (one disaster survey context) and improvements are incremental; impact may remain more contained within computational social science and imputation workflows.
Paper 1 addresses a broader, more fundamental challenge in survey research methodology with a comprehensive five-stage framework applicable across many disciplines. It introduces novel methodological contributions (A-TLM, theory-constrained knowledge graphs, subgroup-stratified bias auditing) with rigorous evaluation against established baselines. Its impact spans social sciences, disaster management, and AI methodology. Paper 2, while technically solid, addresses a narrower problem in collaborative autonomous driving with incremental improvements. Paper 1's methodological contributions and cross-disciplinary relevance give it higher potential impact.
Paper 2 addresses a broader interdisciplinary problem (survey methodology + LLMs + disaster research) with novel theoretical contributions (PMT-constrained knowledge graph, A-TLM method, subgroup-stratified bias auditing). It introduces methodologically rigorous comparisons against established statistical baselines, proposes new reporting standards, and has clear real-world applications in disaster preparedness and survey science—fields with massive user bases. Paper 1, while technically solid, is more narrowly focused on engineering design benchmarking for LLM agents, serving a smaller community. Paper 2's contributions to imputation methodology and bias auditing have broader cross-field applicability.
Paper 1 presents a comprehensive, methodologically rigorous framework addressing fundamental challenges in survey research (declining response rates, missing data, AI fraud) with novel contributions including theory-constrained LLM imputation, subgroup-stratified bias auditing, and hallucination-managed chatbots. It tackles a broadly applicable methodological problem relevant across social sciences, public health, and disaster research. Paper 2, while innovative in applying VLM agents to A/B test simulation, addresses a narrower commercial application with 77% directional alignment—promising but domain-specific. Paper 1's breadth of impact, methodological contributions, and cross-disciplinary relevance give it higher scientific impact potential.
MOCHA addresses a fundamental gap in LLM agent optimization by introducing a principled multi-objective framework (Chebyshev scalarization with annealing) that demonstrably outperforms existing methods across diverse tasks. Its contribution is methodologically novel, broadly applicable to the rapidly growing field of LLM agents and prompt optimization, and solves a well-defined technical problem with strong empirical results. Paper 2, while thorough and practically useful for survey methodology, is more application-specific to disaster preparedness surveys, with incremental improvements over existing imputation methods and narrower cross-field impact potential.
Paper 1 has higher potential impact due to a more novel, generally applicable methodological contribution: a trainable neurosymbolic framework that deterministically maps explicit arguments to ternary verdicts, addressing a central, timely issue (faithful LLM reasoning/explanations) with clear rigor and broad applicability to high-stakes verification beyond a single domain. Paper 2 is strong and practical for survey methodology (notably MNAR imputation and bias auditing), but is more domain-specific and largely an applied evaluation of LLM configurations on one empirical setting, which may limit breadth and novelty relative to Paper 1’s framework-level advance.
Paper 1 addresses a critical and timely crisis in survey research (declining response rates, AI fraud) using highly relevant LLM methodologies. Its application to disaster preparedness offers immediate, high-stakes real-world utility. The introduction of theory-informed LLMs outperforming classical missing-data imputation baselines provides a methodological breakthrough with broad interdisciplinary impact across computational social science, public policy, and NLP, giving it an edge over the more theoretical focus of Paper 2.
Paper 2 presents a rigorous empirical evaluation with quantitative results on a real-world dataset, introduces a novel imputation method (A-TLM) that outperforms established baselines, and addresses concrete methodological challenges in survey research with broad applicability across social sciences. It also contributes actionable findings (subgroup-stratified bias auditing) and demonstrates practical LLM integration. Paper 1, while intellectually interesting in formalizing architectural patterns for LLM agents, is primarily a conceptual/methodological framework without strong empirical validation, limiting its immediate measurable impact.
Paper 1 has higher potential impact due to stronger methodological rigor and clearer real-world relevance: it evaluates LLM integration across the full survey workflow on a real disaster-preparedness survey, benchmarks against established imputation methods under MNAR block-missingness, reports multiple error/bias metrics, and proposes an actionable auditing standard. It also contributes a theory-constrained retrieval/knowledge-graph approach with safety benefits (grounded refusal). Paper 2 is timely and broadly applicable, but is a proof-of-concept with smaller-scale evaluation and relies on LLM-judged improvements, reducing evidential strength.