Investigating Concept Alignment Using Implausible Category Members

Scientific Impact Assessment

Core Contribution

This paper introduces a methodologically simple but conceptually clever approach to probing concept alignment between humans and LLMs: instead of asking about plausible category memberships (which likely overlap heavily with training data), the authors ask about *implausible* memberships (e.g., "Is an olive a vehicle?"). Using the classic Rosch and Mervis (1975) superordinate categories and their 120 exemplar objects, the authors construct 708 within-category and cross-category membership questions, administer them to seven LLMs and 563 human participants, and systematically identify where concept boundaries diverge. The key finding is that LLMs are systematically more permissive than humans in assigning objects to categories, with specific failure modes around "words" being assigned to multiple categories, vegetables being classified as fruit, and over-identification of objects as weapons. The paper also demonstrates that these concept-level misalignments produce problematic downstream behavior.

Methodological Rigor

The experimental design is straightforward and well-executed. The use of a 0-10 rating scale is reasonable, and collecting 20 responses per question per model and at least 20 human judgments provides adequate statistical power. The Mann-Whitney U test is an appropriate choice given the distributional properties of the data, and the Šidák correction for multiple comparisons is properly applied.

However, there are several methodological concerns:

1. Temperature setting: Using temperature 0.7 introduces stochasticity that may inflate or deflate disagreement measures. The choice isn't well-justified, and sensitivity analysis across temperature values is absent.

2. Single prompt format: All results rely on one specific prompt formulation. LLMs are known to be sensitive to prompt phrasing, and the paper doesn't test robustness to paraphrasing. The instruction to "Answer only with a number from 0-10" may interact differently with different models' instruction-following tendencies.

3. Downstream safety demonstrations: Table 2's follow-up questions, while provocative, are presented in multi-turn interactions following the initial categorization question. This creates a priming effect that artificially inflates the likelihood of problematic downstream responses. The safety implications would be more convincing if the downstream scenarios were presented independently.

4. Human baseline concerns: Each human answered only 30 randomly drawn questions, while models answered all 708. The random sampling strategy is reasonable but means individual human responses to any given question come from different participants with potentially different interpretive strategies.

5. Limited stimuli set: The entire analysis rests on 120 objects and 6 categories from a 1975 study. While historically grounded, this is a narrow slice of conceptual space.

Potential Impact

The paper makes a genuine contribution to the growing literature on AI alignment and behavioral evaluation of LLMs. The core insight—that probing implausible category memberships reveals misalignments invisible to standard evaluations—is valuable and transferable. The approach could be extended to other conceptual domains, taxonomies, and tasks.

The practical implications are moderate. The identified misalignments (e.g., vegetables classified as fruit, overly broad weapon categorization) are concerning for deployed systems, particularly in safety-critical applications. The connection to legal precedents (McBoyle v. United States, Nix v. Hedden) is rhetorically effective and highlights that concept boundaries matter in real-world governance.

However, the paper's impact may be limited by:

Timeliness & Relevance

The paper is highly timely. As LLMs are deployed in increasingly consequential settings (content moderation, legal reasoning, autonomous systems), understanding whether they share basic human conceptual frameworks is important. The paper contributes to an active and growing body of work applying cognitive science methods to LLM evaluation. The concept alignment framing—where alignment on basic concepts is argued to be prerequisite to value alignment—is a useful theoretical contribution that connects to broader AI safety concerns.

The observation that safety fine-tuning may cause over-identification of weapons is particularly relevant to current debates about RLHF and its unintended consequences.

Strengths

1. Elegant experimental logic: The cross-category design naturally generates "obvious" questions that bypass training data memorization, providing a cleaner signal about conceptual understanding.

2. Comprehensive model coverage: Testing seven diverse LLMs allows identification of both systematic and idiosyncratic misalignments.

3. Human baseline: Collecting substantial human data (563 participants) provides a principled comparison standard.

4. Accessible and interpretable findings: The results are immediately understandable and memorable (e.g., models thinking "words" are vehicles).

5. Connection to cognitive science foundations: Grounding in Rosch and Mervis gives the work theoretical depth and positions it within a well-established research tradition.

Limitations

1. No mechanistic analysis: The paper identifies misalignments but offers only speculation about causes (training data composition, RLHF artifacts, textual modality). No probing of internal representations is attempted.

2. Narrow stimulus set: 6 categories and 120 objects from 1975 may not capture the full range of conceptual misalignment.

3. Questionable downstream demonstrations: The multi-turn priming setup and cherry-picked examples overstate safety risks. A systematic evaluation of downstream consequences would be more convincing.

4. No intervention or mitigation: The paper identifies problems but offers no solutions or even preliminary investigation of how to correct misalignments.

5. Ecological validity: It's unclear how often real users would encounter situations where these specific misalignments matter in practice.

6. GPT-5 exclusion: The exclusion due to empty responses, without further investigation, is a missed opportunity.

Overall Assessment

This is a well-conceived, clearly written paper that makes a useful empirical contribution to understanding concept alignment between humans and LLMs. The core methodology is sound and the findings are genuinely interesting and sometimes surprising. However, the work is primarily descriptive, limited in scope (small stimulus set, no mechanistic investigation, no proposed solutions), and the safety implications, while thought-provoking, are somewhat overstated given the artificial nature of the downstream demonstrations. The paper opens an interesting research direction rather than resolving one.