QQJ: Quantifying Qualitative Judgment for Scalable and Human-Aligned Evaluation of Generative AI
Marjan Veysi, Pirooz Shamsinejadbabaki, Mohammad Zare, Mohammad Sabouri
Abstract
The rapid progress of generative artificial intelligence has exposed fundamental limitations in existing evaluation methodologies, particularly for open-ended, creative, and human-facing tasks. Traditional automatic metrics rely on surface-level statistical similarity and often fail to reflect human perceptions of quality, while purely human evaluation, although reliable, is costly, subjective, and difficult to scale. Recent approaches using large language models as evaluators offer improved scalability but frequently lack explicit grounding in human-defined evaluation principles, leading to bias and inconsistency. In this paper, we introduce Quantifying Qualitative Judgment (QQJ), a scalable and human-centric evaluation framework that explicitly bridges the gap between human judgment and automated assessment. QQJ separates the definition of quality from its execution by anchoring evaluation in expert-designed, multi-dimensional rubrics and calibrating large language model evaluators to align with expert reasoning using a small, high-quality annotation set. This design enables consistent, interpretable, and scalable evaluation across diverse generative tasks and modalities. Extensive experiments on text and image generation demonstrate that QQJ achieves substantially stronger alignment with human judgment than traditional automatic metrics and unconstrained LLM-based evaluators. Moreover, QQJ exhibits improved stability across repeated evaluations and superior diagnostic capability in identifying critical failure modes such as hallucination and intent mismatch. These results indicate that structured qualitative judgment can be operationalized at scale without sacrificing interpretability or human alignment, positioning QQJ as a practical foundation for reliable evaluation of modern generative AI systems.
AI Impact Assessments
(1 model)Scientific Impact Assessment: QQJ Framework
1. Core Contribution
QQJ proposes a four-stage evaluation pipeline for generative AI: (1) expert-designed multi-dimensional rubric construction, (2) small-scale expert annotation to create a calibration set, (3) alignment of an LLM evaluator to expert reasoning via rubric-guided prompting, and (4) scalable automated evaluation. The central idea is separating the *definition* of quality (by humans) from its *execution* (by LLMs). This is a reasonable conceptual framing, but the underlying idea—using rubrics to guide LLM-based evaluation and calibrating against human annotations—is not fundamentally new. G-Eval, HELM, and numerous rubric-guided LLM evaluation papers have explored similar territory. The paper positions itself as synthesizing these ideas into a unified framework, but the incremental novelty over existing work (particularly G-Eval and structured prompting approaches) is not clearly articulated.
2. Methodological Rigor
This is the paper's most significant weakness. Several critical methodological details are missing or underspecified:
Lack of implementation specifics. The "evaluator alignment" stage is described abstractly as minimizing a loss function between expert and LLM scores, but it is never clarified whether this involves fine-tuning, in-context learning, iterative prompt refinement, or some other mechanism. The formulation presents a minimization over θ (LLM parameters), but the text states "rather than training a new evaluation model, QQJ leverages rubric-guided prompting." These two descriptions are contradictory—if only prompting is used, how is the loss function minimized? This is a fundamental gap.
Vague experimental setup. The paper mentions "three representative generative models spanning different capability regimes" but never names them. The 1,200 text prompts and 600 image prompts are described as "curated" without specifying sources, selection criteria, or availability. The number of expert annotators, their qualifications, inter-annotator agreement statistics, and the size of the calibration set (N) are all unspecified.
Missing baselines and statistical rigor. The comparison in Table II shows Spearman correlations (e.g., QQJ achieves 0.78 for text, 0.73 for image), but no confidence intervals, significance tests, or standard deviations are reported. The improvements over G-Eval (0.63 → 0.78) and LLM Reviewer (0.61 → 0.78) are substantial claims that require statistical validation. The variance comparison in Table III similarly lacks error bars or significance testing.
Questionable reference practices. References [14] and [15] have placeholder author names ("A. of Paper 1," "A. of Paper 2") and incomplete citation information, which is unprofessional. Reference [13] contains "arXiv:2402.XXXX," suggesting an incomplete or fabricated citation.
No reproducibility artifacts. No code, rubrics, datasets, or prompts are released or described in sufficient detail for reproduction.
3. Potential Impact
The *idea* of structured, rubric-guided evaluation calibrated to expert judgment addresses a genuine need. If properly validated, such a framework could influence how practitioners evaluate generative models, particularly in production settings where interpretability matters. The diagnostic capability for hallucination and intent mismatch (Table IV) is a practically relevant contribution.
However, the lack of methodological transparency severely limits the paper's potential influence. Without knowing how the framework is actually implemented, other researchers cannot build upon, reproduce, or compare against QQJ.
4. Timeliness & Relevance
The paper addresses a timely problem. LLM-as-judge evaluation is an active research area with significant practical demand. The focus on structured evaluation, calibration, and diagnostic capability aligns well with current community needs. The inclusion of image generation evaluation alongside text is a positive differentiator, though the image evaluation results are limited (only LLM Reviewer is compared for image generation in Table II).
5. Strengths & Limitations
Strengths:
Limitations:
Additional Observations
The writing is generally clear and well-organized, but the paper reads more as a position paper or framework proposal than an empirical contribution. The experimental section, while structured around reasonable evaluation criteria, lacks the depth and rigor expected for the strong claims made. The paper's framing suggests broad applicability, but evidence is limited to a single experimental setup without cross-validation across different rubric designs, evaluator models, or domains.
The comparison table (Table I) is useful for positioning but somewhat misleading—several entries characterize prior work unfavorably without detailed justification (e.g., HELM as only "Partial" scalability, or RLHF as not scalable).
Generated May 19, 2026
Comparison History (25)
Paper 1 addresses a broad, timely, and widely relevant problem—evaluating generative AI across text and image modalities—with a scalable, interpretable framework validated by extensive experiments. Its impact spans many fields relying on generative models. Paper 2 tackles an important but narrower issue (success provenance in agent evaluation) with a more specialized methodology and dense presentation, limiting its accessibility and breadth of adoption. While rigorous, Paper 2's audience and applicability are more constrained, whereas Paper 1 provides a practical foundation applicable across the rapidly growing generative AI evaluation landscape.
Paper 2 proposes a fundamental paradigm shift from model scaling to system scaling in Agentic AI, a highly timely and rapidly expanding frontier. While Paper 1 offers a valuable evaluation framework, Paper 2's conceptualization of the 'agent harness' and its proposed research agenda for architectural design, governance, and evaluation have the potential to broadly influence how future autonomous systems are built and assessed across the entire AI ecosystem.
Paper 1 (PStar) addresses a more critical and timely problem—reducing hallucinations in VLMs for robotic automation with direct safety implications. Its novel pseudocode-guided reasoning framework with adaptive difficulty assessment offers a concrete, actionable contribution with strong empirical results (outperforming GPT-4V). It has broader impact across robotics, AI safety, and VLM research. Paper 2 (QQJ), while valuable for evaluation methodology, addresses a more incremental improvement in AI evaluation frameworks, which is a narrower, less urgent problem with fewer downstream applications.
Paper 1 addresses a universal and critical bottleneck in generative AI: scalable, human-aligned evaluation. Its framework for bridging human judgment with automated assessment is broadly applicable across diverse modalities (text, image) and domains, promising widespread adoption. In contrast, Paper 2 provides valuable but highly domain-specific insights into LLM behavior within hardware-aware code optimization. Because robust evaluation methodologies are foundational to advancing all GenAI research, Paper 1 has a significantly larger potential audience and broader cross-disciplinary impact.
Paper 2 likely has higher impact: it introduces a principled RL optimization objective (forward-KL–style distribution matching) addressing a core failure mode (mode collapse) with demonstrated gains across combinatorial optimization, math reasoning, and out-of-domain tasks, suggesting broad methodological relevance to RLHF/LLM reasoning training. The approach is timely for on-policy LLM RL and could influence both theory (divergence choice, exploration) and practice (training stability and diversity). Paper 1 is valuable for evaluation, but rubric-calibrated LLM judging is closer to incremental systematization and may face domain-specific subjectivity limits.
While Paper 1 addresses an important problem in GenAI evaluation, the LLM-as-a-judge space is heavily saturated. Paper 2 tackles a critical bottleneck in LLM agent scalability (inference cost and long horizons) by introducing a learned latent action space. This represents a more fundamental architectural and algorithmic innovation that bridges representation learning with agentic planning, likely spurring more significant follow-up research in the rapidly growing field of efficient autonomous AI agents.
Paper 1 makes a stronger theoretical contribution by formalizing interface-constrained SMDPs, providing the first finite-sample guarantee for neural Q-learning under decentralized partial observability, and lifting the AIS framework to multi-agent SMDPs. This addresses a fundamental problem in multi-agent RL with broad implications for LLM pipelines across trust boundaries. Paper 2 proposes a useful evaluation framework (QQJ) but is more incremental—combining rubrics with LLM calibration—and operates in a crowded evaluation methodology space with less theoretical depth and narrower long-term impact.
Paper 2 addresses a foundational and critical bottleneck in generative AI: scalable, human-aligned evaluation. By introducing a framework applicable across multiple modalities (text and image) that outperforms traditional metrics, it has the potential to become a widely adopted benchmark in the field. Evaluation methodologies generally have a broader impact across various subfields compared to domain-specific behavioral analyses like the negotiation limits explored in Paper 1.
Paper 1 addresses a highly critical and timely bottleneck in artificial intelligence: the reliable, scalable, and human-aligned evaluation of generative AI. Its framework for quantifying qualitative judgment has broad applicability across multiple modalities (text, images) and tasks, impacting the broader AI community. In contrast, Paper 2 presents an architectural tweak to PPO for a specific multi-UAV application. While useful in robotics and communications, its scope, novelty, and cross-disciplinary impact are significantly narrower than the foundational AI evaluation problem solved in Paper 1.
Paper 1 addresses a critical bottleneck in generative AI research—scalable, human-aligned evaluation—with a rigorous, multi-modal methodology supported by extensive experiments. Its contribution has broad implications for how AI models are assessed across the field. In contrast, Paper 2 presents a software engineering framework for API abstraction, which, while useful for developers, offers limited methodological innovation and scientific impact compared to advancing foundational AI evaluation techniques.