Fine-tuning Multi-modal LLMs with ART: Art-based Reinforcement Training
Michal Chudoba, Sergey Alyaev, Petra Galuscakova, Tomasz Wiktorski
Abstract
There are two main Parameter-Efficient Fine-Tuning (PEFT) techniques for Large Language Models (LLMs). While Low-Rank Adaptation (LoRA) introduces additional weights between the LLM layers, Soft Prompting introduces additional fine-tuning-specific raw tokens to an LLM input. However, both require modification to the computational graphs of precompiled, preoptimized LLMs. As a result, neither is fully supported in high-throughput engines like vLLM. We propose fine-tuning with ART (Art-based Reinforcement Training). The method injects information into a frozen Multimodal Large Language Model (MLLM) by optimizing only its raw visual input, thus enabling the soft-token approach on pre-compiled computational graphs. It relies on backpropagation of gradients back into a plain pixel array and thus supports any fine-tuning objective. Moreover, the optimized visual input can be stylized as task-relevant computational artworks. The approach's effectiveness is confirmed for different sizes of a popular open Qwen architecture and for several textual benchmarks. Specifically, ART reaches accuracy competitive with LoRA across mathematics and structured-tool-use benchmarks.
AI Impact Assessments
(1 models)Scientific Impact Assessment: Fine-tuning Multi-modal LLMs with ART
1. Core Contribution
ART proposes adapting frozen multimodal LLMs by optimizing a single input image in pixel space, rather than modifying model weights (LoRA) or injecting continuous embeddings (soft prompting). The key insight is that the vision pathway of MLLMs provides a differentiable, continuous channel into the model's embedding space that can be exploited for task adaptation without touching the computational graph. The optimized image is deployed as a standard PNG file, meaning the serving infrastructure treats it as an ordinary multimodal request — no adapter loading, no CUDA graph invalidation, and full compatibility with production engines like vLLM.
The method uses a logit-space parameterization for unconstrained optimization, a two-pass training loop (rollout via serving engine, backpropagation through a frozen model copy), and instantiates the objective with DAPO (a GRPO variant). The contribution is primarily the *where* gradients land (pixels rather than weights), not the optimizer itself.
2. Methodological Rigor
Strengths in experimental design: The paper includes appropriate controls — random images, random strings, fixed seed images — that help disentangle the contribution of ART optimization from the mere activation of the vision tower. The random-string control at matched token count (64 tokens) is particularly valuable, ruling out that the benefit is simply from increased sequence length. Bootstrap confidence intervals with 10,000 resamples add statistical rigor.
Weaknesses: The experimental scope is limited to a single architecture family (Qwen3.5) at only two small scales (0.8B, 2B). The benchmarks are limited to three tasks, with GPQA showing ART actually *hurts* performance. The GPQA evaluation split is only 273 examples with wide confidence intervals, making it hard to draw conclusions. The comparison with LoRA uses identical training conditions (same DAPO loss, same steps, same batch size), which is fair but also somewhat constraining — LoRA might benefit from different hyperparameters or more training steps.
A significant confound is the large baseline boost from simply prepending *any* image. On the 0.8B model, a random image improves GSM8K from 39.65% to 54.59%, while ART optimization adds only another ~4 percentage points (to 58.53%). This raises the question of how much of ART's value comes from the optimization versus the implicit activation of ~100M additional ViT parameters. The authors acknowledge this but don't fully resolve it.
The LoRA baseline underperforming a random image on 0.8B GSM8K (49.51% vs. 54.59%) is surprising and somewhat undermines the comparison — it suggests the LoRA configuration may not be well-tuned, or that the 100-step budget is insufficient for weight-space adaptation.
3. Potential Impact
Deployment advantage: The most compelling practical argument is compatibility with production serving infrastructure. If ART artifacts can be served as standard multimodal requests without adapter management overhead, this simplifies deployment considerably. The 2-3x training speedup over LoRA is also notable.
Limited generality: The method is restricted to multimodal LLMs with vision encoders, which narrows applicability. Many deployment scenarios involve text-only models. The capacity of a 64-token visual prefix (from a 256×256 image) may be fundamentally limited for complex tasks, as suggested by GPQA results.
Steganography angle: The observation that optimization deposits structured high-frequency information into images, measurable via PNG file size growth, is intellectually interesting but more of a characterization than a separate contribution.
4. Timeliness & Relevance
The paper addresses a real pain point: serving multiple LoRA adapters in production is genuinely difficult with current infrastructure. The trend toward smaller, locally-served models makes efficient adaptation important. The connection to GRPO/DAPO-style training is timely given the DeepSeek-R1 wave. However, the restriction to multimodal models for what are fundamentally text tasks feels somewhat forced — it requires using a more expensive model (with a vision tower) to avoid the complexity of LoRA serving.
5. Strengths & Limitations
Key Strengths:
Notable Limitations:
Additional Observations
The framing around "computational art" and "steganography for AI" is creative but somewhat superficial — it adds aesthetic appeal without deepening the technical contribution. The information storage analysis via PNG compression is a proxy at best.
The observation that ViT activation alone provides large boosts on small models is arguably the most interesting finding in the paper, yet it's treated as background rather than a core contribution. Understanding *why* this happens could have broader implications.
Generated Jun 11, 2026
Comparison History (19)
While Paper 1 presents a highly creative fine-tuning approach for MLLMs, Paper 2 establishes a foundational, large-scale benchmark that directly resolves a critical 'comparability crisis' in Wearable Human Activity Recognition. By standardizing 30 datasets and evaluating numerous architectures for both accuracy and on-device efficiency, Paper 2 provides immense methodological rigor and is highly likely to become the standard evaluation framework in its field, ensuring broad and sustained scientific impact.
Paper 2 likely has higher scientific impact due to broader applicability and stronger real-world relevance: it introduces a PEFT-like method that works with frozen, precompiled MLLMs and high-throughput inference engines (e.g., vLLM), addressing a practical deployment bottleneck. Optimizing raw visual inputs as a universal adaptation channel is a novel, timely idea with potential cross-domain uses (efficient customization, secure/controlled adaptation, multimodal prompting). Paper 1 is rigorous and valuable for interpretability science but is narrower in application and impact beyond mechanistic interpretability research.
Paper 1 has higher likely scientific impact due to its deeper, broadly relevant mechanistic analysis of on-policy distillation—an increasingly important post-training paradigm. It provides systematic empirical findings on sparsity and parameter-space geometry (layer/FFN distribution, optimizer implications, spectral structure), yielding actionable insights (subnetwork training) and generalizable understanding that can influence optimization, interpretability, and efficient adaptation across many models and domains. Paper 2 is innovative and practical for deployment constraints (vLLM-compatible PEFT via pixel optimization), but its impact is narrower and may face robustness/generalization limits as a task-specific tuning hack.
Paper 2 addresses a highly critical bottleneck in deploying fine-tuned Large Language Models by enabling PEFT without modifying computational graphs, making it compatible with high-throughput inference engines like vLLM. Its novel approach of using optimized visual inputs as soft prompts is highly innovative and has immediate, widespread real-world applications in the rapidly growing field of MLLMs. While Paper 1 offers a solid methodological improvement for diffusion models, Paper 2's relevance, timeliness, and potential to streamline MLLM deployment across various domains give it a higher potential for broad scientific and practical impact.
Paper 2 provides fundamental theoretical contributions to asynchronous SGD with clipping, proving convergence guarantees under heavy-tailed noise with high probability—a first in asynchronous optimization. This addresses a core challenge in distributed/federated learning at scale, with broad applicability across all large-scale ML training. Paper 1, while creative in its approach to PEFT via visual input optimization, addresses a narrower problem (fine-tuning MLLMs without modifying computational graphs) and achieves competitive rather than superior results compared to LoRA. Paper 2's theoretical foundations have wider impact potential across distributed systems and optimization theory.
Paper 1 is more novel and broadly impactful: it introduces a new PEFT-like paradigm by optimizing only raw visual inputs to adapt frozen multimodal LLMs without altering computational graphs, directly addressing a timely systems bottleneck (compatibility with high-throughput engines like vLLM). If robust, this could generalize across objectives, models, and deployment settings, affecting both multimodal learning and LLM serving. Paper 2 is methodologically sound and useful, but its innovation (rarity-gated FiLM) is more incremental and its demonstrated impact is narrower (maritime anomaly detection), limiting cross-field reach.
Paper 2 identifies a fundamental failure mode ('categorical prior lock-in') in LLM in-context learning for structured data, providing mechanistic understanding with broad implications for the growing field of LLM-based data generation. It addresses critical issues of adaptability vs. privacy trade-offs. Paper 1 proposes ART, an interesting but incremental PEFT variant that optimizes visual inputs for MLLMs. While creative, its practical advantages over LoRA are modest, and the 'art stylization' aspect is more aesthetic than scientifically impactful. Paper 2's diagnostic contribution is more likely to influence future research directions.
Paper 1 likely has higher scientific impact: it offers a principled, general theoretical formulation extending RoPE to arbitrary n-D domains with an isotropy condition and a concrete wave-vector design, validated across images/videos/point clouds—broadly useful for many Transformer-based spatial/temporal modalities. This combination of novelty, methodological rigor, and cross-field applicability suggests durable influence. Paper 2 is timely and practical for deployment constraints (no graph changes) but appears more like an engineering workaround (optimizing pixels) with narrower generality and potentially higher brittleness/limited theoretical grounding.
Paper 1 introduces a fundamental shift in how relational learning models are evaluated by revealing that dataset geometry (curvature) is a critical latent factor governing model performance. This curvature-stratified evaluation framework has broad implications across graph learning, affecting how benchmarks are designed and how models are compared. It provides actionable insights (e.g., GFMs showing diminishing returns in certain regimes) and releases reproducible tools. Paper 2 presents an interesting but more incremental PEFT technique with narrower scope—optimizing pixel inputs for MLLMs—and its practical advantages over LoRA remain limited given competitive rather than superior performance.
PUMA addresses a fundamental training efficiency problem in Masked Diffusion Models with a simple, principled solution (aligning training and inference masking patterns) that yields significant 2.5x speedups. It has broad applicability to the growing field of discrete diffusion models and is methodologically rigorous. Paper 2 proposes an interesting but niche PEFT method that optimizes pixel inputs to frozen MLLMs. While creative, its practical advantages over LoRA are limited, the approach is constrained to multimodal models, and the competitive-with-LoRA results don't demonstrate clear superiority. PUMA's impact on training efficiency for an important model class gives it higher potential impact.