Visual Task Modeling and Automated Planning for Humanoid Robots in Education of Individuals with Autism Spectrum Disorder

Lada Kudláčková

Oct 19, 2026arXiv:2610.27162v1

cs.ROcs.AI

Scorecard· 14/16

3.0/10 impact

Well-motivated assistive-education prototype, but incremental over the author's own prior work, with a small, mostly qualitative evaluation and no robot or user validation.

Abstract

In robot-assisted education, a robot must understand how to complete a task in order to effectively guide a student through it. We present a novel method that enables a semi-autonomous, camera-equipped robot to recognise the task goal and plan actions to achieve it. Our work focuses on Shoebox Tasks, an activity commonly used in the education of individuals with Autism Spectrum Disorder (ASD). These tasks are designed so that the goal can be inferred directly from their visual structure. We have developed a solver that uses a vision-based Large Language Model to convert photos of a Shoebox Task into a Planning Domain Definition Language (PDDL) representation and generate action plans for solving it. We have also implemented a robotic interface that enables a Nao robot to autonomously execute various Shoebox Tasks and assist a student by suggesting next steps or signalling detected failures through action demonstrations, voice commands, or gestures. With an adapted robotic interface, the solver can be integrated into a variety of educational robots. No programming expertise is required from teachers, and the solver operates exclusively on captured images as input. Research in special educational needs indicates that students with ASD can benefit substantially from robot-assisted teaching, and our system aims to strengthen these learning opportunities while also fostering interest in robotics among individuals with ASD.

AI Impact Assessments

(1 model)

Paper type: tool/system paper with a small empirical evaluation. It is an incremental extension of the author's own RiE 2025 paper.

Core Contribution

The paper presents a pipeline that photographs a "Shoebox Task" (a TEACCH-style visually structured activity for students with ASD) and passes the image to a vision-language model (ChatGPT-4o or ChatGPT-5). The VLM produces a PDDL problem file with an embedded action plan. A perception–analysis–action loop then re-photographs the scene and asks the VLM to infer which actions have been executed, update the PDDL predicates, and prune the remaining plan. A Nao robot interface either executes pick-and-place actions or gives verbal and gestural guidance.

The claimed increments over the earlier RiE 2025 work are:

a PDDL validator with automatic retry on refusals;

detection of action sequences between images;

planning done by the VLM itself, without an external Fast Downward call.

The core idea is to use the strong visual structure of a pedagogically established activity as a grounding for VLM-to-PDDL translation. This is sensible and well motivated. Its novelty is modest, though. VLM-to-PDDL pipelines are already common, and the domain here is a very simple three-predicate sorting and matching world. In it, the "planning" is essentially an assignment of items to targets, which trivialises the planning component. The decision to drop the external planner is presented as a contribution, but it likely reflects the problem's triviality and removes the soundness guarantees a classical planner provides.

Methodological Rigor

The evaluation is thin.

Experiment 1 covers six tasks with 3–4 items, ten trials each, and two model versions. It reports only error counts and latency, with no confidence intervals and no baselines such as a non-VLM perception pipeline or simpler prompting.

Correctness is judged by manual inspection against loosely defined error categories.

The prompts and domain were iteratively tuned on failure cases and then frozen. The paper does not clarify whether the test tasks were held out from this tuning, so overfitting is plausible.

Default API settings were used, so there is no control over temperature or reasoning effort.

The state-update experiment (Experiment 2), which is the actual novelty, is reported only qualitatively. There are no numbers, no per-task success rates, and no breakdown of failures. The narrative admits that errors become frequent for nine-item tasks, including failure to prune the plan, which is precisely the capability the paper claims.

The system was evaluated only on the solver. All actions were executed manually, and the robot was never tested with students or even in a full end-to-end loop. The Nao's own camera is not used.

No user study, no teacher evaluation, and no educational outcome measures exist. The abstract's framing of strengthening learning opportunities for ASD students is therefore entirely unvalidated.

Plans are an unordered set and the domain is trivially simple, so the claim of "planning" is weak.

Potential Impact

The application is socially meaningful (special education, low-barrier teacher tools, no programming required). The idea of translating visual demonstrations into PDDL for learning by demonstration is attractive but only asserted. Realistic impact is limited:

the system is a prototype relying on a proprietary API with variable latency (up to 86 s for GPT-5, which is impractical for interactive tutoring of children);

reliability degrades beyond four items;

there is no safety or child-facing validation.

The work could serve as a proof of concept for researchers in assistive HRI. It could also inspire follow-up work on VLM-driven task monitoring, but it is unlikely to become a reference. It is a conference paper (AICCONF), likely with a small audience.

Timeliness & Relevance

Using VLMs as zero-shot scene-to-symbolic translators is timely, and so is the interest in LLM-based robot tutors. However, the contribution does not advance the state of the art in neuro-symbolic planning or in social robotics. Aging hardware (Nao v5) and cloud-dependent models also reduce practical relevance.

Strengths

Domain expertise: collaboration with the creator of Shoebox Tasks grounds the problem in real pedagogy.

It is honest about limitations, such as medium-scale failures, hexagon/pentagon confusion, and refusals.

The design is practical: a web upload interface, a validator, and a simple domain that teachers could in principle reuse.

The comparison of GPT-4o and GPT-5 gives a useful practical datapoint on the latency–reliability tradeoff, and GPT-5 shows fewer refusals.

Limitations

Very small evaluation with no statistics, baselines, or quantified state-tracking results.

No end-to-end robot experiments and no user study.

The PDDL domain is trivially simple, and the VLM acts as both perceiver and planner without verification of plan optimality or correctness beyond syntax.

Reproducibility is hampered by proprietary models, an unreleased prompt and code (not stated as available), and model versions that change over time.

The paper is heavily self-referential, with an extension of the author's own earlier work.

The claims about learning by demonstration, teaching programming, and fostering interest in robotics are speculative.

The numbered contributions overstate novelty.

Overall

This is a modest, well-motivated prototype with limited empirical support. Its value lies mainly in the application framing and as a small datapoint on VLM reliability for structured visual tasks. Expected citation and influence are low to moderate.

Rating:3/ 10

Significance 2.5Rigor 3Novelty 3Clarity 6

Generated Oct 21, 2026

Comparison History (0)

No comparisons yet.

Scorecard· 14/16

3.0/10 impact

Well-motivated assistive-education prototype, but incremental over the author's own prior work, with a small, mostly qualitative evaluation and no robot or user validation.