Despite rapid advances in language and vision models, current robots still lag far behind human physical capabilities due to the relative scarcity of real-world data compared to online text and images. How can we leverage abundant language data to advance robotic capabilities? Language provides semantic structure that facilitates the understanding of diverse data, improving sample efficiency in scarce data regimes. It also provides a natural communicative medium when interacting with and learning from humans.
To leverage the first benefit of language, we first take inspiration from how humans teach each other in video tutorials, through simultaneous video and language streams, to more efficiently teach robots new skills. We then show that language can bridge wide visual sim2real gaps, enabling robots to learn tasks with just a few real-world demonstrations by leveraging knowledge from imperfect simulation data. To leverage the second benefit of language, we explore how dialog can enable robots to solve complex manipulation tasks by communicating and collaborating with a wide distribution of human collaborators in the real-world. We develop a robotic framework that requests and proactively offers help through mixed-initiative, free-form dialog, enabling the robot to adapt to changing human preferences and strategically utilize each agent’s physical capabilities. Finally, to accelerate how robots learn to operate new household devices, we combine both benefits of language into a framework that leverages semantics from rich textual corpora, such as user manuals, to establish skill priors that are efficiently refined through active human dialog.
Overall, our thesis provides a foundation for leveraging the key benefits of language to improve the generalization capabilities of robotic manipulation policies to new tasks, domains, devices, and human collaborators.
ML ID: 449
Large language models have substantially advanced automatic code generation, but competitive programming remains a demanding test of their reasoning abilities. Solving these problems requires more than producing syntactically valid code: a model must abstract a narrative specification into its underlying mathematical structure, discover an efficient algorithm, develop the algorithm into a complete plan that accounts for correctness, complexity, and edge cases, and finally translate the plan into an executable program that succeeds on unseen tests under strict resource constraints. This complex process requires strategy exploration and problem-solving skills, and is beyond implementing straightforward functionalities. Human competitors typically develop these skills through algorithmic principles, worked examples, and expert-written editorials rather than through problem–program pairs alone. This thesis proposes to utilize natural language as an intermediate representation between problem understanding and reliable program synthesis.
This thesis investigates how to leverage and improve LLMs’ abilities in natural language reasoning, specifically in the context of solving competitive-level programming problems. It first decomposes competitive programming problem-solving into strategy discovery, specific verbal solution details, and implementation. Across the evaluated settings, large language models can often explain verified human solutions and implement programs from detailed verbal descriptions even when they struggle to solve the same problems directly. These results suggest that the main bottleneck lies less in translating a complete solution into code than in discovering the correct algorithmic strategy and specifying it with sufficient precision.
Building on this diagnosis, the thesis develops an explanation-based distillation framework that first automatically generates editorial-style reasoning from verified human programs and uses these explanations to teach a separate reasoning model to guide code generation on new problems. The findings show that semantically rich natural-language supervision transfers algorithmic knowledge more effectively than directly training in the problem-to-code setting and encourages the use of more efficient strategies rather than superficial or brute-force implementations.
The thesis then introduces CodeTree, which operationalizes the separation of reasoning and implementation through an agent-guided search process: distinct roles propose strategies, implement programs, diagnose failures, and evaluate candidate solutions, while execution evidence and model-generated critique determine whether a search path should be expanded, revised, discarded, or accepted. This study finds that exploring diverse strategies and making informed search decisions are generally more effective than repeatedly refining a single initial solution, although successful role decomposition depends on the underlying model’s ability to follow specialized instructions.
The thesis next examines whether stronger problem-solving performance implies transferable algorithmic understanding. AlgoSimBench evaluates whether models can recognize problems that share an underlying solution method despite differences in wording and narrative context, and demonstrates that solving individual programming tasks and recognizing reusable algorithmic structure are distinct capabilities. The results also show that comparing generated solution attempts can expose algorithmic structure more clearly than comparing raw problem statements, even when the attempted solutions are not fully correct.
Finally, the thesis extends this investigation to code retrieval through InstEmbed, an instruction-sensitive representation-learning framework in which the desired target, definition of similarity, and relevant portion of the query jointly determine what should be retrieved. By contrasting documents that are related to the same query but valid under different instructions, the framework learns to distinguish general semantic relevance from task-specific usefulness and improves both target-aware and noise-robust retrieval.
Together, these studies establish natural language as a practical intermediate representation for learning, exploring, evaluating, and retrieving algorithmic knowledge, bridging the gap between problem and code, while showing that reliable code generation requires explicit attention to both the structure of the reasoning process and the generalizability of the representations it produces.
ML ID: 448
Large multimodal models have become central for solving sequential decision-making tasks, enabling improved learning in diverse areas such as home robotics and automated software development. However, leveraging these models for sequential decision-making requires robust action reasoning capabilities, which remain a significant challenge. This dissertation improves and evaluates action reasoning in large multimodal models. First, we introduce a method to improve the parsing of instructional texts into action sequences by integrating external symbolic planners and planning domains during autoregressive language model decoding. Next, we develop a method that leverages the compositional structure of language instructions to improve generalization and sample efficiency of acquiring new tasks with reinforcement learning. We also construct a new benchmark to evaluate the understanding of dependencies between actions described in instructional texts. Finally, we evaluate the world modeling limitations of frontier models through multimodal entity state tracking. Current models struggle to reason about the effects of actions in multimodal entity state tracking tasks. We extend entity state tracking evaluations to a simulated embodied environment and derive insights for improving the entity-state reasoning abilities of language and vision-language models. Together these contributions enhance the understanding of how models reason about actions and provide insights toward their improvement for real-world sequential decision-making problems.
ML ID: 447
Vision-Language Models (VLMs) have shown strong multimodal reasoning capabilities on Visual-Question-Answering (VQA) benchmarks. However, their robustness against textual misinformation remains under-explored. While existing research has studied the effect of misinformation in text-only domains, it is not clear how VLMs arbitrate between contradictory information from different modalities. To bridge the gap, we first propose the CONTEXT-VQA (i.e., Conflicting Text) dataset, consisting of image-question pairs together with system-atically generated persuasive prompts that deliberately conflict with visual evidence. Then, a thorough evaluation framework is designed and executed to benchmark the susceptibility of various models to these conflicting multimodal inputs. Comprehensive experiments over 11 state-of-the-art VLMs reveal that these models are indeed vulnerable to misleading textual prompts, often overriding clear visual evidence in favor of the conflicting text, and show an average performance drop of over 48.2 percent after only one round of persuasive conversation. Our findings highlight a critical limitation in current VLMs and underscore the need for improved robustness against textual manipulation.
ML ID: 446
Recent progress in LLMs, such as reasoning models, has demonstrated strong abilities to solve complex competitive programming problems, often rivaling top human competitors. However, it remains underexplored whether these abilities generalize to relevant domains that are less seen during training. To address this, we introduce AlgoSimBench, a new benchmark designed to assess LLMs' ability to identify algorithmically similar problems (ASPs)-problems that can be solved using similar algorithmic approaches. AlgoSimBench consists of 1317 problems, annotated with 231 distinct fine-grained algorithm tags, from which we curate 402 multiple-choice questions (MCQs), where each question presents one algorithmically similar problem alongside three textually similar but algorithmically dissimilar distractors. Our evaluation reveals that LLMs struggle to identify ASPs, with the best-performing model (o3-mini) achieving only 65.9 percent accuracy on the MCQ task. To address this challenge, we propose attempted solution matching (ASM), a novel method for improving problem similarity detection. On our MCQ task, ASM yields an absolute accuracy improvement of 6.7 percent to 11.7 percent across different models. We also evaluated code embedding models and retrieval methods on similar problem identification. While the adversarial selection of problems degrades the performance to be less than random, we found that simply summarizing the problem to remove narrative elements eliminates the effect, and combining ASM with a keyword-prioritized method, BM25, can yield up to 52.2 percent accuracy. Code and data are available at https://github.com/lijierui/AlgoSimBench.
ML ID: 442
Effective robotic systems for long-horizon human-robot collaboration must adapt to a wide range of human partners, whose physical behavior, willingness to assist, and understanding of the robot's capabilities may change over time. This demands a tightly coupled communication loop that grants both agents the flexibility to propose, accept, or decline requests as they coordinate toward completing the task effectively. We propose MICoBot, a system that enables the human and robot, both using natural language, to take initiative in formulating, accepting, or rejecting proposals on who can best complete different steps of a task. To handle diverse, task-directed dialog, and find successful collaborative strategies that minimize human effort, MICoBot makes decisions at three levels: (1) a meta-planner considers human dialog to formulate and code a high-level collaboration strategy, (2) a planner optimally allocates the remaining steps to either agent based on the robot's capabilities (measured by a simulation-pretrained affordance model) and the estimated human's willingness to help, and (3) an action executor decides the low-level actions to perform or words to say to the human. In physical robot trials with 18 unique human participants, MICoBot significantly improves task success and user experience over a pure LLM baseline and standard agent allocation models. See additional videos and materials at our project site: https://robin-lab.cs.utexas.edu/MicoBot/.
ML ID: 441
Entity tracking is a fundamental challenge in natural language understanding, requiring models to maintain coherent representations of entities. Previous work has benchmarked entity tracking performance in purely text-based tasks. We introduce MET-Bench, a multimodal entity tracking benchmark designed to evaluate the ability of vision-language models to track entity states across modalities. Using two structured domains, Chess and the Shell Game, we assess how effectively current models integrate textual and image-based state updates. Our findings reveal a significant performance gap between text-based and image-based tracking and that this performance gap stems from deficits in visual reasoning rather than perception. We further show that explicit text-based reasoning strategies improve performance, yet substantial limitations remain, especially in long-horizon multimodal scenarios. Our results highlight the need for improved multimodal representations and reasoning techniques to bridge the gap between textual and visual entity tracking.
ML ID: 437
