Large multimodal models have become central for solving sequential decision-making tasks, enabling improved learning in diverse areas such as home robotics and automated software development. However, leveraging these models for sequential decision-making requires robust action reasoning capabilities, which remain a significant challenge. This dissertation improves and evaluates action reasoning in large multimodal models. First, we introduce a method to improve the parsing of instructional texts into action sequences by integrating external symbolic planners and planning domains during autoregressive language model decoding. Next, we develop a method that leverages the compositional structure of language instructions to improve generalization and sample efficiency of acquiring new tasks with reinforcement learning. We also construct a new benchmark to evaluate the understanding of dependencies between actions described in instructional texts. Finally, we evaluate the world modeling limitations of frontier models through multimodal entity state tracking. Current models struggle to reason about the effects of actions in multimodal entity state tracking tasks. We extend entity state tracking evaluations to a simulated embodied environment and derive insights for improving the entity-state reasoning abilities of language and vision-language models. Together these contributions enhance the understanding of how models reason about actions and provide insights toward their improvement for real-world sequential decision-making problems.

PDF

PhD Thesis, Department of Computer Science, UT Austin.

@phdthesis{cohen:phdthesis26, title={Reasoning about Actions with Large Multimodal Models}, author={Vanya Cohen}, month={August}, school={Department of Computer Science, UT Austin}, url="http://www.cs.utexas.edu/users/ai-labpub-view.php?PubID=128182", year={2026} }

Slides (PDF)