Extracting knowledge and reasoning from large language models (LLMs) offers a path to designing intelligent robots. Common approaches that leverage LLMs for planning are unable to recover when actions fail and resort to retrying failed actions without resolving the underlying cause. We propose a novel approach (CAPE) that generates corrective actions to resolve precondition errors during planning. CAPE improves the quality of generated plans through few-shot reasoning on action preconditions. Our approach enables embodied agents to execute more tasks than baseline methods while maintaining semantic correctness and minimizing re-prompting. In VirtualHome, CAPE improves a human-annotated plan correctness metric from 28.89 percent to 49.63 percent over SayCan, whilst achieving competitive executability. Our improvements transfer to a Boston Dynamics Spot robot initialized with a set of skills (specified in language) and associated preconditions, where CAPE improves correctness by 76.49 percent with higher executability compared to SayCan. Our approach enables embodied agents to follow natural language commands and robustly recover from failures.

PDF

International Conference on Robotics and Automation (ICRA) (2024).

@article{raman:icra24, title={CAPE: Corrective Actions from Precondition Errors using Large Language Models}, author={Shreyas Sundara Raman and Vanya Cohen and Ifrah Idrees and Eric Rosen and Raymond Mooney and Stefanie Tellex and David Paulius}, booktitle={International Conference on Robotics and Automation (ICRA)}, month={May}, url="http://www.cs.utexas.edu/users/ai-labpub-view.php?PubID=128049", year={2024} }