The multi-year project will develop a new approach that helps household robots learn new skills and adapt to changing environments without forgetting what they already know.
Robots excel in a factory where every movement is programmed in an unchanging environment. But put those same robots in a typical home and they stumble. Homes are messy and unstructured: furniture moves, new items appear every day and the tasks required to navigate a dynamic environment change constantly.
Roberto Martín-Martín, an assistant professor in the Department of Computer Science, has embarked on a groundbreaking initiative that was recently recognized with an NSF CAREER award. The multi-year project titled Dual-Axis Continual Learning for Adaptive Mobile Manipulation Skills in Household Environments tackles one of the toughest bottlenecks in robotics: enabling machines to continuously acquire new skills and adapt to fresh surroundings without wiping their existing memory or starting training from scratch.
For a robot to be genuinely useful around the house, it needs mobile manipulation, the ability to navigate spaces while interacting with items using arms or grippers. Combining these two abilities in dynamic spaces introduces distinct challenges. For instance, crossing a room to pick up an object requires fine-grained coordination between mobility and dexterity, making manual programming for every possible scenario impossible.
Martin-Martin and his team at the Robot Interaction and Intelligence (RobIn) Lab at UT Austin are addressing these challenges with a novel framework called Dual-Axis Continual Learning. Rather than treating learning as a single linear process, the framework structures adaptation across two complementary dimensions:
- Task-Axis Continual Learning: Enables the robot to expand its repertoire of skills over time (e.g., learning to fold laundry, unload a dishwasher or open doors) without forgetting previously acquired capabilities.
- Environment-Axis Continual Learning: Allows the robot to take an existing skill and adapt it to new conditions, unfamiliar objects or a change in lighting without losing performance in its original environment.
With the new framework, the robot builds a cumulative memory. If it encounters an unfamiliar style of door handle in a bedroom, it adapts its existing motor skills to address the change, rather than having to re-learn how to grasp the new form from scratch.
The Future of Household Assistants
By replacing rigid linear programming with lifelong learning, Martin-Martin’s research marks a critical step toward general-purpose household robots. Instead of requiring pre-programmed instructions or constant updates that reset local knowledge, future robots equipped with dual-axis learning will seamlessly integrate into households. They will learn unique preferences, master unique layouts, and grow continuously more capable alongside the families they assist.
Roberto Martin-Martin is an Assistant Professor of Computer Science at UT Austin. His research bridges robotics, computer vision, and machine learning, focusing on enabling robots to operate autonomously in human-centric, unstructured environments such as homes and offices. Learn more about his research at the RobIn Lab at UT Austin.