Robots are everywhere, from the Roomba cleaning your floor to the first self-driving cars traveling the roads. As robots advance and help with more and more tedious or dangerous tasks, they need an easy and efficient way to learn and adapt to their surroundings.

Computer science assistant professor and director of the Personal Autonomous Robotics Lab (PeARL) Scott Niekum looks for ways to simplify training these robotic helpers by showing them how to do their jobs through demonstration, bypassing much of the complicated coding and computer science expertise needed by robots today.

The goal is to have robots increase productivity and take over “the 3 D’s” of tasks: jobs that are dirty, dangerous or dull. For homes, that includes cleaning the house and collecting waste, and for workplaces that can range from inspecting sewer pipes and assembling products to working in a hazardous nuclear reactor.

“In homes, we’d like to see robots do everything from taking over monotonous tasks to helping the disabled and elderly gain more autonomy and lead fuller lives,” Niekum said. “Some of the benefits of AI in these situations are that it doesn’t get bored, it can’t get injured — and if it gets broken, it can be replaced or fixed — and it is better at some types of tasks than humans are.”

Right now, robots with artificial intelligence are coded and trained to do specific tasks, a technique that requires a lot of work by robotics experts to “teach” the robots to do their jobs. That’s a significant obstacle to widespread use, Niekum said, since robots that would be used in different homes and workplaces would need to do a wide variety of different tasks. That’s where Niekum’s research comes in: instead of having a robotics expert train a robot to do every single task, customized for every home or company, the people who use the robots would teach them by showing them what to do.

“Typically, a robotics expert such as me or one of my graduate students needs to spend weeks or months programming a robot to do a specific task in a specific environment,” Niekum said. “That model will never scale to widespread deployment, so instead, we try to build learning robots that can simply be shown what to do so that they are customizable by the user.”

However, there are also challenges to teaching this robots this way, Niekum said. In order to show a robot what to do, it must be able to watch a human’s body movements and translate that movement into something that it can accomplish with its own body, even if it has a completely different body structure from a human. In addition, the robot must be able to understand ambiguous movements and cues from humans.

“Humans are very good at judging the intentions and goals of other humans because we have an enormous amount of prior life experiences to draw upon,” Niekum said. “Robots don’t have access to this level of data yet, making it challenging to properly interpret human behaviors.”

Currently, researchers in PeARL are working on a project that uses YouTube videos to help robots learn how to cook. By finding context from additional sources of information on the internet, such as videos and recipes written in natural language instead of code, the robot can better understand any ambiguous signals from a human demonstrator.

PeARL is also investigating how to make robots better understand human intentions through interpreting eye gaze and natural language instructions.

“Gaze and natural language provide very rich information about a person’s goals, beliefs, and intentions,” Niekum said. “However, gaze is difficult to accurately track from video alone, and natural language is notoriously difficult to interpret correctly. Thus, we are researching ways make use of this data, even when our interpretation of it has significant uncertainty.”

In order to allow for their widespread use, however, another important part of Niekum’s research is making sure that these robots are safe and reliable. PeARL researchers are also investigating how to provide guarantees about the safety of a system despite learning new tasks.

“This is challenging, but can be done in a mathematically principled way, not so different in spirit from the tools engineers have developed over the years to make sure that bridges or buildings are safe,” Niekum said.

In addition to being safe, robots need to be able to detect errors through data sources like sound and vision, then correct those errors. For example, the robot needs to know what success and failure looks like, and it needs to be able to select the right viewpoints to see what went wrong.

“With a bit of human supervision, robots can learn the perceptual signals that accompany success and failure in a task — for example, the sight of a screw going in or missing a hole, or the click or absence of a click for a door closing properly,” Niekum said. “If the robot can detect these failures reliably and early, they can often be addressed by trying again or issuing a recovery behavior. Other times — for example, if a glass is broken — they must halt the task and/or alert a human.”

Right now, these research questions are barriers to making robots more commonplace in homes and workplaces, but Niekum and the PeARL researchers are working on these problems and helping make robots safe, reliable and easily customizable by average users.

“I believe that learning from demonstration is the only feasible way for end-users to program robots to perform the multitude of tasks that they desire,” Niekum said. “As mobile robots with manipulation capabilities become lower-cost over time, I suspect that we will see robots in homes and workplaces on a large scale that can be taught via demonstration, as well as learn from other modalities, including natural language conversations.”