Research Interests

I am a Ph.D. candidate in the Department of Computer Science at the University of Texas at Austin studying artificial intelligence, and particularly reinforcement learning. My advisor is Peter Stone and I am a member of the Learning Agents Research Group at UT. I am also a part of the UT Austin Villa robot soccer team, which won the 2009 and 2010 US Opens in the Standard Platform League. I am researching reinforcement learning and robotics, specifically looking to make reinforcement learning apply to very large domains and applying it to make curious robots.

Before coming to UT, I worked in the Motion Analysis Laboratory at Spaulding Rehabilitation Hospital in Boston. There I worked on methods to evaluate the mobility of stroke, arthritis, and Parkinson's patients using wearable sensors. We used machine learning techniques to analyze wearable sensor data and quantify the quality of the patients' movements. I've also worked on a number of projects on my own, including building my own robot from scratch, writing a program to predict the scores of NFL football games based on machine learning techniques, and writing a 3D Connect Four game.

Robot Soccer

I have been a member of the UT Austin Villa robot soccer team since 2006 and participated in the Legged League and Standard Platform League at RoboCup. In 2009, we won the US Open Championship and came in 4th place in the international competition. In 2010, we repeated as US Open champions, and placed 3rd in the international competition. My research focus on robot soccer started with localization (See ICRA paper), but has moved on to encompass all parts of the robot soccer code, including, vision, locomotion, behaviors, coordination, and debug (See team paper).

Reinforcement Learning

Reinforcement Learning is a learning method where an agent can learn to act optimally by interacting with its environment. The agent is in some state and chooses from a set of available actions. Its action leads it to a new state and gives it some reward, which it tries to optimize over time. I am specifically focused on model-based reinforcement learning, where an agent learns a model of its environment and can learn a policy by simulating actions in its model. My research attempts to extend these model-based approaches to larger domains by incorporating generalization into the learning of the model (See ICDL paper, ICRA paper, and video). In addition, I am examining the problem of exploration versus exploitation, looking at when the agent should exploit what it thinks it knows compared to when it should explore more of the environment.

Links:

Personal Home Page
Curriculum Vitae
University of Texas Dept of Computer Science

Contact Info:

Office: ENS32NE (Robotics Lab)
E-mail: todd AT cs DOT utexas DOT edu

Teaching

In the Fall 2009 semester, I was the TA for CS393R: Autonomous Robotics. I won the department's Outstanding TA Award.

In Spring 2009, I was a TA for CS307 Foundations of Computing..

Open Source Code

I have a released a package (rl-texplore-ros-pkg) of reinforcement learning code for ROS. It contains a set of RL agents and environments, as well as a formalism for them to communicate through the use of ROS messages. In particular, the set of RL agents includes an implementation of our TEXPLORE agent (See our ICDL paper) and our real-time architecture for model-based agents (See this paper). A common interface is defined for agents, environments, models, and planners. Therefore, it should be easy to add new agents, or add new model learning or planning methods to the existing general model based agent. The real-time architecture should work with any model learning method that fits the defined interface. In addition, since the RL agents communicate using ROS messages, it is easy to integrate them with robots using an existing ROS architecture to perform reinforcement learning on robots.

• Code Repository
• Documentation
• Tutorial

Publications

Journal Articles

• S. Patel, R. Hughes, T. Hester, J. Stein, M. Akay, J. Dy, and P. Bonato. A Novel Approach to Monitor Rehabilitation Outcomes in Individuals Post Stroke Using Wearable Technology. In Proceedings of the IEEE, March 2010.

• S. Patel, T. Hester, R. Hughes, N. Huggins, A. Flaherty, D. Standaert, J. Growdon, and P. Bonato. Processing Wearable Sensor Data to Optimize Deep-Brain Stimulation. In IEEE Pervasive Computing, Jan 2008.

Book Chapters

• T. Hester and P.Stone. Learning and Using Models. In Reinforcement Learning: State of the Art, M. Wiering, M. van Otterlo, Editors, Springer 2011. (PDF)

• P. Stone, M. Quinlan, and T. Hester. The Essence of Soccer: Can Robots Play Too? In Soccer and Philosophy, Open Court Publishing, 2010. (PDF)

Refereed Conferences

• T. Hester, M. Quinlan, and P. Stone. RTMBA: A Real-Time Model-Based Reinforcement Learning Architecture for Robot Control. In IEEE International Conference on Robotics and Automation (ICRA), May, 2012. (PDF)

• T. Hester and P. Stone. Real Time Targeted Exploration in Large Domains. In IEEE International Conference on Development and Learning (ICDL), August, 2010. (PDF)

• T. Hester, M. Quinlan, and P. Stone. Generalized Model Learning for Reinforcement Learning on a Humanoid Robot. In IEEE International Conference on Robotics and Automation (ICRA), May 2010. (PDF) (Video)

• S. Kalyanakrishnan, T. Hester, M. Quinlan, Y. Bentor, and P. Stone. Three Humanoid Soccer Platforms: Comparison and Synthesis In Proceedings of the RoboCup International Symposium, July 2009. (PDF)

• T. Hester and P. Stone. Generalized Model Learning for Reinforcement Learning in Factored Domains In The Eighth International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS), May 2009. (PDF)

• T. Hester and P. Stone. Negative Information and Line Observations for Monte Carlo Localization. In IEEE International Conference on Robotics and Automation (ICRA), May 2008. (PDF)

• N. Jong, T. Hester, and P. Stone. The Utility of Temporal Abstraction in Reinforcement Learning. In The Seventh International Joint Conference on Autonomous Agents and Multiagent Systems (AAMAS), May 2008. (PDF)

• P. Boissy, T. Hester, D. M. Sherrill, H. Corriveau, and P. Bonato. Monitoring Mobility Assistive Device Use in Post-Stroke Patients. In Proceedings of the 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBS), August 2007.

• S. Patel, T. Hester, R. Hughes, N. Huggins, D. Standaert, A. Flaherty, and P.Bonato. Using Wearable Sensors to Enhance DBS Parameter Adjustment for Parkinson's Disease Patients Through Measures of Motor Response. In Proceedings of the 3rd IEEE EMBS International Summer School and Symposium on Medical Devices and Biosensors, September 2006.

• T. Hester, D. M. Sherrill, M. Hamel, K. Perreault, P. Boissy, and P. Bonato. Identification of Tasks Performed by Stroke Patients Using a Mobility Assistive Device. In Proceedings of the 28th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBS), Aug-Sept 2006.

• P. Boissy, T. Hester, D. M. Sherrill, H. Corriveau, and P. Bonato. Monitoring Mobility Assistive Device Use in Patients After Stroke. In Proceedings of the 16th Congress of the International Society of Electrophysiology and Kinesiology (ISEK), June-July 2006.

• T. Hester, R. Hughes, D. M. Sherrill, S. Patel, N. Huggins, A. Flaherty, D. Standaert, and P. Bonato. Adjusting DBS Settings to Optimize Parkinson’s Control Therapy. In Proceedings of the 16th Congress of the International Society of Electrophysiology and Kinesiology (ISEK), June-July 2006.

Refereed Workshop Papers

• T. Hester and P. Stone. Intrinsically Motivated Model Learning for a Developing Curious Agent In Proceedings of the AAMAS Workshop on Adaptive Learning Agents (ALA), June 2012.

• T. Hester and P. Stone. TEXPLORE: Real-Time Sample-Efficient Reinforcement Learning for Robots. In Proceedings of the AAAI Spring Symposium on Designing Intelligent Robots: Re-Integrating AI, March 2012.

• S. Barrett, K. Genter, T. Hester, M. Quinlan, and P. Stone. Controlled Kicking under Uncertainty. In Proceedings of the 5th Humanoids Workshop on Humanoid Soccer Robots, Dec 2010. (PDF)

• T. Hester and P. Stone. An Empirical Comparison of Abstraction in Models of Markov Decision Processes. In Proceedings of the ICML/UAI/COLT Workshop on Abstraction in Reinforcement Learning, June 2009. (PDF)

• T. Hester, D. M. Sherrill, M. Hamel, K. Perreault, P. Boissy, and P. Bonato. Using Wearable Sensors to Analyze the Quality of Use of Mobility Assistive Devices. In Proceedings of the Third Annual International Workshop on Wearable and Implantable Body Sensor Networks (BSN), April 2006.

• T. Hester, R. Hughes, D. M. Sherrill, B. Knorr, M. Akay, J. Stein, and P. Bonato. Using Wearable Sensors to Measure Motor Abilities following Stroke. In Proceedings of the Third Annual International Workshop on Wearable and Implantable Body Sensor Networks (BSN), April 2006.

• S. Patel, D. Sherrill, R. Hughes, T. Hester, N. Huggins, T. Lie-Nemeth, D. Standaert, and P. Bonato. Analysis of the Severity of Dyskinesia in Patients with Parkinson’s Disease via Wearable Sensors. In Proceedings of the Third Annual International Workshop on Wearable and Implantable Body Sensor Networks (BSN), April 2006.

Technical Reports

• S. Barrett, K. Genter, T. Hester, P. Khandelwal, M. Quinlan, P. Stone, and M. Sridharan. Austin Villa 2011: Sharing is Caring: Better Awareness through Information Sharing. Technical Report UT-AI-TR-12-01, The University of Texas at Austin, Department of Computer Science, AI Laboratory, 2012. (PDF)

• T. Hester, M. Quinlan, and P. Stone. A Real-Time Model-Based Reinforcement Learning Architecture for Robot Control. arXiv e-Prints 1105.1749, arXiv.org, 2011. (PDF)

• S. Barrett, K. Genter, M. Hausknecht, T. Hester, P. Khandelwal, J. Lee, M. Quinlan, A. Tian, P. Stone, and M. Sridharan. Austin Villa 2010 Standard Platform Team Report. Technical Report UT-AI-TR-11-01, The University of Texas at Austin, Department of Computer Science, AI Laboratory, 2011. (PDF)

• T. Hester, M. Quinlan, P. Stone, and M. Sridharan. UT Austin Villa 2009: Naos Across Texas. Technical Report UT-AI-TR-09-08, The University of Texas at Austin, Department of Computer Science, AI Laboratory, 2009. (PDF)

• T. Hester, M. Quinlan, and P. Stone. UT Austin Villa 2008: Standing on Two Legs. Technical Report UT-AI-TR-08-8, The University of Texas at Austin, Department of Computer Science, AI Laboratory, 2008. (PDF)

Invited Talks

• A Real-Time Model-Based Reinforcement Learning Architecture for Robot Control. At The 6th Barbados Workshop on Reinforcement Learning, March 2011.

• Generalized Model Learning for Reinforcement Learning in Factored Domains. At The 4th Barbados Workshop on Reinforcement Learning, April 2009.

Organizing Committees

• 2012 AAAI Spring Symposium on Designing Intelligent Robots: Re-Integrating AI

Program Committees

• 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL)
• 2012 International Conference on Machine Learning (ICML)
• 2012 European Workshop on Reinforcement Learning (EWRL)
• 2012 International Conference on Cognitive Robotics (CogRob)
• 2012 RoboCup Symposium
• AAMAS 2012 Workshop on Adaptive and Learning Agents (ALA)
• 2012 AAAI Spring Symposium on Designing Intelligent Robots: Re-Integrating AI
• 2011 RoboCup Symposium
• 2011 International Joint Conference on Artificial Intelligence (IJCAI)
• AAMAS 2010 Workshop on Agents in Real-time and Dynamic Environments

Technical Committees

• RoboCup Standard Platform League 2010
• RoboCup Standard Platform League 2009

Videos

2010 RoboCup Highlights

Highlights of TT-UT Austin Villa at the 2010 RoboCup Standard Platform League competition in Singapore, where the team took 3rd place.

Learning to Score Penalty Kicks via Reinforcement Learning

The accompanying video for our ICRA 2010 paper, where we learn to score penalty kicks via a novel model-based reinforcement learning method.

2009 RoboCup Highlights

Highlights of TT-UT Austin Villa at the 2009 RoboCup Standard Platform League. TT-UT Austin Villa finished in 4th place, losing to only two teams during the tournament.

2009 US Open Highlights

Highlights of TT-UT Austin Villa at the 2009 US Open. TT-UT Austin Villa won the 2009 US Open with a finals win over UPenn (1-1 tie, 3-2 in penalty kicks).

Aibo Highlights

This video shows highlights (both shots and saves) from demonstrations held during Explore UT on March 7, 2009.