Layered Learning in Multiagent Systems:
A Winning Approach to Robotic Soccer

by Peter Stone

MIT Press, 2000.
ISBN: 0262194384

Available from MIT Press and

This book is based upon my Ph.D. thesis
(Computer Science Department, Carnegie Mellon University, 1998)


This book looks at multiagent systems that consist of teams of autonomous agents acting in real-time, noisy, collaborative, and adversarial environments. The book makes four main contributions to the fields of machine learning and multiagent systems.

First, it describes an architecture within which a flexible team structure allows member agents to decompose a task into flexible roles and to switch roles while acting. Second, it presents layered learning, a general-purpose machine-learning method for complex domains in which learning a mapping directly from agents' sensors to their actuators is intractable with existing machine-learning methods. Third, the book introduces a new multiagent reinforcement learning algorithm--team-partitioned, opaque-transition reinforcement learning (TPOT-RL)--designed for domains in which agents cannot necessarily observe the state-changes caused by other agents' actions. The final contribution is a fully functioning multiagent system that incorporates learning in a real-time, noisy domain with teammates and adversaries--a computer-simulated robotic soccer team.

Peter Stone's work is the basis for the CMUnited Robotic Soccer Team, which has dominated recent RoboCup competitions. RoboCup not only helps roboticists to prove their theories in a realistic situation, but has drawn considerable public and professional attention to the field of intelligent robotics. The CMUnited team won the 1999 Stockholm simulator competition, outscoring its opponents by the rather impressive cumulative score of 110-0.


Chapter 1: Introduction 
            1.1  Motivation
            1.2  Objectives and Approach
            1.3  Contributions
            1.4  Reader's Guide to the Book

Chapter 2: Substrate Systems 
            2.1  Overview
            2.2  The RoboCup Soccer Server
            2.3  The CMUnited-97 Real Robots
            2.4  Network Routing

Chapter 3: Team Member Agent Architecture 
            3.1  Periodic Team Synchronization (PTS) Domains
            3.2  Architecture Overview
            3.3  Teamwork Structure
            3.4  Communication Paradigm
            3.5  Implementation in Robotic Soccer
            3.6  Results
            3.7  Transfer to Real Robots
            3.8  Discussion and Related Work

Chapter 4: Layered Learning 
            4.1  Principles
            4.2  Formalism
            4.3  Instantiation in Simulated Robotic Soccer
            4.4  Discussion
            4.5  Related Work

Chapter 5: Learning an Individual Skill 
            5.1  Ball Interception in the Soccer Server
            5.2  Training
            5.3  Results
            5.4  Discussion
            5.5  Related Work

Chapter 6: Learning a Multi-Agent Behavior 
            6.1  Decision Tree Learning for Pass Evaluation
            6.2  Using the Learned Behaviors
            6.3  Scaling up to Full Games
            6.4  Discussion
            6.5  Related Work

Chapter 7: Learning a Team Behavior 
            7.1  Motivation
            7.2  TPOT-RL
            7.3  TPOT-RL Applied to Simulated Robotic Soccer
            7.4  TPOT-RL Applied to Network Routing
            7.5  Discussion
            7.6  Related Work

Chapter 8: Competition Results 
            8.1  Pre-RoboCup-96
            8.2  MiroSot-96
            8.3  RoboCup-97
            8.4  RoboCup-98
            8.4  RoboCup-99
            8.6  Lessons Learned from Competitions

Chapter 9: Related Work 
            9.1  MAS from an ML Perspective
            9.2  Robotic Soccer

Chapter 10: Conclusion 
            10.1  Contributions
            10.2  Future Directions
            10.3  Concluding Remarks

            A  List of Acronyms
            B  Robotic Soccer Agent Skills
                        B.1  CMUnited-98 Simulator Agent Skills
                        B.2  CMUnited-97 Small-Robot Skills
            C  CMUnited-98 Simulator Team Behavior Modes
                        C.1  Conditions
                        C.2  Effects
            D  CMUnited Simulator Team Source Code

On-line Appendix

This is the on-line appendix of the book (Appendix D).
It contains source code and executables of the CMUnited-97 and CMUnited-98 simulator teams.
Details regarding the contents of the files are available on the respective team pages ( CMUnited-97 and CMUnited-98).

  • CMUnited-97 source code.
  • CMUnited-97 executable for SunOS.
  • CMUnited-98 source code.
  • CMUnited-98 executable for Linux and SunOS.
  • Information, source, and executables pertaining to the CMUnited-99 simulator team are also available from the CMUnited-99 team page.

    Peter Stone
    Go back to my homepage
    Send me mail