Soccer is a rich domain for the study of multiagent learning issues. Teams of players must work together in order to put the ball in the opposing goal while at the same time defending their own. Learning is essential in this task since the dynamics of the system can change as the opponents' behaviors change. The players must be able to adapt to new situations.
Not only must the players learn to adapt to the behaviors of different opponents, but they must learn to work together. Soon after beginning to play, young soccer players learn that they cannot do everything on their own: they must work as a team in order to win.
However, before players can learn any collaborative or adversarial techniques, they must first acquire some low-level skills that allow them to manipulate the ball. Although some low-level skills, such as dribbling, are entirely individual in nature, others, such as passing and receiving passes, are necessitated by the multiagent nature of the domain.
Our approach to multiagent learning in this complex domain is to develop increasingly complex layers of learned behaviors from the bottom up. Beginning with individual skills that are appropriate in a multiagent environment, we identify appropriate behavior parameters and learning methods for these parameters. Next, we incorporate these learned individual skills into higher-level multiagent learning scenarios. Our ongoing goal is to create a full team of agents that use learned behaviors at several different levels to reason strategically in a real-time environment.
In this article, we present detailed experimental results of our successful use of neural networks for learning a low-level behavior. This learned behavior, namely shooting a moving ball, is crucial to successful action in the multiagent domain. It also equips our clients with the skill necessary to learn higher-level collaborative and adversarial behaviors. We illustrate how the learned individual skill can be used as a basis for higher level multiagent learning, discussing the issues that arise as we extend the learning task.