Learning Strategic Behavior in Sequential Decision Tasks
Active from 2009 - 2014
Many routine tasks in the real world can be seen as sequential decision tasks. For instance, navigating a robot through a complex environment, driving a car in congested traffic, and routing packets in a computer network requires making a sequence of decisions that together minimize time and resources used. It would be desirable to automate these tasks, yet it is difficult because the optimal decisions are generally not known. Approximating them by finite-state machines or learning them based on reinforcement leads to reactive behaviors that perform well in short term, but do not amount to intelligent high-level behavior in the long term. The goal of this project is to develop the technology that makes learning such strategic high-level behavior possible.

The main technical challenge is to devise a method that extends sequential decision learning from reactive to strategic behaviors. Such a method needs to be able to (1) retain information from past states, (2) learn multimodal behavior, (3) choose between the different behaviors based on crucial detail, and (4) implement a sequential high-level strategy based on those behaviors. The neuroevolution methods developed in prior work solve the first problem by evolving (through genetic algorithms) recurrent neural networks to represent the behavior. To solve the remaining problems, these methods will be extended with multi-objective optimization, local nodes with cascaded structure, and with evolution of modules and their combinations. Preliminary results indicate that this approach is indeed feasible. In this project, it will be first characterized fully in supervised learning tasks as well as in synthetic sequential decision tasks. It will then be scaled up to a robotic soccer simulation in OpenNERO, and evaluated in two ways: In an objective comparison with other hand-coded and learned soccer teams, and through a subjective analysis (by human evaluators) of the learned strategies. The end result will be a systematic approach to learning strategic high-level behavior in sequential decision tasks.

In the long term, the technology should make it possible to build robust sequential decision systems for real-world tasks. It should lead to safer and more efficient vehicle, traffic, and robot control, improved process and manufacturing optimization, and more efficient computer and communication systems. It should also make the next generation of video games possible, with characters that exhibit realistic, strategic behaviors: Such technology should lead to more effective educational and training games in the future.

This research is supported by the National Science Foundation under grant IIS-0915038.

     [Expand to show all 22][Minimize]
Risto Miikkulainen Faculty risto [at] cs utexas edu
Jacob Schrum Ph.D. Alumni schrum2 [at] southwestern edu
Nate Kohl Ph.D. Alumni nate [at] natekohl net
Vinod Valsalam Ph.D. Alumni vkv [at] alumni utexas net
Chern Han Yong Masters Alumni cherny [at] nus edu sg
Padmini Rajagopalan Ph.D. Student padmini [at] cs utexas edu
Aditya Rawal Ph.D. Student aditya [at] cs utexas edu
Bryan Silverthorn Ph.D. Alumni bsilvert [at] cs utexas edu
Alan J. Lockett Ph.D. Alumni alan lockett [at] gmail com
Igor V. Karpov Ph.D. Student ikarpov [at] gmail com
Adam C. Dziuk Undergraduate Alumni
Chris Bush Undergraduate Alumni
Matthew Johnston Undergraduate Alumni
Timothy Nodine Undergraduate Alumni
David Robson Undergraduate Alumni
Eliana Feasley Formerly affiliated Ph.D. Student elie [at] cs utexas edu
Wesley Tansey Ph.D. Student tansey [at] cs utexas edu
Anand Subramoney Masters Alumni anands [at] cs utexas edu
Dan Lessin Ph.D. Student dlessin [at] cs utexas edu
Joel Lehman Postdoctoral Fellow joel [at] cs utexas edu
Julian Bishop Ph.D. Student julian [at] cs utexas edu
Matthew de Wet Undergraduate Alumni
     [Expand to show all 50][Minimize]
A Measure-Theoretic Analysis of Stochastic Optimization 2013
Alan J. Lockett and Risto Miikkulainen, In Proceedings of the 12th International Workshop on Foundations of Genetic Algorithms (FOGA-2013) 2013. ACM Press.
A Neuroevolution Approach to General Atari Game Playing 2013
Matthew Hausknecht, Joel Lehman, Risto Miikkulainen, and Peter Stone, In IEEE Transactions on Computational Intelligence and AI in Games 2013.
Boosting Interactive Evolution using Human Computation Markets 2013
Joel Lehman and Risto Miikkulainen, To Appear In Proceedings of the 2nd International Conference on the Theory and Practice of Natural Computation, pp. 18 pages 2013. Springer.
Effective Diversity Maintenance in Deceptive Domains 2013
Joel Lehman, Kenneth O. Stanley and Risto Miikkulainen, To Appear In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO) 2013 2013.
Evolutionary Annealing: Global Optimization in Arbitrary Measure Spaces 2013
Alan J Lockett and Risto Miikkulainen, Journal of Global Optimization (2013), pp. 1--34.
Measure-Theoretic Analysis of Performance in Evolutionary Algorithms 2013
Alan J Lockett, In Proceedings of the 2013 IEEE Congress on Evolutionary Computation (CEC-2013) 2013. IEEE Press.
Neuroannealing: Martingale-Driven Optimization for Neural Networks 2013
Alan J Lockett and Risto Miikkulainen, In Proceedings of the 2013 Genetic and Evolutionary Computation Conference (GECCO-2013) 2013. ACM Press.
Neuroevolution 2013
Joel Lehman and Risto Miikkulainen, Scholarpedia, Vol. 8, 6 (2013), pp. 30977.
Open-Ended Behavioral Complexity for Evolved Virtual Creatures 2013
Dan Lessin, Don Fussell, Risto Miikkulainen, In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO) 2013 2013.
Using Symmetry and Evolutionary Search to Minimize Sorting Networks 2013
Vinod K. Valsalam and Risto Miikkulainen, Journal of Machine Learning Research, Vol. 14, Feb (2013), pp. 303--331.
Accelerating Evolution via Egalitarian Social Learning 2012
Wesley Tansey, Eliana Feasley, and Risto Miikkulainen, Proceedings of the 14th Annual Genetic and Evolutionary Computation Conference (GECCO 2012) (2012).
Architecture of a Cyberphysical Avatar 2012
Song Han, Aloysius K. Mok, Jianyong Meng, Yi-Hung Wei, Pei-Chi Huang, Xiuming Zhu, Luis Sentis, Kan Suk Kim, Risto Miikkulainen, and Jacob Menashe, In Proceedings of the International Workshop on Real-Time and Distributed Computing in Emerging Applications (REACTION) 2012.
Believable Bot Navigation via Playback of Human Traces 2012
Igor V. Karpov, Jacob Schrum, Risto Miikkulainen, In Believable Bots, Philip F. Hingston (Eds.), pp. 151--170 2012. Springer Berlin Heidelberg.
Constructing Controllers for Physical Multilegged Robots using the ENSO Neuroevolution Approach 2012
Vinod K. Valsalam, Jonathan Hiller, Robert MacCurdy, Hod Lipson and Risto Miikkulainen, Evolutionary Intelligence, Vol. 5, 1 (2012), pp. 1--12.
Evaluating Modular Neuroevolution in Robotic Keepaway Soccer 2012
Anand Subramoney, Masters Thesis, Department of Computer Science, The University of Texas at Austin. 54 pages.
Evaluation Methods for Active Human-Guided Neuroevolution in Games 2012
Igor Karpov, Leif Johnson, Vinod Valsalam and Risto Miikkulainen, In 2012 AAAI Fall Symposium on Robots Learning Interactively from Human Teachers (RLIHT), November 2012.
Evolution of a Communication Code in Cooperative Tasks 2012
Aditya Rawal, Padmini Rajagopalan, Risto Miikkulainen and Kay Holekamp, In Artificial Life (13th International Conference on the Synthesis and Simulation of Living Systems), East Lansing, Michigan, USA 2012.
Evolving Multimodal Networks for Multitask Games 2012
Jacob Schrum and Risto Miikkulainen, IEEE Transactions on Computational Intelligence and AI in Games, Vol. 4, 2 (2012), pp. 94--111. IEEE.
General-Purpose Optimization Through Information-Maximization 2012
Alan J Lockett, PhD Thesis, Department of Computer Sciences, The University of Texas at Austin. Tech Report AI12-11.
Humanlike Combat Behavior via Multiobjective Neuroevolution 2012
Jacob Schrum, Igor V. Karpov and Risto Miikkulainen, In Believable Bots, Philip F. Hingston (Eds.), pp. 119--150 2012. Springer Berlin Heidelberg.
HyperNEAT-GGP: A HyperNEAT-based Atari General Game Player 2012
Matthew Hausknecht, Piyush Khandelwal, Risto Miikkulainen, Peter Stone, In Genetic and Evolutionary Computation Conference (GECCO) 2012 2012.
Multiagent Learning through Neuroevolution 2012
Risto Miikkulainen, Eliana Feasley, Leif Johnson, Igor Karpov, Padmini Rajagopalan, Aditya Rawal, and Wesley Tansey, In Advances in Computational Intelligence, J. Liu et al. (Eds.), Vol. LNCS 7311, pp. 24-46, Berlin, Heidelberg: 2012. Springer.
Task decomposition with neuroevolution in extended predator-prey domain 2012
Ashish Jain, Anand Subramoney, Risto Miikkulainen, In Proceedings of Thirteenth International Conference on the Synthesis and Simulation of Living Systems, East Lansing, MI, USA 2012.
An Integrated Neuroevolutionary Approach to Reactive Control and High-level Strategy 2011
Nate Kohl, Risto Miikkulainen, IEEE Transactions on Evolutionary Computation (2011).
Avoiding Premature Convergence in NeuroEvolution by Broadening the Evolutionary Search 2011
Matthew de Wet, Technical Report HR-11-02, Department of Computer Science, The University of Texas at Austin.
Creating Intelligent Agents through Shaping of Coevolution 2011
Adam Dziuk, Technical Report HR-11-01, Department of Computer Science, The University of Texas at Austin.
Creating Intelligent Agents through Shaping of Coevolution 2011
Adam Dziuk and Risto Miikkulainen, In Proceedings of the Congress on Evolutionary Computation, New Orleans, LA 2011. IEEE.
Evolving Multimodal Networks for Multitask Games 2011
Jacob Schrum and Risto Miikkulainen, In Proceedings of the IEEE Conference on Computational Intelligence and Games (CIG 2011), pp. 102--109, Seoul, South Korea, September 2011. IEEE. (Best Paper Award).
Evolving Symmetry for Modular System Design 2011
Vinod K. Valsalam and Risto Miikkulainen, IEEE Transactions on Evolutionary Computation, Vol. 15, 3 (2011), pp. 368--386.
Human-Assisted Neuroevolution Through Shaping, Advice and Examples 2011
Igor V. Karpov, Vinod K. Valsalam and Risto Miikkulainen, In Proceedings of the 13th Annual Genetic and Evolutionary Computation Conference (GECCO 2011), Dublin, Ireland, July 2011.
Measure-Theoretic Evolutionary Annealing 2011
Alan J. Lockett and Risto Miikkulainen, In Proceedings of the 2011 IEEE Congress on Evolutionary Computation 2011.
Real-Space Evolutionary Annealing 2011
Alan J Lockett and Risto Miikkulainen, In Proceedings of the 2011 Genetic and Evolutionary Computation Conference (GECCO-2011) 2011.
The Role of Reward Structure, Coordination Mechanism and Net Return in the Evolution of Cooperation 2011
Padmini Rajagopalan, Aditya Rawal, Risto Miikkulainen, Marc A. Wiseman and Kay E. Holekamp, In Proceedings of the IEEE Conference on Computational Intelligence and Games (CIG 2011), Seoul, South Korea 2011.
Utilizing Symmetry and Evolutionary Search to Minimize Sorting Networks 2011
Vinod K. Valsalam and Risto Miikkulainen, Technical Report AITR-11-09, Department of Computer Sciences, The University of Texas at Austin.
UT^2: Human-like Behavior via Neuroevolution of Combat Behavior and Replay of Human Traces 2011
Jacob Schrum, Igor V. Karpov and Risto Miikkulainen, In Proceedings of the IEEE Conference on Computational Intelligence and Games (CIG 2011), pp. 329--336, Seoul, South Korea, September 2011. IEEE.
An Analysis of Automated Decision Making Methodologies in Role Playing Video Games: Centralized Approach 2010
Christopher Bush, Technical Report HR-10-03, Department of Computer Science, The University of Texas at Austin.
An Analysis of Distributed Decision Making Methodologies in Role Playing Video Games 2010
Matthew Johnston, Technical Report HR-10-09, Department of Computer Science, The University of Texas at Austin.
Coevolution of Role-Based Cooperation in Multi-Agent Systems 2010
Chern Han Yong and Risto Miikkulainen, IEEE Transactions on Autonomous Mental Development, Vol. 1 (2010), pp. 170--186.
Constructing Competitive and Cooperative Agent Behavior Using Coevolution 2010
Aditya Rawal, Padmini Rajagopalan and Risto Miikkulainen, In IEEE Conference on Computational Intelligence and Games (CIG 2010), Copenhagen, Denmark, August 2010.
Evolving Agent Behavior In Multiobjective Domains Using Fitness-Based Shaping 2010
Jacob Schrum and Risto Miikkulainen, In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2010), pp. 439--446, Portland, Oregon, July 2010.
Hierarchical Neural Networks for Behavior-Based Decision Making 2010
David Robson, Technical Report HR-10-02, Department of Computer Science, The University of Texas at Austin.
Latent Class Models for Algorithm Portfolio Methods 2010
Bryan Silverthorn and Risto Miikkulainen, In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence 2010.
Neuroevolution 2010
Risto Miikkulainen, In Encyclopedia of Machine Learning, New York 2010. Springer.
Speciation in NEAT 2010
Timothy Nodine, Technical Report HR-10-06, Department of Computer Science, The University of Texas at Austin.
Utilizing Symmetry in Evolutionary Design 2010
Vinod Valsalam, PhD Thesis, Department of Computer Sciences, The University of Texas at Austin. Technical Report AI-10-04.
Evolving Neural Networks for Strategic Decision-Making Problems 2009
Nate Kohl and Risto Miikkulainen, Neural Networks, Special issue on Goal-Directed Neural Systems (2009).
Evolving Symmetric and Modular Neural Network Controllers for Multilegged Robots 2009
Vinod K. Valsalam and Risto Miikkulainen, In xploring New Horizons in Evolutionary Design of Robots: Workshop at the 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2009.
Evolving Symmetric and Modular Neural Networks for Distributed Control 2009
Vinod K. Valsalam and Risto Miikkulainen, In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO), pp. 731--738 2009.
Learning in Fractured Problems for Constructive Neural Network Algorithms 2009
Nate Kohl, PhD Thesis, Department of Computer Sciences, University of Texas at Austin.
Temporal Convolution Machines for Sequence Learning 2009
Alan J Lockett and Risto Miikkulainen, Technical Report AI-09-04, Department of Computer Sciences, the University of Texas at Austin.
ESL This is the C# source code for the experiments with Egalitarian Social Learning (ESL) in a robot foraging domain. The re... 2012

UT^2: Winning Botprize 2012 Entry The Botprize Competition is an annual competition to program bots that appear human-l... 2012

PyEC Python package containing source code for Evolutionary Annealing along with a number of other evolutionary and stochasti... 2011

BREVE Monsters BREVE is a system for designing Artificial Life simulations available at http://spiderlan... 2010

ENSO This package contains software implementing the ENSO approach for evolving symmetric modular neural networks. It also in... 2010

NEAT C++ The NEAT package contains source code implementing the NeuroEvolution of Augmenting Topologies method. The source code i... 2010

OpenNERO OpenNERO is a general research and education platform for artificial intelligence. The platform is based on a simulatio... 2010

Sorting Networks This package contains software utilizing an approach based on symmetry and evolution to minimize the number of comparato... 2010

rtNEAT C++ The rtNEAT package contains source code implementing the real-time NeuroEvolution of Augmenting Topologies method. In ad... 2006

SANE-C The SANE-C package contains the source code for the Hierarchical SANE system, written in C. This package has been rewrit... 1997

     [Expand to show all 20][Minimize]
Evolving Cooperation in Multiagent SystemsChern Yong2007
Learning in Fractured DomainsNate Kohl2009
Multi-modal Behavior in NPCsJacob Schrum2009
Multi-objective Neuroevolution of NPCsJacob Schrum2008
Emergence of Competitive and Cooperative Behavior and Arms Race Through CoevolutionAditya Rawal, Padmini Rajagopalan2010
Evolving Controller Symmetry for Multilegged RobotsVinod Valsalam2010
Evolving Controllers for Physical Multilegged RobotsVinod Valsalam2011
Modular Neuroevolution for Multilegged LocomotionVinod Valsalam2008
A Neuroevolution Approach to General Atari Game PlayingMatthew Hausknecht2013
Egalitarian Social Learning (ESL) in Robot ForagingWesley Tansey2012
Evolution of a Communication Code in Cooperative TasksAditya Rawal, Padmini Rajagopalan, Risto Miikkulainen, Kay Holekamp2012
Fitness-based Shaping in Multi-objective DomainsJacob Schrum2010
Multi-modal Approaches to Evolving Behavior for Multi-task GamesJacob Schrum2011
Multimodal Behavior in Imprison Ms. Pac-ManJacob Schrum2014
Multimodal Behavior in One Life Ms. Pac-ManJacob Schrum2014
The Role of Reward Structure, Coordination Mechanism and Net Return in the Evolution of CooperationPadmini Rajagopalan, Aditya Rawal2011
UT^2: Winner of 2012 BotPrize in Unreal Tournament 2004Jacob Schrum, Igor Karpov2012
Multimodal Behavior in Multiple Lives Ms. Pac-ManJacob Schrum2014
Trading Control Intelligence for Physical Intelligence: Muscle Drives in Evolved Virtual CreaturesDan Lessin, Don Fussell, Risto Miikkulainen2014
Open-Ended Behavioral Complexity for Evolved Virtual CreaturesDan Lessin, Don Fussell, Risto Miikkulainen2013