Tutorial: Evolution of Neural Networks

Risto Miikkulainen
The University of Texas at Austin and Cognizant AI Labs

Description

Neuroevolution is a new and emerging area of reinforcement learning (RL). It is particularly useful in two areas:

Tasks that require memory: Whereas the traditional value-function-based approach most naturally focuses on MDP problems and on maximizing lifetime reward, neuroevolution work focuses mostly on POMDP tasks and on maximizing reward at the end of learning. Knowledge of neuroevolution should therefore be valuable for researchers and students in robotics, intelligent agents, and multiagent systems.
Deep learning: The performance of deep learning depends crucially on the network architecture and hyperparameters, and several techniques have been develop to optimize them. As a population-based search technique, neuroevolution can explore the search space widely, and therefore find innovative and surprising solutions that would be difficut to find with other techniques. Researchers and students working in learning in image processing, speech, language, and prediction and modeling of complex systems, should be able to use neuroevolution to improve their results.

In this tutorial, I will review (1) neuroevolution methods that evolve fixed-topology networks, network topologies, and network construction processes for POMDP tasks, (2) ways of combining gradient-based training with evolutionary methods to discover more powerful deep learning architectures, and (3) applications of these techniques in control, robotics, artificial life, games, image processing, and language.

Presentation Materials

Slides (in 4-up pdf from GECCO 2023), with references.
Video of the tutorial presentation (from GECCO 2023)
Demos (i.e. animations)

Neuroevolution Exercise

NERO is a video game where the player evolves neural network controllers for teams of non-player characters that engage in battle in a simulated environment. It will take some 30mins to get the idea, and upto a few hours of training to build complex teams. (NOTE: These instructions were last checked in 2021.)

NERO Software

You need to download two components:

(1) NERO, one of:
Mac version
Windows version (a possible problem and a fix)
32-bit Linux version (suggestions for 64-bit linux).

(2) tutorial.cs.dso
The same file works in all platforms. You need to save it in your NERO application folder, replacing the tutorial.cs.dso there. That folder is
in MacOS: NERO-2.0/nero/client/scripts/
in Windows: Nero\2.0\nero\client\scripts (a possible problem and a fix)
in Linux: nero2_linux_i386/nero/client/scripts/

Running the Exercise

Start NERO by double clicking its icon in the main folder, or from the command line. Then click on "SINGLE PLAYER", "SIMPLE TUTORIAL", and hit the spacebar (you don't need to read the story). You'll see NERO loading the sandbox environment. In that environment, there is a text window that tells you what to do. In case you weren't able to install the tutorial.ps.dso file, you can follow the text here.

Towards the end of the exercise (e.g. after step 27) you may want to test your team against teams that other people have created. NERO comes with a number of such teams, or you can play against a few teams that people attending this tutorial created; you can get such teams from here; download a team file, place it in nero/data/saves/brains subdirectory, and then select it as an opponent in the battle mode. If you want to contribute your team to this archive, please email it to risto@cs.utexas.edu.