Laurent Orseau, Mark Ring. Self-Modification and Mortality in Artificial
Agents. AGI, 2011.
Theoretical models of artificial general intelligence, such as
AIXI , typically consider an intelligent agent to have unlimited computational
resources, allowing it to keep a perfect memory of its entire
interaction history with its environment. In the real world, an agent's
memory is part of the environment, which means that the latter can
modify the former. This paper develops a theoretical framework for examining
the implications of such real-world memory on universal intelligent
agents. Within this framework we are able to show, for example,
that in certain environments optimality can be achieved only with truly
stochastic behaviors, and that guarantees about the trustworthiness of
memories are difficult to obtain even with infinite computational power.
To describe the probability of an agent's memory state, we propose an
adaptation of the universal prior for the passive and the active case.