Decision-Theoretic Planning Under Risk-Senstive Planning Objectives

Yaxin Liu

Risk attitudes are important for human decision making, especially in
scenarios where huge wins or losses are possible, as exemplified by
planetary rover navigation, oilspill response, and business
applications.  Decision-theoretic planners therefore need to take risk
aspects into account to serve their users better.  However, most
existing decision-theoretic planners use simplistic planning
objectives that are risk-neutral.  The thesis research is the first
comprehensive study of how to incorporate risk attitudes into
decision-theoretic planners and solve large-scale planning problems
represented as Markov decision process models.  The thesis consists of
three parts.

The first part of the thesis work studies risk-sensitive planning in
case where exponential utility functions are used to model risk
attitudes. In this case, there exists an optimal plan that maps states
to action.  I show that existing decision-theoretic planners can be
transformed to take risk attitudes into account, which is more general
than previous approaches that transform the planning tasks rather than
the planning algorithms. The transformed algorithms bear visual
resemblance to the original algorithms but special treatments may be
needed to ensure their validity.  Moreover, different versions of the
transformation are needed if the transition probabilities are
implicitly given, namely, temporally extended probabilities and
probabilities given in a factored form.  I show how the transformation
and its variants can be applied to various decision-theoretic
planners.

The second part of the thesis work studies risk-sensitive planning in
case where general nonlinear utility functions are used to model risk
attitudes.  In this case, there does not exist an optimal plan that
maps states to actions in general, and an optimal plan must take into
consideration the accumulated rewards starting from the initial state.
I show that a state-augmentation approach can be used to reduce a
risk-sensitive planning problem to a risk-neutral planning problem
with an augmented state space.  I further use a functional
interpretation of value functions and approximation methods to solve
the planning problems efficiently with value iteration.  I also show
an exact method for solving risk-sensitive planning problems where
one-switch utility functions are used to model risk attitudes.

The third part of the thesis work studies risk sensitive planning in
case where arbitrary rewards are used.  In this case, many of the
basic properties are unknown, including the existence and finiteness
of optimal expected utilities.  I propose a spectrum of conditions
that can be used to constrain the utility function and the planning
problem so that the optimal expected utilities exist and are finite.
These conditions are the basis for the further development of
computational procedures.  I prove that the existence and finiteness
properties hold for stationary plans, where the action to perform in
each state does not change over time, under different sets of
conditions.

I use two running examples to demonstrate that risk-sensitive planners
can be easily created from their risk-neutral counterparts, and the
resulting optimal plans are qualitatively different from optimal plans
under a risk-neutral planning objective.