Standard Article

Approximate Dynamic Programming I: Modeling

  1. Warren B. Powell

Published Online: 14 JAN 2011

DOI: 10.1002/9780470400531.eorms0042

Wiley Encyclopedia of Operations Research and Management Science

Wiley Encyclopedia of Operations Research and Management Science

How to Cite

Powell, W. B. 2011. Approximate Dynamic Programming I: Modeling. Wiley Encyclopedia of Operations Research and Management Science. .

Author Information

  1. Princeton University, Department of Operations Research and Financial Engineering, Princeton, New Jersey

Publication History

  1. Published Online: 14 JAN 2011

Abstract

The first step in solving a stochastic optimization problem is providing a mathematical model. How the problem is modeled can impact the solution strategy. In this article, we provide a flexible modeling framework that uses a classic control-theoretic framework, avoiding devices such as one-step transition matrices. We describe the five fundamental elements of any stochastic, dynamic program. Different notational conventions are introduced, and the types of policies that can be used to guide decisions are described in detail. This discussion puts approximate dynamic programming in the context of a variety of other algorithmic strategies by using the modeling framework to describe a wide range of policies. A brief discussion of model-free programming is also provided.

Keywords:

  • approximate dynamic programming;
  • Markov decision process;
  • state variable;
  • transition function;
  • model-free dynamic programming