A simple method for dealing with large state spaces
Article first published online: 17 SEP 2012
© 2012 The Author. Methods in Ecology and Evolution © 2012 British Ecological Society
Methods in Ecology and Evolution
Volume 3, Issue 6, pages 949–957, December 2012
How to Cite
Schapaugh, A. W., Tyre, A. J. (2012), A simple method for dealing with large state spaces. Methods in Ecology and Evolution, 3: 949–957. doi: 10.1111/j.2041-210X.2012.00242.x
- Issue published online: 11 DEC 2012
- Article first published online: 17 SEP 2012
- Manuscript Accepted: 24 JUL 2012
- Manuscript Received: 10 MAR 2012
- curse of dimensionality;
- Markov decision process;
- state space;
- stochastic dynamic programming
- Most sequential decision-making problems in conservation can be viewed conceptually and modelled as a Markov decision process. The goal in this context is to construct a policy that associates each state of the system with a particular action. This policy should offer optimal performance in the sense of maximizing or minimizing a specified conservation objective
- Dynamic programming algorithms rely on explicit enumeration to derive the optimal policy. This is problematic from a computational perspective as the size of the state space grows exponentially with the number of state variables.
- We present a state aggregation method where the idea is to capture the most important aspects of the original Markov decision process, find an optimal policy over this reduced space and use this as an approximate solution to the original problem.
- Applying the aggregation method to a species reintroduction problem, we demonstrate how we were able to reduce the number of states by 75% and reduce the size of the transition matrices by almost 94% (324 vs. 5184), and the abstract action matched the optimal action more than 86% of the time.
- We conclude that the aggregation method is not a panacea for the curse of dimensionality, but it does advance our ability to construct approximately optimal policies in systems with large state spaces.