Statistics-based Approaches To Stochastic Optimal Control Problems

Abstract

This dissertation develops practical approaches to stochastic optimal control problems in the absence of linear system transition functions, quadratic cost functions, and/or Gaussian random disturbances. In such type of problems, an analytical solution is impossible and numerical synthesis techniques have to be applied. The ''classic'' algorithms existing in the literature suffer from the problem of the''curse of dimensionality,'' an exponential increase of computation time and memory requirements as the dimension of the problem grows. The statistics-based numerical approaches are presented as the main tools for mitigating the ``curse of dimensionality'' phenomenon. Two approaches are explored: (A) stochastic dynamic programming (SDP), which approximates the future value functions and solves the recursion relation backwards in stages, and (B) stochastic gradient (SG), which approximates the control functions by linear combinations of certain basis functions containing free parameters, and optimizes the parameters through iterations over sequences of the system's random variables. The research presented here views the approximation of future value functions in SDP and the approximation of control functions in SG via a computer experiments perspective, and integrates statistical methods from the area of design and analysis of computer experiments (DACE) into SDP and SG approaches to enable numerical solution to large SOC problems. Recent developments in DACE make it possible to approximate high-dimensional, complex input-output relationships with moderate computation time and memory requirements. A statistical perspective of future value function approximation in high-dimensional, continuous-state SDP was first presented using orthogonal array (OA) experimental designs and multivariate adaptive regression splines (MARS) statistical models. This work utilizes number theoretic methods (NTMs) and artificial neural networks (ANNs) as alternatives to OAs and MARS respectively, and introduces the statistical perspective to SG approach. Comparisons considering the differences in methodological objectives, model accuracy and numerical solutions are presented. Three problems are tested: a nine-dimensional inventory forecasting problem, an eight-dimensional water reservoir network management problem, and a thirty-dimensional water reservoir network management problem. This last application is the largest water reservoir problem solved in the literature.