Header logo is

Learning optimal control solutions: a path integral approach


Conference Paper


Investigating principles of human motor control in the framework of optimal control has had a long tradition in neural control of movement, and has recently experienced a new surge of investigations. Ideally, optimal control problems are addresses as a reinforcement learning (RL) problem, which would allow to investigate both the process of acquiring an optimal control solution as well as the solution itself. Unfortunately, the applicability of RL to complex neural and biomechanics systems has been largely impossible so far due to the computational difficulties that arise in high dimensional continuous state-action spaces. As a way out, research has focussed on computing optimal control solutions based on iterative optimal control methods that are based on linear and quadratic approximations of dynamical models and cost functions. These methods require perfect knowledge of the dynamics and cost functions while they are based on gradient and Newton optimization schemes. Their applicability is also restricted to low dimensional problems due to problematic convergence in high dimensions. Moreover, the process of computing the optimal solution is removed from the learning process that might be plausible in biology. In this work, we present a new reinforcement learning method for learning optimal control solutions or motor control. This method, based on the framework of stochastic optimal control with path integrals, has a very solid theoretical foundation, while resulting in surprisingly simple learning algorithms. It is also possible to apply this approach without knowledge of the system model, and to use a wide variety of complex nonlinear cost functions for optimization. We illustrate the theoretical properties of this approach and its applicability to learning motor control tasks for reaching movements and locomotion studies. We discuss its applicability to learning desired trajectories, variable stiffness control (co-contraction), and parameterized control policies. We also investigate the applicability to signal dependent noise control systems. We believe that the suggested method offers one of the easiest to use approaches to learning optimal control suggested in the literature so far, which makes it ideally suited for computational investigations of biological motor control.

Author(s): Theodorou, E. and Schaal, S.
Book Title: Abstracts of Neural Control of Movement Conference (NCM 2010)
Year: 2010

Department(s): Autonomous Motion
Bibtex Type: Conference Paper (inproceedings)

Address: Naples, Florida, 2010
Cross Ref: p10410
Note: clmc


  title = {Learning optimal control solutions: a path integral approach},
  author = {Theodorou, E. and Schaal, S.},
  booktitle = {Abstracts of Neural Control of Movement Conference (NCM 2010)},
  address = {Naples, Florida, 2010},
  year = {2010},
  note = {clmc},
  crossref = {p10410}