Professor Andrew G. Barto
Lecture Abstract
Learning to Exploit Dynamics in Motor Control
A conspicuous feature of how animals move is that they are able to exploit the natural dynamics of their motor systems and their environments to achieve goals. We have been working toward a view of motor skill acquisition that synthesizes perspectives from decision and control theory, machine learning, and the anatomy and physiology of the motor system to account for an ability to learn to exploit nonlinear dynamics in a goal-directed manner. We place particular emphasis on the theory of optimal control. Unlike the more familiar types of control, such as regulation and tracking, optimal control requires taking full advantage of a system's natural dynamics and is gaining prominence in models of motor control.
Key for this talk is the fact that optimal control forms the basis of simple yet powerful learning strategies that can shape and refine the behavior of a nonlinear dynamic system over time on the basis of experience. Machine learning researchers call these reinforcement learning methods. In this talk, I discuss optimal control in general terms and then present the elements of modern reinforcement learning, emphasizing connections to animal behavior and brain mechanisms. In particular, I discuss Sutton and Barto's Temporal Difference (TD) model of Pavlovian conditioning and the arresting parallels between TD learning and the activity of dopamine neurons in the basal ganglia. I then describe a model of neural motor control mechanisms suggested by these and other observations about how the basal ganglia, cerebellum, and motor cortex might interact in learning fast and accurate reaching movements in a manner that exploits the arm's nonlinear dynamics.
Recommended Readings
(For Friday discussion and Tuesday informal meeting -- try to read all four)
- Machine Learning for Subproblem Selection
- Combining Reinforcement Learning with a Local Control Algorithm
- A Cerebellar Model of Timing and Prediction in the Control of Reaching
- Learning to Reach Via Corrective Movements