Solving the Distal Reward Problem through Linkage of STDP and Dopamine Signaling

Cerebral Cortex (2007) 17:2443--2452 (10.1093/cercor/bhl152)

Eugene M. Izhikevich

The Neurosciences Institute,
10640 John Jay Hopkins Drive,
San Diego, CA, 92121.

Abstract. In Pavlovian and instrumental conditioning, rewards typically come seconds after reward-triggering actions, creating an explanatory conundrum known as “distal reward problem”: How does the brain know what firing patterns of what neurons are responsible for the reward if (1) the patterns are no longer there when the reward arrives and (2) most neurons and synapses are active during the waiting period to the reward? Here we show how the conundrum is resolved by a model network of cortical spiking neurons with spike-timing-dependent plasticity (STDP) modulated by dopamine (DA). Although STDP is triggered by nearly-coincident firing patterns on a millisecond time scale, slow kinetics of subsequent synaptic plasticity is sensitive to changes in the extracellular DA concentration during the critical period of a few seconds. Random firings during the waiting period to the reward do not affect STDP, and hence make the network insensitive to the ongoing activity --- the key feature that distinguishes our approach from previous theoretical studies, which implicitly assume that the network be quiet during the waiting period or that the patterns be preserved until the reward arrives. This study emphasizes the importance of precise firing patterns in brain dynamics and suggests how a global diffusive reinforcement signal in the form of DA can selectively influence the right synapses at the right time.

Full text in PDF file,   MATLAB file reproducing figure 1

[home] [research] [publications] [cv]