The role of experience replay in planning and non-local learning

The ability to simulate situations beyond our local environment is a highly adaptive feature of intelligence. In both biological and artificial agents, the replay of nonlocal experiences enables learning and planning by linking actions and outcomes across time and space. Prior work suggests that humans and animals often engage in replay whenever it is beneficial, and that its omission might underlie reflexive, habitual, or compulsive behaviors. Yet, such a dichotomous view (act vs. deliberate) obscures a complex selection problem: if experiences can be replayed nonlocally and long before they are needed, which experiences should the brain select for replay at each moment to set the stage for the most rewarding future decisions? In this talk, I will describe modeling and experimental work characterizing, in humans and animals, (i) what is replayed, when and (ii) the result of replay on future behavior. First, I will present a reinforcement learning theory describing which experiences should the brain replay, at each moment, to optimize future decisions. This theory quantifies the utility of a particular replayed experience and predicts that forward and backward replay are each favored in different circumstances, matching patterns of place-cell activity frequently observed in the rodent hippocampus. Second, I will present a magnetoencephalography (MEG) study demonstrating that backward replay faciliates nonlocal learning in humans. Third, I will present an fMRI study demonstrating that humans compute values in board games iteratively via forward replay, using a tree-search algorithm. Finally, I will propose that our episodic memory system is capable of replaying experiences in a manner well suited for planning. Overall, these findings establish a framework for investigating the role of replay in adaptive behavior and posit a mechanism whose dysfunction may underlie pathologies like rumination and craving.

Marcelo Mattar is an Assistant Professor in the department of Cognitive Science at University of California, San Diego. His lab studies the neural computations that generate intelligent, goal-directed behavior, focusing primarily on: (i) how our memory systems build internal models of the world, and (ii) how we can use these representations to simulate the future when making a decision. His lab addresses these questions using behavioral experiments, neural recordings, and computational models formalized in the language of reinforcement learning.