Logo image
Back
Novelty and uncertainty regulate the balance between exploration and exploitation through distinct mechanisms in the human brain
Journal article   Open access   Peer reviewed

Novelty and uncertainty regulate the balance between exploration and exploitation through distinct mechanisms in the human brain

Jeffrey Cockburn, Vincent Man, William A. Cunningham and John P. O’Doherty
Neuron (Cambridge, Mass.), Vol.110(16), pp.2691-2702.e8
08/17/2022
DOI: 10.1016/j.neuron.2022.05.025
PMID: 35809575
url
https://doi.org/10.1016/j.neuron.2022.05.025View
Published (Version of record) Open Access

Abstract

Both novelty and uncertainty are potent features guiding exploration; however, they are often experimentally conflated, and an understanding of how they interact to regulate the balance between exploration and exploitation has proved elusive. Using a task designed to decouple the influence of novelty and uncertainty, we identify separable mechanisms through which exploration is directed. We show that uncertainty-directed exploration is sensitive to the prospective benefit offered by new information, whereas novelty-directed exploration is maintained regardless of its potential advantage. Using a computational framework in conjunction with fMRI, we show that uncertainty-directed choice is rooted in an adaptive bias indexing the prospective utility of exploration. In contrast, novelty persistently promotes exploration by optimistically inflating reward expectations while simultaneously dampening uncertainty signals. Our results identify separable neural substrates charged with balancing the explore/exploit trade-off to foster a manageable decomposition of an otherwise intractable problem. •Uncertainty-directed exploration considers the prospective benefit of new information.•Novelty-directed exploration is myopic and motivated by inflated reward expectation.•Option features are integrated by vmPFC to balance the explore/exploit trade-off.•Integrating a mixture of strategies offers a tractable approximation of optimal control. Cockburn et al. show that novelty and uncertainty are used by the human brain to guide distinct exploration strategies. Uncertainty-directed exploration considers the prospective benefit of new information, whereas novelty motivates exploration by inflating the brain’s expectation of reward, offering a feasible decomposition of an otherwise intractable explore/exploit dilemma.
 decision making exploit explore fMRI reinforcement learning

Details

Metrics

7 Record Views
36 Times Cited - Web of Science
Logo image