Using deep reinforcement learning to reveal how the brain encodes abstract state-space representations in high-dimensional environments

Logan Cross; Jeff Cockburn; Yisong Yue; John P O'Doherty

doi:10.1016/j.neuron.2020.11.021

Back

Using deep reinforcement learning to reveal how the brain encodes abstract state-space representations in high-dimensional environments

Journal article

Peer reviewed

Using deep reinforcement learning to reveal how the brain encodes abstract state-space representations in high-dimensional environments

Logan Cross, Jeff Cockburn, Yisong Yue and John P O'Doherty

Neuron (Cambridge, Mass.), Vol.109(4), pp.724-738.e7

02/17/2021

DOI: 10.1016/j.neuron.2020.11.021

PMCID: PMC7897245

PMID: 33326755

Files and links (1)

url

https://www.ncbi.nlm.nih.gov/pmc/articles/7897245View

Open Access

Abstract

Humans possess an exceptional aptitude to efficiently make decisions from high-dimensional sensory observations. However, it is unknown how the brain compactly represents the current state of the environment to guide this process. The deep Q-network (DQN) achieves this by capturing highly nonlinear mappings from multivariate inputs to the values of potential actions. We deployed DQN as a model of brain activity and behavior in participants playing three Atari video games during fMRI. Hidden layers of DQN exhibited a striking resemblance to voxel activity in a distributed sensorimotor network, extending throughout the dorsal visual pathway into posterior parietal cortex. Neural state-space representations emerged from nonlinear transformations of the pixel space bridging perception to action and reward. These transformations reshape axes to reflect relevant high-level features and strip away information about task-irrelevant sensory features. Our findings shed light on the neural encoding of task representations for decision-making in real-world situations.

Adult

Brain - diagnostic imaging

Brain - physiology

Deep Learning

Female

Humans

Magnetic Resonance Imaging - methods

Male

Psychomotor Performance - physiology

Reinforcement, Psychology

Video Games

Young Adult

Details

Title: Subtitle: Using deep reinforcement learning to reveal how the brain encodes abstract state-space representations in high-dimensional environments
Creators: Logan Cross - California Institute of Technology
Jeff Cockburn - California Institute of Technology
Yisong Yue - California Institute of Technology
John P O'Doherty - Division of Humanities and Social Sciences, California Institute of Technology, Pasadena, CA 91125, USA
Resource Type: Journal article
Publication Details: Neuron (Cambridge, Mass.), Vol.109(4), pp.724-738.e7
DOI: 10.1016/j.neuron.2020.11.021
PMID: 33326755
PMCID: PMC7897245
NLM abbreviation: Neuron
ISSN: 0896-6273
eISSN: 1097-4199
Grant note: R01 DA040011 / NIDA NIH HHS P50 MH094258 / NIMH NIH HHS
Language: English
Date published: 02/17/2021
Academic Unit: Psychological and Brain Sciences
Record Identifier: 9984696752102771

Metrics

7 Record Views

39 Times Cited - Web of Science