Successor Representation: Brain Mechanisms and Reinforcement

The brain predicts future outcomes based on past experiences, a process essential for learning and decision-making. Successor representation (SR) is a computational framework that explains how the brain efficiently anticipates rewards by encoding relationships between states rather than memorizing specific sequences. This approach allows for faster adaptation to environmental changes compared to traditional reinforcement learning models.

Understanding SR sheds light on cognitive flexibility, spatial navigation, and neurological conditions. Researchers are studying its neural basis, role in learning, and relevance across species to uncover its full implications.

Neural Basis In The Human Brain

The human brain constructs predictive models by encoding relationships between different states, a process SR supports. Neuroimaging and electrophysiological studies indicate that SR relies on a network of brain regions, particularly the hippocampus, prefrontal cortex, and striatum. The hippocampus, associated with spatial and episodic memory, encodes predictive maps reflecting past experiences. Functional MRI (fMRI) studies show hippocampal activity correlates with expected future states rather than just immediate locations, supporting its role in maintaining predictive maps.

The prefrontal cortex integrates these predictive maps with goal-directed decision-making. Magnetoencephalography (MEG) research demonstrates that prefrontal activity updates SR-based predictions when environmental contingencies shift. The striatum, particularly the dorsolateral and dorsomedial regions, translates these predictions into action selection. Studies using reinforcement learning paradigms show striatal activity tracks discrepancies between expected and actual state transitions, refining SR-based learning.

Single-unit recordings in humans undergoing neurosurgical procedures further support SR’s neural basis. Neurons in the hippocampus and entorhinal cortex fire in patterns consistent with predictive coding. Research on grid cells in the entorhinal cortex shows their firing patterns extend beyond immediate locations, suggesting a mechanism for encoding long-range state transitions. This aligns with computational models proposing the brain maintains predictive maps generalizing across contexts.

Role In Reinforcement Learning

SR offers a distinct approach to reinforcement learning by encoding predictive relationships between states rather than associating actions with immediate rewards. Traditional model-free reinforcement learning relies on trial-and-error mechanisms to update value estimates based on direct reward feedback. While effective in stable environments, these methods struggle with adaptability when task structures change. SR addresses this limitation by maintaining a predictive map of state transitions, allowing for efficient recalibration when reward contingencies shift.

SR facilitates rapid policy adjustments by encoding expected future states independently of reward values. This enables the brain to recompute value functions without relearning the entire environment. For instance, if a reward location changes in a familiar setting, SR-equipped systems update value estimates by modifying the reward association rather than reconstructing the transition model. In contrast, model-free reinforcement learning requires extensive new experiences to adjust behavior. Empirical studies show human participants using SR-based strategies adapt faster to altered reward structures, highlighting its efficiency in dynamic environments.

Neuroscientific evidence reinforces SR’s role in reinforcement learning. Functional MRI studies show hippocampal and striatal regions track state occupancy probabilities even without explicit rewards, suggesting the brain maintains an intrinsic representation of environmental structure. Multi-step decision task research reveals individuals with stronger SR-like representations exhibit more flexible learning, generalizing previous knowledge to novel contexts. This supports computational models linking SR to cognitive flexibility by decoupling state predictions from immediate reward dependencies.

Relevance To Spatial Cognition

Navigating an environment requires constructing internal maps representing spatial relationships between locations. SR contributes by encoding not just locations but how they relate through movement sequences. Unlike purely location-based models, SR predicts future states based on past transitions, enabling efficient route planning and adaptation. This predictive encoding is particularly useful in dynamic settings where landmarks or pathways change, allowing recalibration without complete relearning.

Studies using virtual reality mazes show participants rely on SR-like strategies when learning new environments, generalizing knowledge to novel routes rather than memorizing sequences. Neuroimaging data supports this distinction, revealing hippocampal activity reflects predictive spatial coding, with stronger engagement when navigating shifting pathways.

Research in non-human species highlights SR’s role in spatial reasoning. Rodents navigating T-mazes show hippocampal place cells fire in patterns consistent with predictive spatial encoding, suggesting SR-like mechanisms guide decision-making. Similar observations in bats and primates show neural activity extending beyond immediate locations to anticipate future positions. The presence of SR-based navigation across species underscores its evolutionary advantage in efficient spatial problem-solving.

Observations In Animal Studies

Animal research provides compelling evidence that SR informs how organisms predict and navigate future states. Rodent studies show hippocampal place cells anticipate upcoming positions, indicating animals construct internal models rather than relying solely on stimulus-response associations. Experiments using T-mazes and open-field tasks demonstrate that when reward locations shift, rodents using SR-like strategies adapt more efficiently than those relying on simple reinforcement learning.

Electrophysiological recordings further illuminate SR’s neural mechanisms. Studies tracking hippocampal activity in rats navigating mazes identify place cell sequences extending beyond immediate positions, often representing trajectories toward future goals. This “preplay” phenomenon aligns with SR’s computational principles, where state representations incorporate future transitions. Research on entorhinal grid cells suggests these neurons generalize spatial relationships across environments, reinforcing SR’s role in flexible navigation.

Links To Cognitive Disorders

Dysfunctions in SR have been linked to cognitive disorders affecting learning, decision-making, and predictive processing. Since SR encodes statistical relationships between states, disruptions can impair adaptation to new environments, anticipation of future outcomes, and flexible belief updating. Conditions such as schizophrenia, obsessive-compulsive disorder (OCD), and Parkinson’s disease are associated with SR-related deficits.

Schizophrenia is characterized by difficulties forming coherent mental models, aligning with SR impairments. Studies using reinforcement learning tasks show individuals with schizophrenia struggle to update expectations when task contingencies change, indicating reliance on less flexible learning strategies. Neuroimaging research reveals altered hippocampal-prefrontal connectivity in schizophrenia patients, regions crucial for SR-based predictions.

Similarly, individuals with OCD exhibit maladaptive reinforcement learning patterns, often favoring rigid, habitual behaviors over flexible, goal-directed strategies. This suggests dysfunction in neural circuits responsible for adjusting SR-based predictions, particularly in the striatum and orbitofrontal cortex, which are central to action-outcome learning.

Parkinson’s disease presents another example of SR-related impairments. Degeneration of dopaminergic neurons in the basal ganglia disrupts predictive model encoding, leading to difficulties adjusting to new reward contingencies. Patients often rely on habitual responses rather than flexible learning. Since the striatum plays a significant role in translating SR-based predictions into adaptive behavior, its deterioration in Parkinson’s further underscores SR’s importance. Understanding these links offers potential therapeutic avenues, such as targeted neuromodulation or pharmacological treatments aimed at restoring predictive processing.