Your brain does not experience sight and sound as separate streams that happen to coincide. Instead, it actively merges visual and auditory signals at multiple levels, from ancient midbrain structures to sprawling cortical networks, producing a unified perception that is often richer and faster than either sense alone. This merging process, called multisensory integration, shapes everything from understanding speech to locating a ringing phone in a cluttered room, and it can even trick you into seeing or hearing things that are not there.
The First Stop for Merging Sight and Sound
Before visual and auditory information ever reaches the cortex, the outer layers of the brain that handle complex thought, it passes through a small midbrain structure called the superior colliculus. This region sits atop the brainstem and has been conserved across vertebrate species for hundreds of millions of years. It registers events happening in the space around you, building layered maps: visual information in the uppermost layer, auditory and other senses in deeper layers, and a motor map at the bottom that helps you orient toward whatever grabbed your attention.
Because these maps are spatially aligned, a sound coming from the left and a flash of light coming from the left converge on the same patch of the superior colliculus, reinforcing each other. Research in mice has shown that about half of the neurons studied in the superior colliculus respond to both visual and auditory stimuli, and many of those responses are nonlinear, meaning the combined audiovisual response is stronger (or sometimes weaker) than you would predict by simply adding the two signals together.
This nonlinear boost is one reason a faint rustle in the bushes coupled with a flicker of movement will snap your head around faster than either cue would on its own. The superior colliculus acts as a rapid-response integrator, fusing sensory streams and kicking off orienting movements before you have consciously registered what happened.
Cortical Regions That Bind Complex Objects
While the superior colliculus handles fast, reflexive merging, more nuanced integration occurs in the cortex. A region that consistently lights up in brain-imaging studies is the posterior superior temporal sulcus, or STS, a groove running along the side of the brain roughly behind your ear. When people in experiments identified objects by both sight and sound together, the STS responded more strongly than when either sense was presented alone, a hallmark of genuine integration rather than mere co-activation.
The STS also responded more to meaningful stimuli, like recognizable pictures paired with real sounds, than to scrambled or meaningless versions. This suggests the STS is not just detecting co-occurring signals; it is binding them into coherent object representations. If you see a dog and hear a bark, the STS helps your brain treat those as one event rather than two unrelated inputs. The STS also handles touch-sound and touch-vision pairings, making it a general-purpose multisensory hub rather than a strictly audiovisual one.
Wiring That Skips the Middleman
For a long time, the textbook picture of sensory processing assumed a strict hierarchy: sounds go to auditory cortex, images go to visual cortex, and the two meet only later in so-called “association” areas. That picture turns out to be incomplete. Anatomical tracing studies in primates have found direct, single-synapse connections running from auditory cortex to the primary visual cortex, the very first cortical stop for visual processing. These projections originate in the rear portions of auditory cortex and target parts of visual cortex devoted to peripheral vision.
This finding has been confirmed across primate species that diverged over 30 million years ago, indicating that early-stage audiovisual interaction is a deeply embedded feature of primate brains, not some quirk of one lineage. Similar cross-wiring has been demonstrated in rodents, where primary auditory and primary visual cortex share direct connections with each other and with somatosensory (touch) cortex. In humans, diffusion-imaging studies have identified white-matter tracts that appear to link motion-sensitive visual and auditory regions directly, potentially supporting rapid exchange of motion information between the two senses.
These shortcut connections likely explain why a sudden sound can change what you see almost instantly, well before a signal could travel up to a high-level integration area and back down again.
How Brain Rhythms Synchronize the Senses
Integration is not only about which brain regions are connected; it also depends on timing. Neurons communicate partly through rhythmic electrical oscillations, and one important mechanism for audiovisual merging is called phase reset. When a sound arrives, it can reset the ongoing oscillation in visual cortex, effectively opening a window during which visual signals are processed more efficiently. Recordings taken directly from the surface of the human brain have confirmed that auditory stimulation triggers phase resets in visual cortex, and that presenting both a sound and an image together produces stronger integration effects than either stimulus alone.
The timing precision matters too. When researchers measured brainwave coherence as people judged whether a beep and a flash occurred simultaneously, they found that phase alignment was greatest when the two stimuli were closest together in time. This enhancement was most prominent in the beta frequency band, roughly 13 to 30 cycles per second, over the posterior temporal cortex. In plain terms, the brain’s electrical rhythms act like a metronome that helps align the processing of sight and sound so they can be fused into a single percept.
The Temporal Binding Window
Your brain does not require sight and sound to arrive at exactly the same instant in order to merge them. There is a window of time, typically a few hundred milliseconds wide, during which the brain treats audiovisual signals as belonging together. This is called the temporal binding window, and its width varies from person to person. Research has shown that a wider window makes you more susceptible to audiovisual illusions: your brain is more willing to glue together signals that are slightly out of sync, which can be helpful in noisy environments but also makes you easier to fool.
The window is not fixed. Training studies have demonstrated that practicing a simultaneity-judgment task can narrow it, at least under certain conditions. In one experiment, people who trained with bright visual stimuli successfully tightened their binding window for bright stimuli, though the improvement did not transfer to dim stimuli. This specificity suggests the window is tuned by experience in a context-dependent way rather than shifting as a single global dial.
Illusions That Reveal the Machinery
Some of the most compelling evidence for deep audiovisual integration comes from perceptual illusions, situations where one sense literally changes what the other sense perceives.
In the McGurk effect, watching a person’s lips form one syllable while hearing a different syllable produces a third, blended percept. A classic example: if you hear “ba” while watching lips say “ga,” most people report hearing “da,” a sound that was neither spoken nor played. Brain imaging has pinpointed the left STS as a critical site for this illusion. The strength of the STS response predicts how susceptible a given person is: people with a stronger STS signal are more likely to hear the fused syllable. Stimulating the STS and posterior parietal cortex with mild electrical current can actually reduce the McGurk illusion, confirming that these regions are not just correlated with the effect but causally involved in producing it.
The sound-induced flash illusion works in the opposite direction, using hearing to alter vision. When a single flash of light is accompanied by two rapid beeps, many people report seeing two flashes. This is not a subtle bias; it is a genuine shift in visual experience. One study found that even when people received performance feedback telling them they were wrong, the illusion persisted, suggesting it arises at a level of processing that resists conscious correction. Individual differences in susceptibility appear to relate to the amount of grey matter in early visual cortex, hinting that the structural wiring of your visual system partly determines how much your hearing can push your vision around.
When Vision Overrules Sound
Vision does not always lose the tug-of-war. In the ventriloquism illusion, seeing a visual event at one location causes a sound to be perceived as coming from that location, even when the sound actually originates elsewhere. This is why a ventriloquist’s dummy appears to be talking: your visual system drags your perception of the voice’s location toward the dummy’s moving mouth.
Neuroimaging and electrical recordings show that this illusion corresponds to a biased pattern of activity in the planum temporale, a part of auditory cortex. On trials where people report the ventriloquism illusion, the auditory cortex hemisphere on the same side as the visual stimulus becomes relatively quieter, while the opposite hemisphere stays strong, effectively shifting the brain’s spatial representation of the sound toward the visual event. The illusion reaches full strength after just a single paired presentation and does not decay over time, behaving like a “sample and hold” process that locks in the combined percept. A separate but related phenomenon, the ventriloquism aftereffect, accumulates with repeated exposure and gradually fades, suggesting two distinct mechanisms at work.
How Audiovisual Integration Develops
Babies are not born with fully mature audiovisual integration. A cross-sectional study examining people from 3 months of age through 30 years found that adult-like integration responses in brain electrical activity do not emerge until around 8 years old and continue to refine through adolescence, with the speed of neural responses getting faster with age. This long developmental timeline means that the experiences children have during their early years, the sounds paired with sights, the faces paired with voices, are actively shaping the wiring that supports multisensory perception.
The temporal binding window also changes with development. Young children tend to have wider windows, making them more tolerant of mismatches between sight and sound. As the brain matures and gains experience with the statistical regularities of the audiovisual world, the window narrows, allowing for more precise temporal discrimination.
Audiovisual Processing in Autism
Research on autism spectrum conditions has provided a striking illustration of what happens when the temporal binding window is atypical. One study found that children with autism showed a binding window roughly twice as wide as that of typically developing children, spanning about 600 milliseconds compared to about 300 milliseconds. This wider window means that signals arriving well out of sync are still merged, which could contribute to the sensory overwhelm that many autistic individuals describe: the brain is binding together too many signals that do not actually belong together.
The picture is not entirely consistent across studies, however. A study in autistic adults found that they were actually less susceptible to the sound-induced flash illusion, and that this reduced susceptibility was associated with a narrower binding window. One possible reconciliation is developmental. A longitudinal-style study of autistic individuals aged 6 through 18 found that their binding windows narrowed significantly with age, while neurotypical children’s windows remained relatively stable across the same age range. The gap between autistic and neurotypical groups shrank as the autistic group matured, suggesting that the integration differences seen in younger autistic children may partially resolve over time through a delayed but ongoing maturational process.
Aging and Enhanced Integration
At the other end of the lifespan, something counterintuitive happens. A systematic review of studies comparing older and younger adults found that older adults generally show greater audiovisual integration, meaning they benefit more from having both senses available than younger adults do. When both a sound and a visual cue are present, older adults show larger performance gains compared to when only one modality is available.
This increased reliance on multisensory information may be a compensatory strategy. As individual senses decline with age, the brain leans more heavily on combining whatever signals are available. The downside is that a wider integration window also means older adults may be more susceptible to audiovisual illusions, binding together signals that do not actually belong together. The practical implication is significant: environments designed for older adults, from hospital rooms to transit systems, can leverage multisensory cues to improve communication and safety, pairing visual alerts with sounds rather than relying on one channel alone.
The Brain’s Guesswork About What Belongs Together
Given that the world constantly bombards you with many simultaneous sounds and sights, how does the brain decide which ones belong together? A framework called Bayesian causal inference has emerged as a leading explanation. The basic idea is that the brain is constantly making a probabilistic bet: did this sound and this image come from the same event, or from two separate events? If the brain decides they share a common cause, the signals are merged and each one influences the other. If they are judged to come from different sources, the brain keeps them separate.
This framework has been tested across a wide range of perceptual tasks and has outperformed simpler models, including “mandatory integration” (always merge) and “sensory dominance” (one sense always wins). In distance-perception experiments, for instance, Bayesian causal inference predicted human performance better than any alternative model. The brain does not blindly fuse everything or always defer to one sense; it weighs the reliability of each signal and the spatial and temporal plausibility of a shared origin before making its call.
When One Sense Is Lost
Cross-modal plasticity, the brain’s ability to repurpose sensory territory when a sense is lost, provides perhaps the most dramatic evidence of how interconnected the visual and auditory systems really are. In people who are born blind or lose vision early in life, regions that would normally process visual information are recruited to handle auditory and tactile tasks instead. This reorganization is not just a curiosity; it has functional consequences. Some blind individuals develop enhanced spatial hearing or finer tactile discrimination, abilities supported by the co-opted visual cortex.
Sensory-substitution devices take advantage of this plasticity. These tools convert visual information into auditory or tactile patterns, a camera feed translated into a soundscape, for example. When sighted people trained on such devices performed depth-perception tasks using only the auditory translation of visual scenes, brain imaging revealed activation in visual association areas, specifically occipito-parietal and occipito-temporal cortex. Those regions responded as though they were receiving visual input, even though the information arrived as sound. This suggests that parts of the visual cortex are not strictly “visual” at all but are organized around the type of computation (like depth or spatial layout) rather than the sensory channel that delivers the data.
Attention Steers the Whole Process
None of this integration happens in a vacuum. What you are paying attention to dramatically shapes how sight and sound interact. When researchers compared brain activity during conditions requiring divided attention (monitoring both senses) versus selective attention (focusing on one), they found differences in the activation of the frontoparietal network, the sensory cortices themselves, and subcortical structures like the putamen. Attending to one modality can enhance processing in that modality’s cortex while suppressing the other, effectively turning the integration dial up or down depending on the task.
This top-down control has real-world consequences. In a noisy restaurant, you can boost your speech comprehension by watching the speaker’s face, a deliberate recruitment of visual information to supplement a degraded auditory signal. Conversely, closing your eyes during a concert can heighten your auditory experience by reducing visual competition for neural resources. The integration system is not a passive reflex; it is shaped moment by moment by your goals, expectations, and the demands of the situation.
Audiovisual Integration in Other Species
Humans are not unique in combining sight and sound. The superior colliculus structure that performs rapid audiovisual merging is present across vertebrates, from fish to mammals, reflecting its ancient evolutionary origins. Field studies of wild yellow-bellied marmots illustrate how multisensory integration operates outside the lab. When researchers presented marmots with predator cues, combining a visual stimulus (like a predator model) with an auditory stimulus (an alarm call) produced stronger antipredator responses than either cue alone, but only under low-noise conditions. When background noise was high, the auditory signal alone drove most of the response, and adding a visual cue on top did not help much. This pattern mirrors the reliability-weighting principle seen in human Bayesian models: the brain leans on whichever channel is most informative under current conditions.
Synesthesia and the Extremes of Cross-Modal Linking
At the far end of the audiovisual integration spectrum sits synesthesia, a condition in which stimulation of one sense automatically triggers a perception in another. Some synesthetes hear sounds when they see moving patterns, a variant sometimes called “hearing motion” or motion-sound synesthesia. An MRI study of such individuals found structural differences in the superior and inferior colliculi, the same subcortical structures involved in normal audiovisual integration, along with altered connectivity between these midbrain hubs and cortical motion and temporal regions. The synesthetes also showed stronger activation in early visual areas when viewing motion stimuli. These findings suggest that synesthesia may not be an entirely separate phenomenon but rather an extreme expression of the same cross-modal wiring that underlies everyday multisensory integration, with the volume turned up by differences in subcortical anatomy.