# data[].attributes.body
In this exhibition we will be talking and learning all about consciousness and perception of the human mind and body. They are two separate entities although they work together in many ways. With this exhibition we will explore into detail, how consciousness and perception are on their own as well as how they are together. We will examine scientific articles, as well as videos and images. These artifacts will help us dive deeper into the unanswered questions of perception and consciousness and increase our understanding on both topics.
First we will explore into consciousness by itself, what it is, and where it is. We will engage into what consciousness as a whole really is, and how it's described. Then we'll move into perception on its own, what is perception, what senses and other things play roles in perception, how does perception differ between fictional and non fictional things. After examining perception and consciousness on their own, we'll study them together. How does perception affect consciousness? When does consciousness become perception? How do they work together to form each others response?
At the end of this exhibition it is not expected for any of the posed questions to be answered, but more expected to give some insight into the topics to give way for more questions.

# data[].attributes.headline
Consciousness and Perception Joined But Separate

# data[].attributes.subject
Consciousness and Perception Joined But Separate

# data[].attributes.body
How do we separate between consciousness and perception once they are joined together? When we experience something, what part of the experience is your consciousness and what part of that experience is your perception? In this video from the University of California Berkeley, David Presti briefly speaks on this topic and idea.

# data[].attributes.attachment_link.title
Brain, Mind, Perception, and Consciousness

# data[].attributes.body
What is consciousness? How is something we all experience, that is not tangible described? It's everything, anything and nothing all at the same time. It's what we feel and don't feel, and what we see and don't see. Christof Koch dives deeper into this topic below.

# data[].attributes.attachment_link.readable_text
What Is Consciousness?

Scientists are beginning to unravel a mystery that has long vexed philosophers

By Christof Koch

Consciousness is everything you experience. It is the tune stuck in your head, the sweetness of chocolate mousse, the throbbing pain of a toothache, the fierce love for your child and the bitter knowledge that eventually all feelings will end.

The origin and nature of these experiences, sometimes referred to as qualia, have been a mystery from the earliest days of antiquity right up to the present. Many modern analytic philosophers of mind, most prominently perhaps Daniel Dennett of Tufts University, find the existence of consciousness such an intolerable affront to what they believe should be a meaningless universe of matter and the void that they declare it to be an illusion. That is, they either deny that qualia exist or argue that they can never be meaningfully studied by science.

If that assertion was true, this essay would be very short. All I would need to explain is why you, I and most everybody else is so convinced that we have feelings at all. If I have a tooth abscess, however, a sophisticated argument to persuade me that my pain is delusional will not lessen its torment one iota. As I have very little sympathy for this desperate solution to the mind-body problem, I shall move on.

On supporting science journalism

If you're enjoying this article, consider supporting our award-winning journalism by subscribing. By purchasing a subscription you are helping to ensure the future of impactful stories about the discoveries and ideas shaping our world today.

The majority of scholars accept consciousness as a given and seek to understand its relationship to the objective world described by science. More than a quarter of a century ago Francis Crick and I decided to set aside philosophical discussions on consciousness (which have engaged scholars since at least the time of Aristotle) and instead search for its physical footprints. What is it about a highly excitable piece of brain matter that gives rise to consciousness? Once we can understand that, we hope to get closer to solving the more fundamental problem.

We seek, in particular, the neuronal correlates of consciousness (NCC), defined as the minimal neuronal mechanisms jointly sufficient for any specific conscious experience. What must happen in your brain for you to experience a toothache, for example? Must some nerve cells vibrate at some magical frequency? Do some special "consciousness neurons" have to be activated? In which brain regions would these cells be located?

Neuronal Correlates of Consciousness

When defining the NCC, the qualifier "minimal" is important. The brain as a whole can be considered an NCC, after all: it generates experience, day in and day out. But the seat of consciousness can be further ring-fenced. Take the spinal cord, a foot-and-a-half-long flexible tube of nervous tissue inside the backbone with about a billion nerve cells. If the spinal cord is completely severed by trauma to the neck region, victims are paralyzed in legs, arms and torso, unable to control their bowel and bladder, and without bodily sensations. Yet these tetraplegics continue to experience life in all its variety-they see, hear, smell, feel emotions and remember as much as before the incident that radically changed their life.

Or consider the cerebellum, the "little brain" underneath the back of the brain. One of the most ancient brain circuits in evolutionary terms, it is involved in motor control, posture and gait and in the fluid execution of complex sequences of motor movements. Playing the piano, typing, ice dancing or climbing a rock wall-all these activities involve the cerebellum. It has the brain's most glorious neurons, called Purkinje cells, which possess tendrils that spread like a sea fan coral and harbor complex electrical dynamics. It also has by far the most neurons, about 69 billion (most of which are the star-shaped cerebellar granule cells), four times more than in the rest of the brain combined.

What happens to consciousness if parts of the cerebellum are lost to a stroke or to the surgeon's knife? Very little! Cerebellar patients complain of several deficits, such as the loss of fluidity of piano playing or keyboard typing but never of losing any aspect of their consciousness. They hear, see and feel fine, retain a sense of self, recall past events and continue to project themselves into the future. Even being born without a cerebellum does not appreciably affect the conscious experience of the individual.

All of the vast cerebellar apparatus is irrelevant to subjective experience. Why? Important hints can be found within its circuitry, which is exceedingly uniform and parallel (just as batteries may be connected in parallel). The cerebellum is almost exclusively a feed-forward circuit: one set of neurons feeds the next, which in turn influences a third set. There are no complex feedback loops that reverberate with electrical activity passing back and forth. (Given the time needed for a conscious perception to develop, most theoreticians infer that it must involve feedback loops within the brain's cavernous circuitry.) Moreover, the cerebellum is functionally divided into hundreds or more independent computational modules. Each one operates in parallel, with distinct, nonoverlapping inputs and output, controlling movements of different motor or cognitive systems. They scarcely interact-another feature held indispensable for consciousness.

One important lesson from the spinal cord and the cerebellum is that the genie of consciousness does not just appear when any neural tissue is excited. More is needed. This additional factor is found in the gray matter making up the celebrated cerebral cortex, the outer surface of the brain. It is a laminated sheet of intricately interconnected nervous tissue, the size and width of a 14-inch pizza. Two of these sheets, highly folded, along with their hundreds of millions of wires-the white matter-are crammed into the skull. All available evidence implicates neocortical tissue in generating feelings.

We can narrow down the seat of consciousness even further. Take, for example, experiments in which different stimuli are presented to the right and the left eyes. Suppose a picture of Donald Trump is visible only to your left eye and one of Hillary Clinton only to your right eye. We might imagine that you would see some weird superposition of Trump and Clinton. In reality, you will see Trump for a few seconds, after which he will disappear and Clinton will appear, after which she will go away and Trump will reappear. The two images will alternate in a never-ending dance because of what neuroscientists call binocular rivalry. Because your brain is getting an ambiguous input, it cannot decide: Is it Trump, or is it Clinton?

If, at the same time, you are lying inside a magnetic scanner that registers brain activity, experimenters will find that a broad set of cortical regions, collectively known as the posterior hot zone, is active. These are the parietal, occipital and temporal regions in the posterior part of cortex [ see graphic below] that play the most significant role in tracking what we see. Curiously, the primary visual cortex that receives and passes on the information streaming up from the eyes does not signal what the subject sees. A similar hierarchy of labor appears to be true of sound and touch: primary auditory and primary somatosensory cortices do not directly contribute to the content of auditory or somatosensory experience. Instead it is the next stages of processing-in the posterior hot zone-that give rise to conscious perception, including the image of Trump or Clinton.

More illuminating are two clinical sources of causal evidence: electrical stimulation of cortical tissue and the study of patients following the loss of specific regions caused by injury or disease. Before removing a brain tumor or the locus of a patient's epileptic seizures, for example, neurosurgeons map the functions of nearby cortical tissue by directly stimulating it with electrodes. Stimulating the posterior hot zone can trigger a diversity of distinct sensations and feelings. These could be flashes of light, geometric shapes, distortions of faces, auditory or visual hallucinations, a feeling of familiarity or unreality, the urge to move a specific limb, and so on. Stimulating the front of the cortex is a different matter: by and large, it elicits no direct experience.

A second source of insights are neurological patients from the first half of the 20th century. Surgeons sometimes had to excise a large belt of prefrontal cortex to remove tumors or to ameliorate epileptic seizures. What is remarkable is how unremarkable these patients appeared. The loss of a portion of the frontal lobe did have certain deleterious effects: the patients developed a lack of inhibition of inappropriate emotions or actions, motor deficits, or uncontrollable repetition of specific action or words. Following the operation, however, their personality and IQ improved, and they went on to live for many more years, with no evidence that the drastic removal of frontal tissue significantly affected their conscious experience. Conversely, removal of even small regions of the posterior cortex, where the hot zone resides, can lead to a loss of entire classes of conscious content: patients are unable to recognize faces or to see motion, color or space.

So it appears that the sights, sounds and other sensations of life as we experience it are generated by regions within the posterior cortex. As far as we can tell, almost all conscious experiences have their origin there. What is the crucial difference between these posterior regions and much of the prefrontal cortex, which does not directly contribute to subjective content? The truth is that we do not know. Even so-and excitingly-a recent finding indicates that neuroscientists may be getting closer.

The Consciousness Meter

An unmet clinical need exists for a device that reliably detects the presence or absence of consciousness in impaired or incapacitated individuals. During surgery, for example, patients are anesthetized to keep them immobile and their blood pressure stable and to eliminate pain and traumatic memories. Unfortunately, this goal is not always met: every year hundreds of patients have some awareness under anesthesia.

Another category of patients, who have severe brain injury because of accidents, infections or extreme intoxication, may live for years without being able to speak or respond to verbal requests. Establishing that they experience life is a grave challenge to the clinical arts. Think of an astronaut adrift in space, listening to mission control's attempts to contact him. His damaged radio does not relay his voice, and he appears lost to the world. This is the forlorn situation of patients whose damaged brain will not let them communicate to the world-an extreme form of solitary confinement.

In the early 2000s Giulio Tononi of the University of Wisconsin-Madison and Marcello Massimini, now at the University of Milan in Italy, pioneered a technique, called zap and zip, to probe whether someone is conscious or not. The scientists held a sheathed coil of wire against the scalp and "zapped" it-sent an intense pulse of magnetic energy into the skull-inducing a brief electric current in the neurons underneath. The perturbation, in turn, excited and inhibited the neurons' partner cells in connected regions, in a chain reverberating across the cortex, until the activity died out. A network of electroencephalogram (EEG) sensors, positioned outside the skull, recorded these electrical signals. As they unfolded over time, these traces, each corresponding to a specific location in the brain below the skull, yielded a movie.

These unfolding records neither sketched a stereotypical pattern, nor were they completely random. Remarkably, the more predictable these waxing and waning rhythms were, the more likely the brain was unconscious. The researchers quantified this intuition by compressing the data in the movie with an algorithm commonly used to "zip" computer files. The zipping yielded an estimate of the complexity of the brain's response. Volunteers who were awake turned out have a "perturbational complexity index" of between 0.31 and 0.70, dropping to below 0.31 when deeply asleep or anesthetized. Massimini and Tononi tested this zap-and-zip measure on 48 patients who were brain-injured but responsive and awake, finding that in every case, the method confirmed the behavioral evidence for consciousness.

The team then applied zap and zip to 81 patients who were minimally conscious or in a vegetative state. For the former group, which showed some signs of nonreflexive behavior, the method correctly found 36 out of 38 patients to be conscious. It misdiagnosed two patients as unconscious. Of the 43 vegetative-state patients in which all bedside attempts to establish communication failed, 34 were labeled as unconscious, but nine were not. Their brains responded similarly to those of conscious controls-implying that they were conscious yet unable to communicate with their loved ones.

Ongoing studies seek to standardize and improve zap and zip for neurological patients and to extend it to psychiatric and pediatric patients. Sooner or later scientists will discover the specific set of neural mechanisms that give rise to any one experience. Although these findings will have important clinical implications and may give succor to families and friends, they will not answer some fundamental questions: Why these neurons and not those? Why this particular frequency and not that? Indeed, the abiding mystery is how and why any highly organized piece of active matter gives rise to conscious sensation. After all, the brain is like any other organ, subject to the same physical laws as the heart or the liver. What makes it different? What is it about the biophysics of a chunk of highly excitable brain matter that turns gray goo into the glorious surround sound and Technicolor that is the fabric of everyday experience?

Ultimately what we need is a satisfying scientific theory of consciousness that predicts under which conditions any particular physical system-whether it is a complex circuit of neurons or silicon transistors-has experiences. Furthermore, why does the quality of these experiences differ? Why does a clear blue sky feel so different from the screech of a badly tuned violin? Do these differences in sensation have a function, and if so, what is it? Such a theory will allow us to infer which systems will experience anything. Absent a theory with testable predictions, any speculation about machine consciousness is based solely on our intuition, which the history of science has shown is not a reliable guide.

Fierce debates have arisen around the two most popular theories of consciousness. One is the global neuronal workspace (GNW) by psychologist Bernard J. Baars and neuroscientists Stanislas Dehaene and Jean-Pierre Changeux. The theory begins with the observation that when you are conscious of something, many different parts of your brain have access to that information. If, on the other hand, you act unconsciously, that information is localized to the specific sensory motor system involved. For example, when you type fast, you do so automatically. Asked how you do it, you would not know: you have little conscious access to that information, which also happens to be localized to the brain circuits linking your eyes to rapid finger movements.

Toward a Fundamental Theory

GNW argues that consciousness arises from a particular type of information processing-familiar from the early days of artificial intelligence, when specialized programs would access a small, shared repository of information. Whatever data were written onto this "blackboard" became available to a host of subsidiary processes: working memory, language, the planning module, and so on. According to GNW, consciousness emerges when incoming sensory information, inscribed onto such a blackboard, is broadcast globally to multiple cognitive systems-which process these data to speak, store or call up a memory or execute an action.

Because the blackboard has limited space, we can only be aware of a little information at any given instant. The network of neurons that broadcast these messages is hypothesized to be located in the frontal and parietal lobes. Once these sparse data are broadcast on this network and are globally available, the information becomes conscious. That is, the subject becomes aware of it. Whereas current machines do not yet rise to this level of cognitive sophistication, this is only a question of time. GNW posits that computers of the future will be conscious.

Integrated information theory (IIT), developed by Tononi and his collaborators, including me, has a very different starting point: experience itself. Each experience has certain essential properties. It is intrinsic, existing only for the subject as its "owner"; it is structured (a yellow cab braking while a brown dog crosses the street); and it is specific-distinct from any other conscious experience, such as a particular frame in a movie. Furthermore, it is unified and definite. When you sit on a park bench on a warm, sunny day, watching children play, the different parts of the experience-the breeze playing in your hair or the joy of hearing your toddler laugh-cannot be separated into parts without the experience ceasing to be what it is.

Tononi postulates that any complex and interconnected mechanism whose structure encodes a set of cause-and-effect relationships will have these properties-and so will have some level of consciousness. It will feel like something from the inside. But if, like the cerebellum, the mechanism lacks integration and complexity, it will not be aware of anything. As IIT states it, consciousness is intrinsic causal power associated with complex mechanisms such as the human brain.

IIT theory also derives, from the complexity of the underlying interconnected structure, a single nonnegative number Φ (pronounced " fy ") that quantifies this consciousness. If Φ is zero, the system does not feel like anything to be itself. Conversely, the bigger this number, the more intrinsic causal power the system possesses and the more conscious it is. The brain, which has enormous and highly specific connectivity, possesses very high Φ, which implies a high level of consciousness. IIT explains a number of observations, such as why the cerebellum does not contribute to consciousness and why the zap-and-zip meter works. (The quantity the meter measures is a very crude approximation of Φ.)

IIT also predicts that a sophisticated simulation of a human brain running on a digital computer cannot be conscious-even if it can speak in a manner indistinguishable from a human being. Just as simulating the massive gravitational attraction of a black hole does not actually deform spacetime around the computer implementing the astrophysical code, programming for consciousness will never create a conscious computer. Consciousness cannot be computed: it must be built into the structure of the system.

Two challenges lie ahead. One is to use the increasingly refined tools at our disposal to observe and probe the vast coalitions of highly heterogeneous neurons making up the brain to further delineate the neuronal footprints of consciousness. This effort will take decades, given the byzantine complexity of the central nervous system. The other is to verify or falsify the two, currently dominant, theories. Or, perhaps, to construct a better theory out of fragments of these two that will satisfactorily explain the central puzzle of our existence: how a three-pound organ with the consistency of tofu exudes the feeling of life.

This article is part of a special report, "The Biggest Questions in Science," sponsored by The Kavli Prize. It was produced independently by Scientific American and Nature editors, who have sole responsibility for all the editorial content.

Christof Koch is meritorious investigator at the Allen Institute in Seattle and the chief scientist at the Tiny Blue Dot Foundation in Santa Monica, as well as author of the forthcoming Then I am Myself the World - What Consciousness is and How to Expand It . He serves on Scientific American's board of advisers.

More by Christof Koch

This article was originally published with the title "What Is Consciousness?" in Scientific American Magazine Vol. 318 No. 6 (), p. 60
doi:10.1038/scientificamerican0618-60

# data[].attributes.attachment_link.description
Scientists are beginning to unravel a mystery that has long vexed philosophers

# data[].attributes.body
Of course we know consciousness is within us and comes from us, but where inside of us is it? Where are these connections and feelings that formulate into consciousness coming from? In this article from Science Alert, written by Fiona McDonald the question of where consciousness is located is attempted. Many scientists and researchers have no final definitive answer but there are many general ideas. In this article those ideas are explained.

# data[].attributes.attachment_link.readable_text
Scientists have struggled for millennia to understand human consciousness - the awareness of one's existence. Despite advances in neuroscience, we still don't really know where it comes from, and how it arises.

But researchers think they might be closer to identifying its physical origins, after a study pinpointing a network of three specific regions in the brain that appear to be crucial to consciousness.

Published in November 2016, the research could be a huge deal for our understanding of what it means to be human, and it could also help researchers find new treatments for patients in vegetative states.

"For the first time, we have found a connection between the brainstem region involved in arousal and regions involved in awareness, two prerequisites for consciousness," said lead researcher Michael Fox from the Beth Israel Deaconess Medical Centre at Harvard Medical School back in 2016.

"A lot of pieces of evidence all came together to point to this network playing a role in human consciousness."

Consciousness is generally thought of as being comprised of two critical components - arousal and awareness.

Researchers had already shown that arousal is likely regulated by the brainstem - the portion of the brain that links up with the spinal cord - seeing as it regulates when we sleep and wake, and our heart rate and breathing.

Awareness has been more elusive. Researchers have long thought that it resides somewhere in the cortex - the outer layer of the brain - but no one has been able to pinpoint where.

The Harvard team identified not only the specific brainstem region linked to arousal, but also two cortex regions, that all appear to work together to form consciousness.

To figure this out, the team analysed 36 patients in hospital with brainstem lesions - 12 of them were in a coma (unconscious) and 24 were defined as being conscious.

The researchers then mapped their brainstems to figure out if there was one particular region that could explain why some patients had maintained consciousness despite their injuries, while others had become comatose.

What they found was one small area of the brainstem - known as the rostral dorsolateral pontine tegmentum - that was significantly associated with coma. Ten out of the 12 unconscious patients had damage in this area, while just one out of the 24 conscious patients did.

That suggests that this tiny region of the brainstem is important for consciousness, but it's not the full story.

To figure out which other parts of the brain were fully connected to this region, the team looked at a brain map - or connectome - of a healthy human brain, which shows all the different connections that we know of so far in our brains (you can see a connectome in the image at the top of this story).

They identified two areas in the cortex that were linked up to the rostral dorsolateral pontine tegmentum, and were most likely to play a role in regulating consciousness. One was in the left, ventral, anterior insula (AI), and the other was in the pregenual anterior cingulate cortex (pACC).

Both of these regions have been linked by previous studies to arousal and awareness, but this is the first time they've been connected to the brainstem.

The team double-checked their work by looking at fMRI scans of 45 patients in comas or vegetative states, and showed that all of them had the network between these three regions disrupted.

It's a pretty exciting first step, but the researchers acknowledged that they need to verify their find across a larger group of patients.

Independent teams will also need to confirm their results before we can say for sure that these three regions are the physical source of consciousness in our brains.

In the meantime, the research will hopefully lead to new treatment options for patients in comas and vegetative states, who might have otherwise healthy brains but simply can't regain consciousness.

"This is most relevant if we can use these networks as a target for brain stimulation for people with disorders of consciousness," said Fox at the time.

"If we zero in on the regions and network involved, can we someday wake someone up who is in a persistent vegetative state? That's the ultimate question."

The research was published in Neurology.
A version of this article was first published in November 2016.

# data[].attributes.attachment_link.title
Harvard Scientists Think They've Pinpointed The Physical Source of Consciousness

# data[].attributes.attachment_link.description
Scientists have struggled for millennia to understand human consciousness - the awareness of one's existence. Despite advances in neuroscience, we still don't really know where it comes from, and how it arises. But researchers think they might be closer to identifying its physical origins, after a study pinpointing a network of three specific regions in the brain that appear to be crucial to consciousness.

# data[].attributes.body
We know consciousness resides in our brains, but physically where is it and where is it coming from?
In this image below from Scientific American, the area of the brain where consciousness arises from is highlighted. This image can give a visual idea of where this huge idea of consciousness we've been talking about comes from.

# data[].attributes.body
What is the reality of consciousness? What is it? How does it arise and come about? In this TED Talk by Anil Seth, these questions are answered.

# data[].attributes.attachment_link.title
Your brain hallucinates your conscious reality | Anil Seth | TED

# data[].attributes.body
When we experience something through perception, how do we know if it's real or not? Especially if we can't physically feel it or visually see it? How do we know whether anything is reality or not? In this TED Talk by Sean Tiffee, the idea of perception and the reality of it is explored much more.

# data[].attributes.attachment_link.title
Mind the Gap Between Perception and Reality | Sean Tiffee | TEDxLSCTomball

# data[].attributes.body
How do we experience things we know are there but don't actually feel or see? How does our perception of things allow us to be certain of something when we don't visually see it or physically feel it? In this Big Think video, Alva Noe describes more into detail these questions and their explanations.

# data[].attributes.body
How do we know when what we experience is our consciousness or our perception? How do we differ between the two? When do the two separately join together as one? In this short video from Science and Non Duality, Donald Hoffman gives an explanation to these ideas. These questions have never been definitively answered although Donald Hoffman gives insight into what scientists believe today.

# data[].attributes.attachment_link.title
Brain activity, perception, and conscious experience, Donald Hoffman

# data[].attributes.body
What is perception?  Perception is something so important to our every day lives. Perception is essentially how we take in, experience and feel this great big world. This is explained in much more detail by Kendra Cherry in the article below.

# data[].attributes.attachment_link.readable_text
Perception refers to our sensory experience of the world. It is the process of using our senses to become aware of objects, relationships. It is through this experience that we gain information about the environment around us.

Perception relies on the cognitive functions we use to process information, such as utilizing memory to recognize the face of a friend or detect a familiar scent. Through the perception process, we are able to both identify and respond to environmental stimuli.

Perception includes the five senses; touch, sight, sound, smell, and taste. It also includes what is known as proprioception, which is a set of senses that enable us to detect changes in body position and movement.

Many stimuli surround us at any given moment. Perception acts as a filter that allows us to exist within and interpret the world without becoming overwhelmed by this abundance of stimuli.

Types of Perception

The types of perception are often separated by the different senses. This includes visual perception, scent perception, touch perception, sound perception, and taste perception. We perceive our environment using each of these, often simultaneously.

There are also different types of perception in psychology, including:

Person perception refers to the ability to identify and use social cues about people and relationships.

Social perception is how we perceive certain societies and can be affected by things such as stereotypes and generalizations.

Another type of perception is selective perception. This involvespaying attention to some parts of our environment while ignoring others.

The different types of perception allow us to experience our environment and interact with it in ways that are both appropriate and meaningful.

How Perception Works

Through perception, we become more aware of (and can respond to) our environment. We use perception in communication to identify how our loved ones may feel. We use perception in behavior to decide what we think about individuals and groups.

We are perceiving things continuously, even though we don't typically spend a great deal of time thinking about them. For example, the light that falls on our eye's retinas transforms into a visual image unconsciously and automatically. Subtle changes in pressure against our skin, allowing us to feel objects, also occur without a single thought.

Perception Process

To better understand how we become aware of and respond to stimuli in the world around us, it can be helpful to look at the perception process. This varies somewhat for every sense.

In regard to our sense of sight, the perception process looks like this:

1. Environmental stimulus: The world is full of stimuli that can attract attention. Environmental stimulus is everything in the environment that has the potential to be perceived.

2. Attended stimulus: The attended stimulus is the specific object in the environment on which our attention is focused.

3. Image on the retina: This part of the perception process involves light passing through the cornea and pupil, onto the lens of the eye. The cornea helps focus the light as it enters and the iris controls the size of the pupils to determine how much light to let in. The cornea and lens act together to project an inverted image onto the retina.

4. Transduction: The image on the retina is then transformed into electrical signals through a process known as transduction. This allows the visual messages to be transmitted to the brain to be interpreted.

5. Neural processing: After transduction, the electrical signals undergo neural processing. The path followed by a particular signal depends on what type of signal it is (i.e. an auditory signal or a visual signal).

6. Perception: In this step of the perception process, you perceive the stimulus object in the environment. It is at this point that you become consciously aware of the stimulus.

7. Recognition: Perception doesn't just involve becoming consciously aware of the stimuli. It is also necessary for the brain to categorize and interpret what you are sensing. The ability to interpret and give meaning to the object is the next step, known as recognition.

8. Action: The action phase of the perception process involves some type of motor activity that occurs in response to the perceived stimulus. This might involve a major action, like running toward a person in distress. It can also involve doing something as subtle as blinking your eyes in response to a puff of dust blowing through the air.

Think of all the things you perceive on a daily basis. At any given moment, you might see familiar objects, feel a person's touch against your skin, smell the aroma of a home-cooked meal, or hear the sound of music playing in your neighbor's apartment. All of these help make up your conscious experience and allow you to interact with the people and objects around you.

Recap of the Perception Process

1. Environmental stimulus

2. Attended stimulus

3. Image on the retina

4. Transduction

5. Neural processing

6. Perception

7. Recognition

8. Action

Factors Influencing Perception

What makes perception somewhat complex is that we don't all perceive things the same way. One person may perceive a dog jumping on them as a threat, while another person may perceive this action as the pup just being excited to see them.

Our perceptions of people and things are shaped by our prior experiences, our interests, and how carefully we process information. This can cause one person to perceive the exact same person or situation differently than someone else.

Perception can also be affected by our personality. For instance, research has found that four of the Big 5 personality traits-openness, conscientiousness, extraversion, and neuroticism-can impact our perception of organizational justice.

Conversely, our perceptions can also affect our personality. If you perceive that your boss is treating you unfairly, for example, you may show traits related to anger or frustration. If you perceive your spouse to be loving and caring, you may show similar traits in return.

Are Perception and Attitude the Same?

While they are similar, perception and attitude are two different things. Perception is how we interpret the world around us, while our attitude (our emotions, beliefs, and behaviors) can impact these perceptions.

Tips to Improve Perception

If you want to improve your perception skills, there are some things that you can do. Actions you can take that may help you perceive more in the world around you-or at least focus on the things that are important-include:

Pay attention. Actively notice the world around you, using all your senses. What do you see, hear, taste, smell, or touch? Using your sense of proprioception, notice the movements of your arms and legs, or your changes in body position.

Make meaning of what you perceive. The recognition stage of the perception process is essential since it allows you to make sense of the world around you. Place objects in meaningful categories, so you can understand and react appropriately.

Take action. The final step of the perception process involves taking some sort of action in response to your environmental stimulus. This could involve a variety of actions, such as stopping to smell the flower you see on the side of the road, incorporating more of your senses.

Potential Pitfalls of Perception

The perception process does not always go smoothly, and there are a number of things that may interfere with our ability to interpret and respond to our environment. One is having a disorder that impacts perception.

Perceptual disorders are cognitive conditions marked by an impaired ability to perceive objects or concepts. Some disorders that may affect perception include:

Spatial neglect syndromes, which involve not attending to stimuli on one side of the body

Prosopagnosia, also called face blindness, is a disorder that makes it difficult to recognize faces

Aphantasia, a condition characterized by an inability to visualize things in your mind

Schizophrenia, which is marked by abnormal perceptions of reality

Some of these conditions may be influenced by genetics, while others result from stroke or brain injury.

Perception can also be negatively affected by certain factors. For instance, one study found that when people viewed images of others, they perceived individuals with nasal deformities as having less satisfactory personality traits. So, factors such as this can potentially affect personality perception.

History of Perception

Interest in perception dates back to the time of ancient Greek philosophers who were interested in how people know the world and gain understanding. As psychology emerged as a science separate from philosophy, researchers became interested in understanding how different aspects of perception worked-particularly, the perception of color.

In addition to understanding basic physiological processes, psychologists were also interested in understanding how the mind interprets and organizes these perceptions.

Gestalt psychologists proposed a holistic approach, suggesting that the sum equals more than the sum of its parts. Cognitive psychologists have also worked to understand how motivations and expectations can play a role in the process of perception.

As time progresses, researchers continue to investigate perception on the neural level. They also look at how injury, conditions, and substances might affect perception.

Additional Reading

# data[].attributes.attachment_link.description
Learn about perception in psychology and the process we use to recognize and respond to our environment. We also share types of perception and how to improve yours.

# data[].attributes.body
Perception sometimes is overlooked, many people don't realize that our senses such as taste, smell, hearing, touch, and sight are all very large aspects of creating perception. In order for perception to be what it is, many factors such as the senses must be involved. In this article from The Neural Bases of Multisensory Processes, written by Mirjam Keetels and Jean Vroomen the idea of the connection between perception and the human senses is explained.

# data[].attributes.attachment_link.readable_text
9.1. INTRODUCTION

Most of our real-world perceptual experiences are specified by synchronous redundant and/or complementary multisensory perceptual attributes. As an example, a talker can be heard and seen at the same time, and as a result, we typically have access to multiple features across the different senses (i.e., lip movements, facial expression, pitch, speed, and temporal structure of the speech sound). This is highly advantageous because it increases perceptual reliability and saliency and, as a result, it might enhance learning, discrimination, or the speed of a reaction to the stimulus ( Sumby and Pollack 1954; Summerfield 1987). However, the multisensory nature of perception also raises the question about how the different sense organs cooperate so as to form a coherent representation of the world. In recent years, this has been the focus of much behavioral and neuroscientific research ( Calvert et al. 2004). The most commonly held view among researchers in multisensory perception is what has been referred to as the "assumption of unity." It states that, as information from different modalities share more (amodal) properties, the more likely the brain will treat them as originating from a common object or source (see, e.g., Bedford 1989; Bertelson 1999; Radeau 1994; Stein and Meredith 1993; Welch 1999; Welch and Warren 1980). Without a doubt, the most important amodal property is temporal coincidence (e.g., Radeau 1994). From this perspective, one expects intersensory interactions to occur if, and only if, information from the different sense organs arrives at around the same time in the brain; otherwise, two separate events are perceived rather than a single multimodal one.

The perception of time and, in particular, synchrony between the senses is not straightforward because there is no dedicated sense organ that registers time in an absolute scale. Moreover, to perceive synchrony, the brain has to deal with differences in physical (outside the body) and neural (inside the body) transmission times. Sounds, for example, travel through air much slower than visual information does (i.e., 300,000,000 m/s for vision vs. 330 m/s for audition), whereas no physical transmission time through air is involved for tactile stimulation as it is presented directly at the body surface. The neural processing time also differs between the senses, and it is typically slower for visual than it is for auditory stimuli (approximately 50 vs. 10 ms, respectively), whereas for touch, the brain may have to take into account where the stimulation originated from as the traveling time from the toes to the brain is longer than from the nose (the typical conduction velocity is 55 m/s, which results in a ∼30 ms difference between toe and nose when this distance is 1.60 m; Macefield et al. 1989). Because of these differences, one might expect that for audiovisual events, only those occurring at the so-called "horizon of simultaneity" ( Poppel 1985; Poppel et al. 1990)-a distance of approximately 10 to 15 m from the observer-will result in the approximate synchronous arrival of auditory and visual information at the primary sensory cortices. Sounds will arrive before visual stimuli if the audiovisual event is within 15 m from the observer, whereas vision will arrive before sounds for events farther away. Although surprisingly, despite these naturally occurring lags, observers perceive intersensory synchrony for most multisensory events in the external world, and not only for those at 15 m.

In recent years, a substantial amount of research has been devoted to understanding how the brain handles these timing differences ( Calvert et al. 2004; King 2005; Levitin et al. 2000; Spence and Driver 2004; Spence and Squire 2003). Here, we review several key issues about intersensory timing. We start with a short overview of how intersensory timing is generally measured, and then discuss several factors that affect the point of subjective simultaneity and sensitivity. In the sections that follow, we address several ways in which the brain might deal with naturally occurring lags between the senses.

9.2. MEASURING INTERSENSORY SYNCHRONY: TEMPORAL ORDER JUDGMENT TASK AND SIMULTANEITY JUDGMENT TASK

Before examining some of the basic findings, we first devote a few words on how intersensory synchrony is usually measured. There are two classic tasks that have been used most of the time in the literature. In both tasks, observers are asked to judge-in a direct way-the relative timing of two stimuli from different modalities: the temporal order judgment (TOJ) task and the simultaneity judgment (SJ) task. In the TOJ task, stimuli are presented in different modalities at various stimulus onset asynchronies (SOA; Dixon and Spitz 1980; Hirsh and Sherrick 1961; Sternberg and Knoll 1973), and observers may judge which stimulus came first or which came second. In an audiovisual TOJ task, participants may thus respond with "sound-first" or "light-first." If the percentage of "sound-first" responses is plotted as a function of the SOA, one usually obtains an S-shaped logistic psychometric curve. From this curve, one can derive two measures: the 50% crossover point, and the steepness of the curve at the 50% point. The 50% crossover point is the SOA at which observers were-presumably-maximally unsure about temporal order. In general, this is called the "point of subjective simultaneity" (PSS) and it is assumed that at this SOA, the information from the different modalities is perceived as being maximally simultaneous. The second measure-the steepness at the crossover point-reflects the observers' sensitivity to temporal asynchronies. The steepness can also be expressed in terms of the just noticeable difference (JND; half the difference in SOA between the 25% and 75% point), and it represents the smallest interval observers can reliably notice. A steep psychometric curve thus implies a small JND, and sensitivity is thus good as observers are able to detect small asynchronies (see Figure 9.1).

The second task that has been used often is the SJ task. Here, stimuli are also presented at various SOAs, but rather than judging which stimulus came first, observers now judge whether the stimuli were presented simultaneously or not. In the SJ task, one usually obtains a bell-shaped Gaussian curve if the percentage of "simultaneous" responses is plotted as a function of the SOA. For the audiovisual case, the raw data are usually not mirror-symmetric, but skewed toward more "simultaneous" responses on the "light-first" side of the axis. Once a curve is fitted on the raw data, one can, as in the TOJ task, derive the PSS and the JND: the peak of the bell shape corresponds to the PSS, and the width of the bell shape corresponds to the JND.

The TOJ and SJ tasks have, in general, been used more or less interchangeably, despite the fact that comparative studies have found differences in performance measures derived from both tasks. Possibly, it reflects that judgments about simultaneity and temporal order are based on different sources of information ( Hirsh and Fraisse 1964; Mitrani et al. 1986; Schneider and Bavelier 2003; Zampini et al. 2003a). As an example, van Eijk et al. (2008) examined task effects on the PSS. They presented observers a sound and light, or a bouncing ball and an impact sound at various SOAs, and had them perform three tasks: an audiovisual TOJ task ("sound-first" or "light-first" responses required), an SJ task with two response categories (SJ2; "synchronous" or "asynchronous" responses required), and an SJ task with three response categories (SJ3; "sound-first," "synchronous," or "light-first" responses required). Results from both stimulus types showed that the individual PSS values for the two SJ tasks correlated well, but there was no correlation between the TOJ and SJ tasks. This made the authors conclude, arguably, that the SJ task should be preferred over the TOJ task if one wants to measure perception of audiovisual synchrony.

In our view, there is no straightforward solution about how to measure the PSS or JND for intersensory timing because the tasks are subject to different kinds of response biases (see Schneider and Bavelier 2003; Van Eijk et al. 2008; Vatakis et al. 2007, 2008b for discussion). In the TOJ task, in which only temporal order responses can be given ("sound-first" or "light-first"), observers may be inclined to adopt the assumption that stimuli are never simultaneous, which thus may result in rather low JNDs. On the other hand, in the SJ task, observers may be inclined to assume that stimuli actually belong together because the "synchronous" response category is available. Depending on criterion settings, this may result in many "synchronous" responses, and thus, a wide bell-shaped curve which will lead to the invalid conclusion that sensitivity is poor.

In practice, both the SJ and TOJ task will have their limits. The SJ2 task suffers heavily from the fact that observers have to adopt a criterion about what counts as "simultaneous/nonsimultaneous." And in the SJ3 task, the participant has to dissociate sound-first stimuli from synchronous ones, and light-first stimuli from synchronous ones. Hence, in the SJ3 task there are two criteria: a "sound-first/ simultaneous" criterion, and a "light-first/simultaneous" criterion. If observers change, for whatever reason, their criterion (or criteria) along the experiment or between experimental manipulations, it changes the width of the curve and the corresponding JND. If sensitivity is the critical measure, one should thus be careful using the SJ task because JNDs depend heavily on these criterion settings.

A different critique can be applied to the TOJ task. Here, the assumption is made that observers respond at about 50% for each of the two response alternatives when maximally unsure about temporal order. Although in practice, participants may adopt a different strategy and respond, for example, "sound-first" (and others may, for arbitrary reasons, respond "light-first") whenever unsure about temporal order. Such a response bias will shift the derived 50% point toward one side of the continuum or the other, and the 50% point will then not be a good measure of the PSS, the point at which simultaneity is supposed to be maximal. If performance of an individual observer on an SJ task is compared with a TOJ task, it should thus not come as too big of a surprise that the PSS and JND derived from both tasks do not converge.

9.3. POINT OF SUBJECTIVE SIMULTANEITY

The naïve reader might think that stimuli from different modalities are perceived as being maximally simultaneous if they are presented the way nature does, that is, synchronous, so at 0 ms SOA. Although surprisingly, most of the time, this is not the case. For audiovisual stimuli, the PSS is usually shifted toward a visual-lead stimulus, so perceived simultaneity is maximal if vision comes slightly before sounds (e.g., Kayser et al. 2008; Lewald and Guski 2003; Lewkowicz 1996; Slutsky and Recanzone 2001; Zampini et al. 2003a, 2005b, 2005c). This bias was found in a classic study by Dixon and Spitz (1980). Here, participants monitored continuous videos consisting of an audiovisual speech stream or an object event consisting of a hammer hitting a peg. The videos started off in synchrony and were then gradually desynchronized at a constant rate of 51 ms/s up to a maximum asynchrony of 500 ms. Observers were instructed to respond as soon as they noticed the asynchrony. They were better at detecting the audiovisual asynchrony if the sound preceded the video rather than if the video preceded the sound (131 vs. 258 ms thresholds for speech, and 75 vs. 188 ms thresholds for the hammer, respectively). PSS values also pointed in the same direction as simultaneity was maximal when the video preceded the audio by 120 ms for speech, and by 103 ms for the hammer. Many other studies have reported this vision-first PSS ( Dinnerstein and Zlotogura 1968; Hirsh and Fraisse 1964; Jaskowski et al. 1990; Keetels and Vroomen 2005; Spence et al. 2003; Vatakis and Spence 2006a; Zampini et al. 2003a), although some also reported opposite results ( Bald et al. 1942; Rutschmann and Link 1964; Teatini et al. 1976; Vroomen et al. 2004). There have been many speculations about the underlying reason for this overall visual-lead asymmetry, the main one being that observers are tuned toward the natural situation in which lights arrive before sounds on the sense organs ( King and Palmer 1985). There will then be a preference for vision to have a head start over sound so as to be perceived as simultaneous.

Besides this possibility, though, there are many other reasons why the PSS can differ quite substantially from 0 ms SOA. To point out just a few: the PSS depends, among others, on stimulus intensity (more intense stimuli are processed faster or come to consciousness more quickly ( Jaskowski 1999; Neumann and Niepel 2004; Roefs 1963; Sanford 1971; Smith 1933), stimulus duration ( Boenke et al. 2009), the nature of the response that participants have to make (e.g., "Which stimulus came first?" vs. "Which stimulus came second?"; see Frey 1990; Shore et al. 2001), individual differences ( Boenke et al. 2009; Mollon and Perkins 1996; Stone et al. 2001), and the modality to which attention is directed ( Mattes and Ulrich 1998; Schneider and Bavelier 2003; Shore et al. 2001, 2005; Stelmach and Herdman 1991; Zampini et al. 2005c). We do not intend to list all the factors known thus far, but we only pick out the one that has been particularly important in theorizing about perception in general, that is, the role of attention.

A vexing issue in experimental psychology is the idea that attention speeds up sensory processing. Titchener (1908) termed it the "law of prior entry," implying that attended objects come to consciousness more quickly than unattended ones. Many of the old studies on prior entry suffered from the fact that they might simply reflect response biases (see Schneider and Bavelier 2003; Shore et al. 2001; Spence et al. 2001; Zampini et al. 2005c for discussions on the role of response bias in prior entry). As an example, observers may, whenever unsure, just respond that the attended stimulus was presented first without really having that impression. This strategy would reflect a change in decision criterion rather than a low-level sensory interaction between attention and the attended target stimulus. To disentangle response biases from truly perceptual effects, Spence et al. (2001) performed a series of important TOJ experiments in which visual-tactile, visual-visual, or tactile-tactile stimulus pairs were presented from the left or right of fixation. The focus of attention was directed toward either the visual or tactile modality by varying the probability of each stimulus modality (e.g., in the attend-touch condition, there were 50% tactile-tactile pairs, 0% visual-visual, and 50% critical tactile-visual pairs). Participants had to indicate whether the left or right stimulus was presented first. The idea tested was that attention to one sensory modality would speed up perception of stimuli in that modality, thus resulting in a change of the PSS (see also Mattes and Ulrich 1998; Schneider and Bavelier 2003; Shore et al. 2001, 2005; Stelmach and Herdman 1991; Zampini et al. 2005c). Their results indeed supported this notion: when attention was directed to touch, visual stimuli had to lead by much greater intervals (155 ms) than when attention was directed to vision (22 ms) for them to be perceived as simultaneous. Additional experiments demonstrated that attending to one side (left or right) also speeded perception of stimuli presented at that side. Therefore, both spatial attention and attention to modality were effective in shifting the PSS, presumably because they speeded up perceptual processes. To minimize the contribution of any simple response bias on the PSS, Spence et al. (2001) performed these experiments in which attention was manipulated in a dimension (modality or side) that was orthogonal to that of responding (side or modality, respectively). Thus, while attending to vision or touch, participants had to judge which side came first; and while attending to the left or right, participants judged which modality came first. The authors reported similar shifts of the PSS in these different tasks, thus favoring a perceptual basis for prior entry.

Besides such behavioral data, there is also extensive electrophysiological support for the idea that attention affects perceptual processing. Very briefly, in the electroencephalogram (EEG) one can measure the event-related response (ERP) of stimuli that were either attended or unattended. Naively speaking, if attention speeds up stimulus processing, one would expect ERPs of attended stimuli to be faster than unattended ones. In a seminal study by Hillyard and Munte (1984), participants were presented a stream of brief flashes and tones on the left or right of fixation. The participant's task was to attend either the auditory or visual modality, and to respond to infrequent targets in that modality at an attended location (e.g., respond to a slightly longer tone on the left). The attended modality was constant during the experiment (but varied between subjects), and the relevant location was specified at the beginning of each block of trials. The authors found enhanced negativity in the ERP for stimuli at attended locations if compared to nonattended locations. The negativity started at about 150 ms poststimulus for visual stimuli and at about 100 ms for auditory stimuli. Evidence for a cross-modal link in spatial attention was also found, as the enhancement (although smaller) was also found for stimuli at the attended location in the unattended modality (see also Spence and Driver 1996; Spence et al. 2000 for behavioral results). Since then, analogous results have been found by many others. For example, Eimer and Schröger (1998) found similar results using a different design in which the side of the attended location varied from trial to trial. Again, their results demonstrated enhanced negativities (between 160 and 280 ms after stimulus onset) for attended locations as compared to unattended locations, and the effect was again bigger for the relevant rather than irrelevant modality.

The critical issue for the idea prior entry is whether these ERP effects also reflect that attended stimuli are processed faster. In most EEG studies, attention affects the amplitude of the ERP rather than speed (for a review, see Eimer and Driver 2001). The problem is that there are many other interpretations for an amplitude modulation rather than increased processing speed (e.g., less smearing of the EEG signal over trials if attended). A shift in the latencies of the ERP would have been easier to interpret in terms of increased processing speed, but the problem is that even if a latency shift in the ERP is obtained, it is usually small if compared to the behavioral data. As an example, in an ERP study by Vibell et al. (2007), attention was directed toward the visual or tactile modality in a visual-tactile TOJ task. Results showed that the peak latency of the visual evoked potentials (P1 and N1) was earlier when attention was directed to vision (P1 = 147 ms, and N1 = 198 ms) rather than when directed to touch (P1 = 151 ms, and N1 = 201 ms). This shift in the P1 may be taken as evidence that attention indeed speeds up perception in the attended modality, but it should also be noted that the 4-ms shift in the ERP is in a quite different order of magnitude than the 38 ms shift of the PSS in the behavioral data, or the 133 ms shift reported by Spence et al. (2001) in a similar study. In conclusion, there is both behavioral and electrophysiological support for the idea that attention speeds up perceptual processing, but the underlying neural mechanisms remain, for the time being, elusive.

9.4. SENSITIVITY FOR INTERSENSORY ASYNCHRONY

Besides the point at which simultaneity is perceived to be maximal (the PSS), the second measure that one can derive from the TOJ and SJ task-but which is unfortunately not always reported-is the observers' sensitivity to timing differences, the JND. The sensitivity to intersensory timing differences is not only of interest for theoretical reasons, but it is also of practical importance, for example, in video broadcasting or multimedia Internet where standards are required for allowable audio or video delays ( Finger and Davis 2001; Mortlock et al. 1997; Rihs 1995). One of the classic studies on sensitivity for intersensory synchrony was done by Hirsh and Sherrick (1961). They presented audio-visual, visual-tactile, and audio-tactile stimuli in a TOJ task and reported JNDs to be approximately 20 ms regardless of the modalities used. Although more recent studies have found substantially bigger JNDs and larger differences between the sensory modalities. For simple cross-modal stimuli such as auditory beeps and visual flashes, JNDs have been reported in the order of approximately 25 to 50 ms ( Keetels and Vroomen 2005; Zampini et al. 2003a, 2005b), but for audio-tactile pairs, Zampini et al. (2005a) obtained JNDs of about 80 ms, and for visual-tactile pairs, JNDs have been found in the order of 35 to 65 ms ( Keetels and Vroomen 2008b; Spence et al. 2001). More importantly, JNDs are not constant, but have been shown to depend on various other factors like the spatial separation between the components of the stimuli, stimulus complexity, whether it is speech or not, and-more controversial-the semantic congruency. Some of these factors will be described below.

A factor that has been shown to affect sensitivity for intersensory timing is the spatial separation between the components of a stimulus pair. Typically, sensitivity for temporal order improves if the components of the cross-modal stimuli are spatially separated (i.e., lower JNDs; Bertelson and Aschersleben 2003; Spence et al. 2003; Zampini et al. 2003a, 2003b, 2005b). Bertelson and Aschersleben, for example, reported audiovisual JNDs to be lower when a beep and a flash were presented from different locations rather than from a common and central location. Zampini et al. (2003b) qualified these findings and observed that sensitivity in an audiovisual TOJ task improved if the sounds and lights were presented from different locations, but only so if presented at the left and right from the median (at 24°). No effect of separation was found for vertically separated stimuli. This made Zampini et al. conclude that the critical factor for the TOJ improvement was that the individual components of an audiovisual stimulus were presented in different hemifields. Keetels and Vroomen (2005), though, examined this notion and varied the (horizontal) size of the spatial disparity. Their results showed that JNDs also improved when spatial disparity was large rather than small, even if stimuli did not cross hemifields. Audiovisual JNDs thus depend on both the relative position from which stimuli are presented and on whether hemifields are crossed or not. Spence et al. (2001) further demonstrated that sensitivity improves for spatially separated visual-tactile stimulus pairs, although no such effect was found for audio-tactile pairs ( Zampini et al. 2005a). In blind people, on the other hand, audio-tactile temporal sensitivity was found to be affected by spatial separation ( Occelli et al. 2008) and similar spatial modulation effects were demonstrated in rear space ( Kitagawa 2005).

What is the underlying reason that sensitivity to temporal differences improves if the sources are spatially separated? Or, why does the brain fail to notice temporal intervals when stimuli comes from a single location? Two accounts have been proposed ( Spence et al. 2003). First, it has been suggested that intersensory pairing impairs sensitivity for temporal order. The idea underlying "intersensory pairing" is that the brain has a list of criteria on which it decides whether information from different modalities belong together or not. Commonality in time is, without a doubt, a very important criterion, but there may be others like commonality in space, association based on cooccurrence, or semantic congruency. Stimuli from the same location may, for this reason, be more likely paired into a single multimodal event if compared to stimuli presented far apart (see Radeau 1994). Any such tendency to pair stimuli could then make the relative temporal order of the components lost, thereby worsening the temporal sensitivity in TOJ or SJ tasks.

In contrast with this notion, many cross-modal effects occur despite spatial discordance, and there are reasons to argue that spatial congruency may not be an important criterion for intersensory pairing ( Bertelson 1994; Colin et al. 2001; Jones and Munhall 1997; Keetels et al. 2007; Keetels and Vroomen 2007, 2008a; Stein et al. 1996; Teder-Salejarvi et al. 2005; Vroomen and Keetels 2006). But why, then, does sensitivity for temporal order improve with spatially separated stimuli if not because intersensory pairing is impeded? A second reason why JNDs may improve is that of spatial redundancy. Whenever multisensory information is presented from different locations, observers actually have extra spatial information on which to base their response. That is, observers may initially not know which modality had been presented first, but still know on which side the first stimulus appeared, and they may then infer which modality had been presented first. As an example, in an audiovisual TOJ task, an observer may have noticed that the first stimulus came from the left (possibly because attention was captured by the first stimulus toward that side). They may also remember that the light was presented on the right. By inference, then, the sound must have been presented first. Sensitivity for temporal order for spatially separated stimuli then improves because there are extra spatial cues that are not present for colocated stimuli.

9.4.2. Stimulus Complexity Affects JND

Many studies exploring temporal sensitivity have used relatively simple stimuli such as flashes and beeps that have a single and rather sharp transient onset. However, in real-world situations, the brain has to deal with much more complex stimuli that often have complicated variations in temporal structure over time (e.g., seeing and hearing someone speaking; or seeing, hearing, and touching the keys on a computer keyboard). How does the brain notice timing differences between these more complicated and dynamic stimuli? Theoretically, one might expect that more complex stimuli also provide a richer base on which to judge temporal order. Audiovisual speech would be the example "par excellence" because it is rich in content and fluctuating over time. Although in fact, several studies have found the opposite, and in particular for audiovisual speech, the "temporal window" for which the auditory and visual streams are perceived as synchronous is rather wide ( Conrey and Pisoni 2006; Dixon and Spitz 1980; Jones and Jarick 2006; Stekelenburg and Vroomen 2007; a series of studies by Vatakis and Spence 2006a; Vatakis, Ghanzanfar and Spence 2008a; van Wassenhove et al. 2007). For example, in a study by van Wassenhove et al. (2007), observers judged in an SJ task whether congruent audiovisual speech stimuli and incongruent McGurk-like speech stimuli ( McGurk and MacDonald 1976) were synchronous or not. The authors found a temporal window of 203 ms for the congruent pairs (ranging from −76 ms sound-first to +127 ms vision-first, with PSS at 26 ms vision-first) and a 159 ms window for the incongruent pairs (ranging from −40 to +119 ms, with PSS at 40 ms vision-first). These windows are rather wide if compared to the much smaller windows found for simple flashes and beeps (mostly below 50 ms; Hirsh and Sherrick 1961; Keetels and Vroomen 2005; Zampini et al. 2003a, 2005b). The relatively wide temporal window for complex stimuli has also been demonstrated by indirect tests. For example, the McGurk effect was found to diminish if the auditory and visual information streams are out of sync, but this only occurred at rather long intervals (comparable with the ones found in SJ tasks; Grant et al. 2004; Massaro et al. 1996; McGrath and Summerfield 1985; Munhall et al. 1996; Pandey et al. 1986; Tanaka et al. 2009b; van Wassenhove et al. 2007).

There have been several recent attempts to compare sensitivity for intersensory timing in audiovisual speech with other audiovisual events such as music (guitar and piano) and object actions (e.g., smashing a television set with a hammer, or hitting a soda can with a block of wood; Vatakis and Spence 2006a, 2006b). Observers made TOJs about which stream (auditory or visual) appeared first. Overall, results showed better temporal sensitivity for audiovisual stimuli of "lower complexity" in comparison with stimuli having continuously varying properties (i.e., syllables vs. words and/or sentences). Similar findings were reported by Stekelenburg and Vroomen (2007), who compared JNDs of audiovisual speech (pronunciation of the syllable /bi/) with that of natural nonspeech events (a video of a handclap) in a TOJ task. Again, JNDs were much better for the nonspeech events (64 ms) than for speech (105 ms).

On the basis of these findings, some have concluded that "speech is special" ( van Wassenhove et al. 2007; Vatakis et al. 2008a) or that when "stimulus complexity" increases, sensitivity for temporal order deteriorates ( Vatakis and Spence 2006a). Although in our view, these proposals do not really clarify the issue because the notion of "speech is special" and "stimulus complexity" are both ill-defined, and most likely, these concepts are confounded with other stimulus factors that can be described more clearly. As an example, it is known that the rate at which stimuli are presented affects audiovisual JNDs for intersensory timing ( Benjamins et al. 2008; Fujisaki and Nishida 2005). Sensitivity may also be affected by whether there is anticipatory information that predicts the onset of an audiovisual event ( Stekelenburg and Vroomen 2007; Van Eijk 2008; Vroomen and Stekelenburg 2009), and by whether there is a sharp transition that can serve as a temporal anchor ( Fujisaki and Nishida 2005). Each of these stimulus characteristics-and likely many others-need to be controlled if one wants to compare across stimuli in a nonarbitrary way. Below, we address some of these factors.

It has been demonstrated that perception of intersensory synchrony breaks down if stimuli are presented with a temporal frequency of above ∼4Hz. This is very slow if compared to unimodal visual or auditory sensitivity for temporal coherence. Fujisaki and Nishida (2005) examined this using audiovisual stimuli consisting of a luminance-modulated Gaussian blob and an amplitude-modulated white noise presented at various rates. They demonstrated that synchrony-asynchrony discrimination for temporally dense random pulse trains became nearly impossible at temporal frequencies larger than 4 Hz, even when the audiovisual interval was large enough for discrimination of single pulses (the discrimination thresholds were 75, 81, and 119 ms for single pulses, 2 and 4 Hz repetitive stimuli, respectively). This 4-Hz boundary was also reported by Benjamins et al. (2008). They explored the temporal limit of audiovisual integration using a visual stimulus that alternated in color (red or green) and a sound that alternated in frequency (high or low). Observers had to indicate which sound (high or low) accompanied the red disk. Their results demonstrated that at rates of 4.2 Hz and higher, observers were no longer able to match the visual and auditory stimuli across modalities (proportion correct matches dropped from 0.9 at 1.9 Hz to 0.5 at a 4.2 Hz). Further experiments also demonstrated that manipulating other temporal stimulus characteristics such as the stimulus offsets and/or audiovisual SOAs did not change the 4-Hz threshold. Here, it should be mentioned that the 4-Hz rate is also the approximate rate with which syllables are spoken in continuous speech, and temporal order in audiovisual speech might thus be difficult simply because stimulus presentation is too fast, and not because speech is special.

9.4.4. Predictability Affects JND

Another factor that may play a role in intersensory synchrony judgments, but one that has not yet been studied extensively, is the extent to which (one of the components of) a multisensory event can be predicted. As an example, for many natural events-such as the clapping of hands-vision provides predictive information about when a sound is to occur, as there is visual anticipatory information about sound onset. Stimuli with predictive information allow observers to make a clear prediction about when a sound is to occur, and this might improve sensitivity for temporal order. A study by van Eijk et al. (2008, Chapter 4) is of relevance here. They explored the effect of visual predictive information (or, the way the authors called it, "apparent causality") on perceived audiovisual synchrony. Visual predictive information was either present or absent by showing all or part of a Newton's cradle toy (i.e., a ball that appears to fall from a suspended position on the left of the display, strikes the leftmost of four contiguous balls, and then launches the rightmost ball into an arc motion away from the other balls). The collision of the balls was accompanied by a sound that varied around the time of the impact. The predictability of the sound was varied by showing either the left side of the display (motion followed by a collision and sound so that visual motion predicted sound occurrence) or the right side of the display (a sound followed by visual motion; so no predictable information about sound onset). In line with the argument made here, the authors reported better temporal sensitivity if visual predictive information about sound onset was available (the left display) rather than if it was absent (the right display).

A more controversial issue in the literature on intersensory timing is the extent to which information from different modalities is treated by the brain as belonging to the same event. Some have headed it under the already mentioned notion of "intersensory pairing," others under the "unity assumption" ( Welch and Warren 1980). The idea is that observers find it difficult to judge temporal order if the information streams naturally belong together, for reasons other than temporal coincidence, because there is then more intersensory integration; in which case, temporal order is lost. Several studies have examined this issue but with varying outcomes. In a study by Vatakis and Spence (2007), participants judged the temporal order of audiovisual speech stimuli that varied in gender and phonemic congruency. Face and voice congruency could vary in gender (a female face articulating /pi/ with a sound of either a female or male /pi/), or phonemic content (a face saying /ba/ with a voice saying /ba/ or /da/). In support of the unity assumption, results showed that for both the gender and phonemic congruency manipulation, sensitivity for temporal order improved if the auditory and visual streams were incongruent rather than congruent. In a recent study, Vatakis et al. (2008a) qualified these findings and reported that this effect may be specific for human speech. In this study, the effect of congruency was examined using matching or mismatching call types of monkeys ("cooing" vs. "grunt" or threat calls). For audiovisual speech, the sensitivity of temporal order was again better for the incongruent rather than congruent trials, but there was no congruency effect for the monkey calls. In another study, Vatakis and Spence (2008) also found no congruency effect for audiovisual music and object events that either matched (e.g., the sight of a note being played on a piano together with the corresponding sound, or the video of a bouncing ball with a corresponding sound) or mismatched. At this stage, it therefore appears that the "unity assumption" may only apply to audiovisual speech. It leaves one to wonder, though, whether this effect is best explained in terms of the "special" nature of audiovisual speech, or whether other factors are at play (e.g., the high level of exposure to speech stimuli in daily life, the possibly more attention-grabbing nature of speech stimuli, or the specific low-level acoustic stimulus features of speech; Vatakis et al. 2008a).

9.5. HOW THE BRAIN DEALS WITH LAGS BETWEEN THE SENSES

In any multisensory environment, the brain has to deal with lags in arrival and processing time between the different senses. Surprisingly though, despite these lags, temporal coherence is usually maintained, and only in exceptional circumstances such as the thunder, which is heard after the lightning, a single multisensory event is perceived as being separated. This raises the question of how temporal coherence is maintained. In our view, at least four options are available: (1) the brain might be insensitive for small lags, or it could just ignore them (a window of temporal integration); (2) the brain might be "intelligent" and bring deeply rooted knowledge about the external world into play that allows it to compensate for various external factors; (3) the brain might be flexible and shift its criterion about synchrony in an adaptive fashion (recalibration); or (4) the brain might actively shift the time at which one information stream is perceived to occur toward the other (temporal ventriloquism). Below, we discuss each of these notions. It should be noted beforehand that none of these options mutually excludes the other.

9.5.1. Window of Temporal Integration

The first notion, that the brain is rather insensitive to lags, comes close to the idea that there is a "window of temporal integration." Any information that falls within this hypothetical window is potentially assigned to the same external event and streams within the window are then treated as to have occurred simultaneously (see Figure 9.2, panel 1). Many have alluded to this concept, but what is less satisfying about it is that it is basically a description rather than an explanation. To make this point clear, some have reported that the temporal window for audiovisual speech can be quite large because it can range from approximately 40 ms audio-first to 240 ms vision-first. However, sensitivity for intersensory asynchronies (JND) is usually much smaller than the size of this window. For example, Munhall et al. (1996) demonstrated that exact temporal coincidence between the auditory and visual parts of audiovisual speech stimuli is not a very strict constraint on the McGurk effect ( McGurk and MacDonald 1976). Their results demonstrated that the McGurk effect was biggest when vowels were synchronized (see also McGrath and Summerfield 1985), but the effect survived even if audition lagged vision by 180 ms (see also Soto-Faraco and Alsius 2007, 2009; these studies show that participants can still perceive a McGurk effect when they can quite reliably perform TOJs). Outside the speech domain, similar findings have been reported. In a study by Shimojo et al. (2001), the role of temporal synchrony was examined using the streaming-bouncing illusion (i.e., two identical visual targets that move across each other and are normally perceived as a streaming motion are typically perceived to bounce when a brief sound is presented at the moment that the visual targets coincide; Sekuler et al. 1997). The phenomenon is dependent on the timing of the sound relative to the coincidence of the moving objects. Although it has been demonstrated that a brief sound induced the visual bouncing percept most effectively when it was presented about 50 ms before the moving objects coincide, their data furthermore showed a rather large temporal window of integration because intervals ranging from 250 ms before visual coincidence to 150 ms after coincidence still induced the bouncing percept (see also Bertelson and Aschersleben 1998, for the effect of temporal asynchrony on spatial ventriloquism; or Shams et al. 2002, for the illusory-flash effect). All these intersensory effects thus occur at asynchronies that are much larger than JNDs normally reported when directly exploring the effect of asynchrony using TOJ or SJ tasks ( van Wassenhove et al. 2007). One might argue that despite the fact that observers do notice small delays between the senses, the brain can still ignore it if it is of help for other purposes, such as understanding speech ( Soto-Faraco and Alsius 2007, 2009). But the question then becomes, why is there more than one window; that is, one for understanding, the other for noticing timing differences.

Besides the width of the temporal window varying with the purpose of the task, it has also been found to vary for different kinds of stimuli. As already mentioned, the temporal window is much smaller for clicks and flashes than it is for audiovisual speech. However, why would the size be different for different stimuli? Does the brain have a separate window for each stimulus and each purpose? If so, we are left with explaining how and why it varies. Some have taken the concept of a window quite literally, and have argued that "speech is special" because the window for audiovisual speech is wide ( van Wassenhove et al. 2007; Vatakis et al. 2008a). Although we would rather refrain from such speculations, and consider it more useful to examine what the critical features are that determine when perception of simultaneity becomes easy (a small window) or difficult (a large window). The size of the window is thus, in our view, the factor that needs to be explained rather than that it is the explanation itself.

9.5.2. Compensation for External Factors

Sugita and Suzuki (2003) explored compensation for distance with an audiovisual TOJ task. The visual stimuli were delivered by light-emitting diodes (LEDs) at distances ranging from 1 to 50 m in free-field circumstances (and were compensated for by intensity, although not size). Of importance, the sounds were delivered through headphones, and no attempt was made to equate the distance of the sound with that of the light. Note that this, in essence, undermines the whole idea that the brain compensates for lags of audiovisual events out in space. Nevertheless, PSS values were found to shift with visual stimulus distance. When the visual stimulus was 1 m away, the PSS was at about a ∼5 ms sound delay, and the delay increased when the LEDs were farther away. The increment was consistent with the velocity of sounds up to a viewing distance of about 10 m, after which it leveled off. This led the authors to conclude that lags between auditory and visual inputs are perceived as synchronous not because the brain has a wide temporal window for audiovisual integration, but because the brain actively changes the temporal location of the window depending on the distance of the source.

Alais and Carlile (2005) came to similar conclusions, but with different stimuli. In their study, auditory stimuli were presented over a loudspeaker and auditory distance was simulated by varying the direct-to-reverberant energy ratio as a depth cue for sounds ( Bronkhorst 1995; Bronkhorst and Houtgast 1999). The near sounds simulated a depth of 5 m and had substantial amounts of direct energy with a sharp transient onset; the far sounds simulated a depth of 40 m and did not have a transient. The visual stimulus was a Gaussian blob on a computer screen in front of the observer without variations in the distance. Note that, again, no attempt was made to equate auditory and visual distance, thus again undermining the underlying notion. The effect of apparent auditory distance on temporal alignment with the blob on the screen was measured in a TOJ task. The authors found compensation for depth, thus the PSS in the audiovisual TOJ task shifted with the apparent distance of the sound in accordance with the speed of sounds through air up to 40 m. Although on closer inspection of their data, it is clear that the shift in the PSS was mainly caused by the fact that sensitivity for intersensory synchrony became increasingly worse for more distant sounds. Judging from their figures, sensitivity for nearby sounds at 5 m was in the normal range, but for the most distant sound, sensitivity was extremely poor as it never reached plateau, and even at a sound delay of 200 ms, 25% of the responses was still "auditory-first" (see also Arnold et al. 2005; Lewald and Guski 2004). This suggests that observers, while performing the audiovisual TOJ task, could not use the onset of the far sound as a cue for temporal order, possibly because it lacks a sharp transient and that they had to rely on other cues instead. Besides controversial stimuli and data, there are others who simply failed to observe compensation for distance ( Arnold et al. 2005; Heron et al. 2007; Lewald and Guski 2004; Stone et al. 2001). For example, Stone et al. (2001) used an audiovisual SJ task and varied stimulus-observer distances from 0.5 m in the near condition to 3.5 m in the far condition. This resulted in a 3-m difference that would theoretically correspond to an 11 ms difference in the PSS if sound-travel time would not be compensated (sound velocity of 330 m/s corresponds to ∼3.5 m/11 ms). For three out of five subjects, the PSS values were indeed shifted in that direction, which led the authors to conclude that distance was not compensated. Against this conclusion, it should be said that the SJ tasks depend heavily on criterion settings, that "three-out-of-five" is not persuasively above chance, and that the range of distances was rather restricted.

Less open to these kinds of criticisms is a study by Lewald and Guski (2004). They used a rather wide range of distances (1, 5, 10, 20, and 50 m), and their audiovisual stimuli (a sequence of five beeps/flashes) were delivered by colocated speakers/LEDs placed in the open field. Note that in this case, there were no violations in the "naturalness" of the audiovisual stimuli and that they were physically colocated. Using this setup, the authors did not observe compensation for distance. Rather, their results showed that when the physical observer-stimulus distance increased, the PSS shifted precisely with the variation in sound transmission time through air. For audiovisual stimuli that are far away, sounds thus had to be presented earlier than for nearby stimuli to be perceived as simultaneous, and there was no sign that the brain would compensate for sound-traveling time. The authors also suggested that the discrepancy between their findings and those who did find compensation for distance lies in the fact that the latter simulated distance rather than using the natural situation.

Similar conclusions were also reached by Arnold et al. (2005), who examined whether the stream/ bounce illusion ( Sekuler et al. 1997) varies with distance. The authors examined whether the optimal time to produce a "bounce" percept varied with the distance of the display, which ranged from ∼1 to ∼15 m. The visual stimuli were presented on a computer monitor-keeping retinal properties constant-and the sounds were presented either over loudspeakers at these distances or over headphones. The optimal time to induce a bounce percept shifted with the distance of the sound if they were presented over loudspeakers, but there was no shift if the sound was presented over headphones. Similar effects of timing shifts with viewing distance after loudspeaker, but not headphone, presentation were obtained in an audiovisual TOJ task in which observers judged whether a sound came before or after two disks collided. This led the authors to conclude that there is no compensation for distance if distance is real and presented over speakers rather than simulated and presented over headphones.

This conclusion might well be correct, but it raises the question of how to account for the findings by Kopinska and Harris (2004). These authors reported complete compensation for distance despite using colocated sounds and lights produced at natural distances. In their study, the audiovisual stimulus was a bright disk that flashed once on a computer monitor and it was accompanied by a tone burst presented from the computer's inbuilt speaker. Participants were seated at various distances from the screen (1, 4, 8, 16, 24, and 32 m) and made TOJs about the flash and the sound. The authors also selectively slowed down visual processing by presenting the visual stimulus at 20° of eccentricity rather than in the fovea, or by having observers wear darkened glasses. As an additional control, they used simple reaction time tasks and found that all these variations-distance, eccentricity, and dark glasses-had predictable effects on auditory or visual speeded reaction. However, audiovisual simultaneity was not affected by distance, eccentricity, or darkened glasses. Thus, there was no shift in the PSS despite the fact that the change in distance, illumination, and retinal location affected simple reaction times. This made the authors conclude that observers recover the external world by taking into account all kinds of predictable variations, most importantly distance, alluding to similar phenomena such as size or color constancy.

There are some studies that varied audiovisual distance in a natural way, but came to diametrically opposing conclusions: Lewald and Guski (2004) and Arnold et al. (2005) found no compensation for distance, whereas Kopinska and Harris (2004) reported complete compensation. What's the critical difference between them? Our conjecture is that they differ in two critical aspects, that is, (1) whether distance was randomized on a trial-by-trial basis or blocked, and (2) whether sensitivity for temporal order was good or poor. In the study by Lewald and Guski, the distance of the stimuli was varied on a trial-by-trial basis as they used a setup of five different speakers/LEDs. In Kopinska and Harris's study, though, the distance between the observer and the screen was blocked over trials because otherwise subjects would have to be shifted back and forth after each trial. If the distance is blocked, then either adaptation to the additional sound lag may occur (i.e., recalibration), or subjects may equate response probabilities to the particular distance that they are seated. Either way, the effect of distance on the PSS will diminish if trials are blocked, and no shift in the PSS will then be observed, leading to the "wrong" conclusion that distance is compensated. This line of reasoning corresponds with a recent study by Heron et al. (2007). In their study, participants performed a TOJ task in which audiovisual stimuli (a white disk and a click) were presented at varying distances (0, 5, 10, 20, 30, and 40 m). Evidence for compensation was only found after a period of adaptation (1 min + 5 top-up adaptation stimuli between trials) to the naturally occurring audiovisual asynchrony associated with a particular viewing distance. No perceptual compensation for distance-induced auditory delays could be demonstrated whenever there was no adaptation period (although we should notice that in the present study, observer distance was always blocked).

The second potentially relevant difference between studies that do or do not demonstrate compensation is the difficulty of the stimuli. Lewald and Guski (2004) used a sequence of five pulses/ sounds, whereas Kopinska and Harris (2004) presented a single sound/flash. In our experience, a sequence of pulses/flashes drastically improves accuracy for temporal order if compared to a single pulse/flash because there are many more cues in the signal. In the study by Arnold et al. (2005), judgments about temporal order could also be relatively accurate because the two colliding disks provided anticipatory information about when to expect the sound. Most likely, observers in the study of Kopinska and Harris were inaccurate because their single sound/flash stimuli without anticipatory information were difficult (unfortunately, none of the studies reported JNDs). In effect, this amounts to adding noise to the psychometric function, which then effectively masks the effect of distance on temporal order. It might easily lead one to conclude "falsely" that there is compensation for distance.

9.5.3. Temporal Recalibration

The first evidence of recalibration in the temporal domain came from two studies with very similar designs: an exposure-test paradigm. Both Fujisaki et al. (2004) and Vroomen et al. (2004) first exposed observers to a train of sounds and light flashes with a constant but small intersensory interval, and then tested them by using an audiovisual TOJ or SJ task. The idea was that observers would adapt to small audiovisual lags in such a way that the adapted lag is eventually perceived as synchronous. Therefore, after a light-first exposure, light-first trials would be perceived as synchronous, and after a sound-first exposure, a sound-first stimulus would be perceived as synchronous (see Figure 9.3). Both studies indeed observed that the PSS was shifted in the direction of the exposure lag. For example, Vroomen and Keetels exposed subjects for ∼3 min to a sequence of sound bursts/ light flashes with audiovisual lags of either ±100 or ±200 ms (sound-first or light-first). During the test, the PSS was shifted, on average, by 27 and 18 ms (PSS difference between sound-first and light-first) for the SJ and TOJ tasks, respectively. Fujisaki et al. used slightly bigger lags (±235 ms sound-first or light-first) and found somewhat bigger shifts in the PSS (59 ms shifts of the PSS in SJ and 51 ms in TOJ), but data were, in essence, comparable. Many others have reported similar effects ( Asakawa et al. 2009; Di Luca et al. 2007; Hanson et al. 2008; Keetels and Vroomen 2007, 2008b; Navarra et al. 2005, 2007, 2009; Stetson et al. 2006; Sugano et al. 2010; Sugita and Suzuki 2003; Takahashi et al. 2008; Tanaka et al. 2009a; Yamamoto et al. 2008).

The mechanism underlying temporal recalibration, though, remains elusive at this point. One option is that there is a shift in the criterion for simultaneity in the adapted modalities ( Figure 9.2, panel 3a). After exposure to light-first pairings, participants may thus change their criterion for audiovisual simultaneity in such a way that light-first stimuli are taken to be simultaneous. On this view, other modality-pairings (e.g., vision-touch) would be unaffected and the change in criterion should then not affect unimodal processing of visual and auditory stimuli presented in isolation. Another strong prediction is that stimuli that were once synchronous, before adaptation, can become asynchronous after adaptation. The most dramatic case of this phenomenon can be found in motor-visual adaptation. In a study by Eagleman and Holcombe (2002), participants were asked to repeatedly tap their finger on a key, and after each key tap, a delayed flash was presented. If the visual flash occurred at an unexpectedly short delay after the tap (or synchronous), it was actually perceived as occurring before the tap, an experience that runs against the law of causality.

It may also be the case that one modality (vision, audition, or touch) is "shifted" toward the other, possibly because the sensory threshold for stimulus detection in one of the adapted modalities is changed (see Figure 9.2, panel 3b). For example, as an attempt to perceive simultaneity during light-first exposure, participants might delay processing time in the visual modality by adopting a more stringent criterion for sensory detection of visual stimuli. After exposure to light-first audiovisual pairings, one might then expect slower processing times of visual stimuli in general, and other modality pairings that involve the visual modality, say vision-touch, would then also be affected.

Two strategies have been undertaken to explore the mechanism underlying temporal recalibration. The first is to examine whether temporal recalibration generalizes to other stimuli within the adapted modalities, the second is to examine whether temporal recalibration affects different modality pairings than the ones adapted. Fujisaki et al. (2004) have already demonstrated that the effect of adaptation in temporal misalignment was effective even when the visual test stimulus was very different from the exposure situation. The authors exposed observers to asynchronous tone-flash stimulus pairs and later tested them on the "stream/bounce" illusion ( Sekuler et al. 1997). Fujisaki et al. reported that the optimal delay for obtaining a bounce percept in the stream/bounce illusion was shifted in the same direction as the adapted lag. Furthermore, after exposure to a "wall-display," in which tones were timed with a ball bouncing off the inner walls of a square, similar shifts in the PSS on the bounce percept were found (a ∼45 ms difference when comparing the PSS of the -235 ms sound-first exposure with the +235 ms vision-first exposure). Audiovisual temporal recalibration thus generalized well to other visual stimuli.

Navarra et al. (2005) and Vatakis et al. (2008b) also tested generalization for audiovisual temporal recalibration using stimuli from different domains (speech/nonspeech). Their observers had to monitor a continuous speech stream for target words that were presented either in synchrony with the video of a speaker, or with the audio stream lagging 300 ms behind. During the monitoring task, participants performed a TOJ ( Navarra et al. 2005; Vatakis et al. 2007) or SJ task ( Vatakis et al. 2008b) on simple flashes and white noise bursts that were overlaid on the video. Their results showed that sensitivity, rather than a shift in the PSS, became worse if subjects were exposed to desynchronized rather than synchronized audiovisual speech. Similar effects (larger JNDs) were found with music stimuli. This led the authors to conclude that the "window of temporal integration" was widened (see Figure 9.2, panel 3c) because of asynchronous exposure (see also Navarra et al. 2007 for effects on JND after adaptation to asynchronous audio-tactile stimuli). The authors argued that this effect on the JND may reflect an initial stage of recalibration in which a more lenient criterion is adopted for simultaneity. With prolonged exposure, subjects may then shift the PSS. An alternative explanation-also considered by the authors, but rejected-might be that subjects became confused by the nonmatching exposure stimuli, which as a result may also affect the JND rather than the PSS because it adds noise to the distribution.

The second way to study the underlying mechanisms of temporal recalibration is to examine whether temporal recalibration generalizes to different modality pairings. Hanson et al. (2008) explored whether a "supramodal" mechanism might be responsible for the recalibration of multisensory timing. They examined whether adaptation to audiovisual, audio-tactile, and tactile-visual asynchronies (10 ms flashes, noise bursts, and taps on the left index finger) generalized across modalities. The data showed that a brief period of repeated exposure to ±90 ms asynchrony in any of these pairings resulted in shifts of about 70 ms of the PSS on subsequent TOJ tasks, and that the size and nature of the shifts were very similar across all three pairings. This made them conclude that there is a "general mechanism." Opposite conclusions though, were reached by Harrar and Harris (2005). They exposed participants for 5 min to audiovisual pairs with a fixed time lag (250 ms light-first), but did not obtain shifts in the PSSs for touch-light pairs. In an extension of this topic ( Harrar and Harris 2008), observers were exposed for 5 min to ∼100 ms lags of light-first stimuli for the audiovisual case, and touch-first stimuli for the auditory-tactile and visual-tactile case. Participants were tested on each of these pairs before and after exposure. Shifts of the PSS in the predicted direction were only found in the audiovisual exposure-test stimuli, but not for the other cases. Di Luca et al. (2007) also exposed participants to asynchronous audiovisual pairs (∼200 ms lags of sound-first and light-first) and measured the PSS for audiovisual, audio-tactile, and visual-tactile test stimuli. Besides obtaining a shift in the PSS for audiovisual pairs, the effect was found to generalize to audio-tactile, but not to visual-tactile test pairs. This pattern made the authors conclude that adaptation resulted in a phenomenal shift of the auditory event ( Di Luca et al. 2007).

To summarize, until now, there is no clear explanation for the mechanism underlying temporal recalibration as there is some discrepancy in the data regarding generalization across modalities. It seems safe to conclude that the audiovisual exposure-test situation is the most reliable one to obtain a shift in the PSS. Arguably, audiovisual pairs are more flexible because the brain has to correct for timing differences between auditory and visual stimuli because of naturally occurring delays caused by distance. Tactile stimuli might be more rigid in time because visual-tactile and audio-tactile events always occur at the body surface, so less compensation for latency differences might be required here. As already mentioned above, a widening of the JND, rather than a shift in the PSS, has also been observed and it might possibly reflect an initial stage of recalibration in which a more lenient criterion about simultaneity is adopted. The reliability of each modality on its own is also likely to play a role. For visual stimuli, it is known that they are less reliable in time than auditory or tactile stimuli ( Fain 2003), and as a consequence they may be more malleable ( Ernst and Banks 2002; Ernst et al. 2000; Ernst and Bulthoff 2004), but there is also evidence that the auditory modality is, in fact, shifted.

9.5.4. Temporal Ventriloquism

The fourth possibility of how the brain might deal with lags between the senses, and how they may get unnoticed, is that the perceived timing of a stimulus in one modality is actively shifted toward the other (see Figure 9.2, panel 4). This phenomenon is also known as "temporal ventriloquism," and it is named in analogy with the spatial ventriloquist effect. For spatial ventriloquism, it was already known for a long time that listeners who heard a sound while seeing a spatially displayed flash had the (false) impression that the sound originated from the flash. This phenomenon was named the "ventriloquist illusion" because it was considered a stripped-down version of what the ventriloquist was doing when performing on stage. The temporal ventriloquist effect is analogous to the spatial variant, except that here, sound attracts vision in the time dimension rather than vision attracting sound in the spatial dimension. There are, by now, many demonstrations of this phenomenon, and we describe several in subsequent paragraphs. They all show that small lags between sound and vision go unnoticed because the perceived timing of visual events is flexible and is attracted toward events in other modalities.

Scheier et al. (1999) were one of the first to demonstrate temporal ventriloquism using a visual TOJ task (see Figure 9.4). Observers were presented with two lights at various SOAs, one above and one below a fixation point, and their task was to judge which light came first (the upper or the lower). To induce temporal ventriloquism, Scheier et al. added two sounds that could either be presented before the first and after the second light (condition AVVA), or the sounds could be presented in between the two lights (condition VAAV). Note that they used a visual TOJ task, and that sounds were task-irrelevant. The results showed that observers were more sensitive (i.e., smaller intervals were still perceived correctly) in the AVVA condition compared to the VAAV condition (visual JNDs were approximately 24 and 39 ms, respectively). Presumably, the two sounds attracted the temporal occurrence of the two lights, and thus, effectively pulled the lights farther apart in the AVVA condition, and closer together in the VAAV condition. In single-sound conditions, AVV and VVA, sensitivity was not different from a visual-only baseline, indicating that the effects were not because of the initial sound acting as a warning signal, or some cognitive factor related to the observer's awareness of the sounds.

Morein-Zamir et al. (2003) replicated these effects and further explored the sound-light intervals at which the effect occurred. Sound-light intervals of ∼100 to ∼600 ms were tested, and it was shown that the second sound was mainly responsible for the temporal ventriloquist effect up to a sound-light interval of 200 ms, whereas the interval of the first sound had little effect.

The results were also consistent with earlier findings of Fendrich and Corballis (2001) who used a paradigm in which participants judged when a flash occurred by reporting the clock position of a rotating marker. The repeating flash was seen earlier when it was preceded by a click and later when the click lagged the visual stimulus. Another demonstration of temporal ventriloquism using a different paradigm came from a study by Vroomen and de Gelder (2004b). Here, temporal ventriloquism was demonstrated using the flash-lag effect (FLE). In the typical FLE ( Mackay 1958; Nijhawan 1994, 1997, 2002), a flash appears to lag behind a moving visual stimulus even though the stimuli are presented at the same physical location. To induce temporal ventriloquism, Vroomen and de Gelder added a single click presented slightly before, at, or after the flash (intervals of 0, 33, 66, and 100 ms). The results showed that the sound attracted the temporal onset of the flash and shifted it in the order of ∼5%. A sound ∼100 ms before the flash thus made the flash appear ∼5 ms earlier, and a sound 100 ms after the flash made the flash appear ∼5 ms later. A sound, including the synchronous one, also improved sensitivity on the visual task because JNDs on the visual task were better if a sound was present rather than absent.

Yet another recent manifestation of temporal ventriloquism used an apparent visual motion paradigm. Visual apparent motion occurs when a stimulus is flashed in one location and is followed by another identical stimulus flashed in another location ( Korte 1915). Typically, an illusory movement is observed that starts at the lead stimulus and is directed toward the second lagging stimulus (the strength of the illusion depends on the exposure time of the stimuli, and the temporal and spatial separation between them). Getzmann (2007) explored the effects of irrelevant sounds on this motion illusion. In their study, two temporally separated visual stimuli (SOAs ranged from 0 to 350 ms) were presented and participants classified their impression of motion using a categorization system. The results demonstrated that sounds intervening between the visual stimuli facilitated the impression of apparent motion relative to no sounds, whereas sounds presented before the first and after the second visual stimulus reduced motion perception (see Bruns and Getzmann 2008 for similar results). The idea was that because exposure time and spatial separation were both held constant in this study, the impression of apparent motion was systematically affected by the perceived length of the interstimulus interval. The effect was explained in terms of temporal ventriloquism, as sounds attracted the illusory onset of visual stimuli.

Freeman and Driver (2008) investigated whether the timing of a static sound could influence spatiotemporal processing of visual apparent motion. Apparent motion was induced by visual stimuli alternating between opposite hemifields. The perceived direction typically depends on the relative timing interval between the left-right and right-left flashes (e.g., rightward motion dominating when left-right interflash intervals are shortest; von Grunau 1986). In their study, the interflash intervals were always 500 ms (ambiguous motion), but sounds could slightly lead the left flash and lag the right flash by 83 ms or vice versa. Because of temporal ventriloquism, this variation made visual apparent motion depend on the timing of the sound stimuli (e.g., more rightward responses if a sound preceded the left flash, and lagged the right flash, and more leftward responses if a sound preceded the right flash, and lagged the left flash).

The temporal ventriloquist effect has also been used as a diagnostic tool to examine whether commonality in space is a constraint on intersensory pairing. Vroomen and Keetels (2006) adopted the visual TOJ task of Scheier et al. (1999) and replicated that sounds improved sensitivity in the AVVA version of the visual TOJ task. Importantly, the temporal ventriloquist effect was unaffected by whether sounds and lights were colocated or not. For example, the authors varied whether the sounds came from a central location or a lateral one, whether the sounds were static or moving, and whether the sounds and lights came from the same or different sides of fixation at either small or large spatial disparities. All these variations had no effect on the temporal ventriloquist effect, despite that discordant sounds were shown to attract reflexive spatial attention and to interfere with speeded visual discrimination. These results made the author conclude that intersensory interactions in general do not require spatial correspondence between the components of the cross-modal stimuli (see also Keetels et al. 2007).

In another study ( Keetels and Vroomen 2008a), it was explored whether touch affects vision on the time dimension as audition does (visual-tactile ventriloquism), and whether spatial disparity between the vibrator and lights modifies this effect. Given that tactile stimuli are spatially better defined than tones because of their somatotopic rather than tonotopic initial coding, this study provided a strong test case for the notion that spatial co-occurrence between the senses is required for intersensory temporal integration. The results demonstrated that tactile-visual stimuli behaved like audiovisual stimuli, in that temporally misaligned tactile stimuli captured the onsets of the lights and spatial discordance between the stimuli did not harm this phenomenon.

Besides exploring whether spatial disparity affects temporal ventriloquism, the effect of synesthetic congruency between modalities was also recently explored ( Keetels and Vroomen 2010; Parise and Spence 2008). Parise and Spence (2008) suggested that pitch size synesthetic congruency (i.e., a natural association between the relative pitch of a sound and the relative size of a visual stimulus) might affect temporal ventriloquism. In their study, participants made visual TOJs about small-sized and large-sized visual stimuli whereas high-pitched or low-pitched tones were presented before the first and after the second light. The results showed that, at large sound-light intervals, sensitivity for visual temporal order was better for synesthetically congruent than incongruent pairs. In a more recent study, Keetels and Vroomen (2010) reexamined this effect and showed that this congruency effect could not be attributed to temporal ventriloquism, as it disappeared at short sound-light intervals if compared to a synchronous AV baseline condition that excludes response biases. In addition, synesthetic congruency did not affect temporal ventriloquism even if participants were made explicitly aware of congruency before testing, challenging the view that synesthetic congruency affects temporal ventriloquism.

Stekelenburg and Vroomen (2005) also investigated the time course and the electrophysiological correlates of the audiovisual temporal ventriloquist effect using ERPs in the FLE. Their results demonstrated that the amplitude of the visual N1 was systematically affected by the temporal interval between the visual target flash and the task-irrelevant sound in the FLE paradigm ( Mackay 1958; Nijhawan 1994, 1997, 2002). If a sound was presented in synchrony with the flash, the N1 amplitude was larger than when the sound lagged the visual stimulus, and it was smaller when the sound lead the flash. No latency shifts, however, were found. Yet, based on the latency of the cross-modal effect (N1 at 190 ms) and its localization in the occipitoparietal cortex, this study confirmed the sensory nature of temporal ventriloquism. An explanation for the absence of a temporal shift of the ERP components may lie in the small size of the temporal ventriloquist effect found (3 ms). Such a small temporal difference may not be reliably reflected in the ERPs because it reaches the lower limit of the temporal resolution of the sampled EEG.

In most of the studies examining temporal ventriloquism (visual TOJ, FLE, reporting clock position or motion direction), the timing of the visual stimulus is the task-relevant dimension. Although recently, Vroomen and Keetels (2009) explored whether a temporally offset sound could improve the identification of a visual stimulus whereas temporal order is not involved. In this study, it was examined whether four-dot masking was affected by temporal ventriloquism. In the four-dot masking paradigm, visual target identification is impaired when a briefly presented target is followed by a mask that consists of four dots that surround but do not touch the visual target ( Enns 2004; Enns and DiLollo 1997, 2000). The idea tested was that a sound presented slightly before the target and slightly after the mask might lengthen the perceived interval between target and mask. By lengthening the perceived target-mask interval, there is more time for the target to consolidate, and in turn target identification should be easier. Results were in line with this hypothesis as a small release from four-dot masking was reported (1% improvement, which corresponds to an increase of the target-mask ISI of 4.4 ms) if two sounds were presented at approximately 100-ms intervals before the target and after the mask, rather than if only a single sound was presented before the target or a silent condition.

To summarize, there are by now many demonstrations that vision is flexible on the time dimension. In general, the perceived timing of a visual event is attracted toward other events in audition and touch, provided that the lag between them is less than ∼200 ms. The deeper reason why there is this mutual attraction is still untested. Although in our view, it serves to reduce natural lags between the senses so that they become unnoticed, thus maintaining coherence between the senses.

9.6. TEMPORAL SYNCHRONY: AUTOMATIC OR NOT?

An important property about the perception of intersensory synchrony is to know whether it is perceived in an automatic fashion or not. As is often the case, there are two opposing views on this issue. Some have reported that the detection of temporal alignment is a slow, serial, and attention-demanding process, whereas others have argued that it is fast and only requires a minimal amount of attention that is needed to perceive the visual stimulus, but once this criterion is met, audiovisual or visual-tactile integration comes for free.

An important signature of automatic processing is that the stimulus in question is salient and "pops out." If so, the stimulus is easy to find among distracters. What about intersensory synchrony: does it "pop out"? In a study by van de Par and Kohlrausch (2004), this question was addressed by presenting observers a visual display of a number of independently moving circles moving up and down along a Gaussian profile. Along with the motion display, a concurrent sound was presented in which amplitude was modulated coherently with one of the circles. The participant's task was to identify the coherently moving visual circle as quickly as possible. The authors found that response times increased approximately linearly with the numbers of distracters (∼500 ms/distracter), indicating a slow serial search process rather than pop-out.

Fujisaki et al. (2006) came to similar conclusions. They examined search functions for a visual target that changed in synchrony with an auditory stimulus. The visual display consisted of two, four, or eight luminance-modulated Gaussian blobs presented at 5, 10, 20, and 40 Hz that were accompanied by a white noise sound whose amplitude was modulated in synch with one of the visual stimuli. Other displays contained clockwise/counterclockwise rotations of windmills synchronized with a sound whose frequency was modulated up or down at a rate of 10 Hz. The observers' task was to indicate which visual stimulus was luminance-modulated in synch with the sound. Search functions for both displays were slow (∼1 s/distractor in target-present displays), and increased linearly with the number of visual distracters. In a control experiment, it was also shown that synchrony discrimination was unaffected by the presence of distractors if attention was directed at the visual target. Fujisaki et al. therefore concluded that perception of audiovisual synchrony is a slow and serial process based on a comparison of salient temporal features that need to be individuated from within-modal signal streams.

Others, though, came to quite opposing conclusions and found that intersensory synchrony can be detected in an automatic fashion. Most notably, van der Burg et al. (2008b) reported an interesting study in which they showed that a simple auditory pip can drastically reduce search times for a color-changing object that is synchronized with the pip. The authors presented a horizontal or vertical target line among a large array of oblique lines. Each of the lines (target and distracters) changed color from green-to-red or red-to-green in a random fashion. If a pip sound was synchronized with a color change, visual attention was automatically drawn to the location of the line that changed color. When the sound was synchronized with the color change of the target, search times improved drastically and the number of irrelevant distracters had virtually no effect on search times (a nearly flat slope indicating pop-out). The authors concluded that the temporal information of the auditory signal was integrated with the visual signal generating a relatively salient emergent feature that automatically draw spatial attention (see also van der Burg et al. 2008a). Similar effects were also demonstrated for tactile stimuli instead of auditory pips ( Olivers and van der Burg 2008; van der Burg et al. 2009).

Kanai et al. (2007) also explored temporal correspondences in visually ambiguous displays. They presented multiple disks flashing sequentially at one of eight locations in a circle, thus inducing the percept of a disk revolving around fixation. A sound was presented at one particular location in every cycle, and participants had to indicate the disk that was temporally aligned with the sound. The disk seen as being synchronized with the sound was perceived as brighter with a sharper onset and offset ( Vroomen and de Gelder 2000). Moreover, it fluctuated over time and its position changed every 5 to 10 s. Kanai et al. explored whether this flexibility was dependent on attention by having observers perform a concurrent task in which they had to count the number of X's in a letter stream. The results demonstrated that the transitions disappeared whenever attention was distracted from the stimulus. On the other hand, if attention was directed to one particular visual event-either by making it "pop-out" by a using a different color, by presenting a cue next to the target dot, or by overtly cueing it-the perceived timing of the sound was attracted toward that event. These results thus suggest that perception of intersensory synchrony is flexible, and is not completely immune to attention.

These opposing views on the role of attention can be reconciled on the assumption that perception of synchrony depends on a matching process of salient temporal features ( Fujisaki et al. 2006; Fujisaki and Nishida 2007). Saliency may be lost when stimuli are presented at fast rates (typically above 4 Hz), when perceptually grouped into other streams, or if they lack a sharp transition ( Keetels et al. 2007; Sanabria et al. 2004; Vroomen and de Gelder 2004a; Watanabe and Shimojo 2001). In line with this notion, studies reporting that audiovisual synchrony detection is slow, either presented stimuli at fast rates (>4 Hz up to 80/s) or the stimuli lacked a sharp onset/offset (e.g., van de Par, using a Gaussian amplitude modulation). Others reporting automatic detection of auditory-visual synchrony used much slower rates (1.11 Hz; van der Burg et al. 2008b) and sharp transitions (a pip).

9.7. NEURAL SUBSTRATES OF TEMPORAL SYNCHRONY

Although temporal correspondence is frequently considered one of the most important constraints on cross-modal integration (e.g., Bedford 1989; Bertelson 1999; Radeau 1994; Stein and Meredith 1993; Welch 1999; Welch and Warren 1980), the neural correlates for the ability to detect and use temporal synchrony remain largely unknown. Most likely, however, a whole network is involved. Seminal studies examining the neural substrates of intersensory temporal correspondence were done in animals. The finding that the firing rate of a subsample of cells in the superior colliculi (SC) increases dramatically and more than what can be expected by summing the unimodal impulses when auditory (tones) and visual stimuli (flashes) occur in close temporal and spatial proximity ( Meredith et al. 1987; Stein et al. 1993) is well-known. More recently, Calvert et al. (2001) used functional magnetic resonance imaging (fMRI) on human subjects for studying brain areas that demonstrate facilitation and suppression effects in the blood oxygenation level-dependent (BOLD) signal for temporally aligned and temporally misaligned audiovisual stimuli. Their stimulus consisted of a reversing checkerboard pattern of alternating black and white squares with sounds presented either simultaneously with the onset of a reversal (synchronous condition) or a randomly phase-shifted asynchronous condition. The results showed an involvement of the SC as its response was superadditive for temporally matched stimuli and depressed for the temporally mismatched ones. Other cross-modal interactions were also identified in a network of cortical brain areas that included several frontal cortical sites; the right inferior frontal gyrus, multiple sites within the right lateral sulcus, and the ventromedial frontal gyrus. Furthermore, response enhancement and depression was observed in the insula bilaterally, right superior parietal lobule, right inferior parietal sulcus, left superior occipital gyrus, and left superior temporal sulcus (STS).

Bushara et al. (2001) examined the effect of temporal asynchrony in a positron emission tomography study. Here, observers had to decide whether a colored circle was presented simultaneously with a tone or not. The stimulus pairs could either be auditory-first (AV) or vision-first (VA) at three levels of SOAs that varied in difficulty. A control condition (C) was included in which the auditory and visual stimuli were presented simultaneously, and in which participants performed a visual color discrimination task whenever a sound was present. The brain areas involved in auditory-visual synchrony detection were identified by subtracting the activity of the control condition from that in the asynchronous conditions (AV-C and VA-C). Results revealed a network of heteromodal brain areas that included the right anterior insula, the right ventrolateral prefrontal cortex, right inferior parietal lobe, and left cerebellar hemisphere. The activity in the areas that correlated positively with decreasing asynchrony revealed a cluster within the right insula, suggesting that this region is most important for the detection of auditory-visual synchrony. Given that interactions were also found between the insula, the posterior thalamus, and the SC, it was suggested that intersensory temporal processing is mediated via subcortical tecto-thalamo-insula pathways.

In a positron emission tomography study by Macaluso et al. (2004), subjects were looking at a video monitor that showed a face mouthing words. In different blocks of trials, the audiovisual signals were either presented synchronously or asynchronously (the auditory stimulus was leading by a clearly noticeable 240 ms). In addition, the visual and auditory sources were either presented at the same location or in opposite hemifields. Results showed that activity in ventral occipital areas and left STS increased during synchronous audiovisual speech, regardless of the relative location of the auditory and visual input.

More recently, in an fMRI study, Dhamala et al. (2007) examined the networks that are involved in the perception of physically synchronous versus asynchronous audiovisual events. Two timing parameters were varied: the SOA between sound and light (−200 to +200 ms) and the stimulation rate (0.5-3.5 Hz). In the behavioral task, observers had to report whether stimuli were perceived as simultaneous, sound-first, light-first, or "Can't tell," resulting in the classification of three distinct perceptual states, that is, the perception of synchrony, asynchrony, and "no clear perception." The fMRI data showed that each of these stages involved activation in different brain networks. Perception of asynchrony activated the primary sensory, prefrontal, and inferior parietal cortices, whereas perception of synchrony disengaged the inferior parietal cortex and further recruited the SC.

An fMRI study by Noesselt et al. (2007) also explored the effect of temporal correspondence between auditory and visual streams. The stimuli were arranged such that auditory and visual streams were temporally corresponding or not, using irregular and arrhythmic temporal patterns that either matched between audition and vision or mismatched substantially whereas maintaining the same overall temporal statistics. For the coincident audiovisual streams, there was an increase in the BOLD response in multisensory STS contralateral to the visual stream. The contralateral primary visual and auditory cortex were also found to be affected by the synchrony-asynchrony manipulations, and a connectivity analysis indicated enhanced influence from mSTS on primary sensory areas during temporal correspondence.

In an EEG paradigm, Senkowski et al. (2007) examined the neural mechanisms underlying intersensory synchrony by measuring oscillatory gamma-band responses (GBRs; 30-80 Hz). Oscillatory GBRs have been linked to feature integration mechanisms and to multisensory processing. The authors reasoned that GBRs might also be sensitive to the temporal alignment of intersensory stimulus components. The temporal synchrony of auditory and visual components of a multisensory signal was varied (tones and horizontal gratings with SOAs ranging from −125 to +125 ms). The GBRs to the auditory and visual components of multisensory stimuli were extracted for five subranges of asynchrony and compared with GBRs to unisensory control stimuli. The results revealed that multisensory interactions were strongest in the early GBRs when the sound and light stimuli were presented with the closest synchrony. These effects were most evident over medial-frontal brain areas after 30 to 80 ms and over occipital areas after 60 to 120 ms, indicating that temporal synchrony may have an effect on early intersensory interactions in the human cortex.

Overall, it should be noted that there is a lot of variation in the outcomes of studies that have examined the neural basis of intersensory temporal synchrony. At present, the issue is far from resolved and more research has to be performed to unravel the exact neural substrates underlying it. The overall finding is that the SC and mSTS are repeatedly reported in intersensory synchrony detection studies, which at least suggests a prominent role for these structures in the processing of intersensory stimuli based on their temporal correspondence. For the time being, however, it is unknown how these areas would affect the perception of intersensory synchrony if they were damaged or temporarily blocked by, for example, transcranial magnetic stimulation.

9.8. CONCLUSIONS

In recent years, a substantial amount of research has been devoted to understanding how the brain handles lags between the senses. The most important conclusion we draw is that intersensory timing is flexible and adaptive. The flexibility is clearly demonstrated by studies showing one or another variant of temporal ventriloquism. In that case, small lags go unnoticed because the brain actively shifts one information stream (usually vision) toward the other, possibly to maintain temporal coherence. The adaptive part rests on studies of temporal recalibration demonstrating that observers are flexible in adopting what counts as synchronous. The extent to which temporal recalibration generalizes to other stimuli and domains, however, remains to be further explored. The idea that the brain compensates for predictable variability between the senses-most notably distance-is, in our view, not well-founded. We are more enthusiastic about the notion that intersensory synchrony is perceived mostly in an automatic fashion, provided that the individual components of the stimuli are sufficiently salient. The neural mechanisms that underlie this ability are of clear importance for future research.

REFERENCES

1. Asakawa K, Tanaka A, Imai H. 31st Annual Meeting of the Cognitive Science Society. Amsterdam, The Netherlands: Jul 29, Aug 29, 2009. 2009. Temporal Recalibration in Audio-Visual Speech Integration Using a Simultaneity Judgment Task and the McGurk Identification Task. Paper presented at the.

2. Bertelson P. Ventriloquism: A case of crossmodal perceptual grouping. In: Aschersleben G, Bachmann T, Musseler J, editors. Cognitive Contributions to the Perception of Spatial and Temporal Events. North-Holland: Elsevier; 1999. pp. 347-63.

3. Calvert G, Spence C, Stein B. The Handbook of Multisensory Processes. Cambridge, MA: The MIT Press; 2004.

4. Di Luca M, Machulla T, Ernst M.O. International Intersensory Research Symposium 2007: Perception and Action. Sydney, Australia: Jul 3, 2007. 2007. Perceived Timing Across Modalities.

5. Fain G.L. Sensory Transduction. Sunderland, MA: Sinauer Associates; 2003.

6. Finger R, Davis A.W. Technical Report SN187-D. Los Gatos, CA: Pixel Instrument Corporation; 2001. Measuring Video Quality in Videoconferencing Systems.

7. Jaskowski P. Reaction time and temporal-order judgment as measures of perceptual latency: The problem of dissociations. In: Aschersleben G, Bachmann T, Musseler J, editors. Cognitive Contributions to the Perception of Spatial and Temporal Events. North-Holland: Elsevier Science B.V; 1999. pp. 265-82.

8. Levitin D, MacLean K, Mathews M, Chu L. The perception of cross-modal simultaneity. International Journal of Computing and Anticipatory Systems. 2000:323-9.

9. Neumann O, Niepel M. Timing of "perception" and perception of "time" In: Kaernbach C, Schroger E, Muller H, editors. Psychophysics Beyond Sensation: Laws and Invariants of Human Cognition. Lawrence Erlbaum Associates, Inc; 2004. pp. 245-70.

10. Pöppel E. Grenzes des bewusstseins, Stuttgart: Deutsche Verlags-Anstal, translated as Mindworks: Time and Conscious Experience. New York: Harcourt Brace Jovanovich; 1985. 1988.

11. Rihs S. The Influence of Audio on Perceived Picture Quality and Subjective Audio-Visual Delay Tolerance. Paper presented at the MOSAIC Workshop: Advanced methods for the evaluation of television picture quality, Eindhoven. Sep 18, 1995.

12. Shore D.I, Spence C, Klein R.M. Prior entry. In: Itti L, Rees G, Tsotsos J, editors. Neurobiology of Attention. North Holland: Elsevier; 2005. pp. 89-95.

13. Spence C, Driver J. Crossmodal Space and Crossmodal Attention. Oxford: Oxford University Press; 2004.

14. Stein B.E, Meredith M.A. The Merging of the Senses. Cambridge, MA: The MIT Press; 1993.

15. Sternberg S, Knoll R.L. The perception of temporal order: Fundamental issues and a general model. In: Kornblum S, editor. Attention and Performance. IV. New York: Academic Press; 1973. pp. 629-85.

16. Summerfield Q. A comprehensive account of audio-visual speech perception. In: Dodd B, Campbell R, editors. Hearing by Eye: The Psychology of Lip-Reading. London: Lawrence Erlbaum Associates; 1987. pp. 3-51.

17. Titchener E.B. Lectures on the Elementary Psychology of Feeling and Attention. New York: Macmillan; 1908.

18. van de Par S, Kohlrausch A. 18th International Congress on Acoustics. Kyoto, Japan: 2004. Visual and auditory object selection based on temporal correlations between auditory and visual cues. Paper presented at the.

19. Van Eijk R.L. Audio-Visual Synchrony Perception. Thesis, Technische Universiteit Eindhoven. The Netherlands: 2008.

20. Von Helmholtz H. Handbuch der Physiologischen Optik. Leipzig: Leopold Voss; 1867.

21. Vroomen J, de Gelder B. Perceptual effects of cross-modal stimulation: Ventriloquism and the freezing phenomenon. In: Calvert G.A, Spence C, Stein B.E, editors. The Handbook of Multisensory Processes. Cambridge, MA: MIT Press; 2004a.

22. Welch R.B. Meaning, attention, and the "unity assumption" in the intersensory bias of spatial and temporal perceptions. In: Aschersleben G, Bachmann T, Musseler J, editors. Cognitive Contributions to the Perception of Spatial and Temporal Events. Amsterdam: Elsevier; 1999. pp. 371-87.

23. Yamamoto S, Miyazaki M, Iwano T, Kitazawa S. 9th International Multisensory Research Forum. Hamburg, Germany: Jul 16, 2008. 2008. Bayesian calibration of simultaneity in audio-visual temporal order judgment. Paper presented at the.

*

In the McGurk illusion (McGurk and MacDonald 1976), it is shown that the perception of nonambiguous speech tokens can be modified by the simultaneous presentation of visually incongruent articulatory gestures. Typically, when presented with an auditory syllable /ba/ dubbed onto a face articulating /ga/, participants report hearing /da/. The occurrence of this so-called McGurk effect has been taken as a particularly powerful demonstration of the use of visual information in speech perception.

*

It has also been reported that the presentation rate may shift the PSS. In a study by Arrighi et al. (2006), participants were presented a video of hands drumming on a conga at various rates (1, 2, and 4 Hz). Observers were asked to judge whether the auditory and visual streams appeared to be synchronous or not (an SJ task). Results showed that the auditory delay for maximum simultaneity (the PSS) varied inversely with drumming tempo from about 80 ms at 1 Hz, and 60 ms at 2 Hz, to 40 ms at 4 Hz. Video sequences of random drumming motion and of a disk moving along the motion profile matching the hands of to the drummer produced similar results, with higher tempos requiring less auditory delay.

# data[].attributes.attachment_link.title
Perception of Synchrony between the Senses

# data[].attributes.attachment_link.description
In recent years, a substantial amount of research has been devoted to understanding how the brain handles these timing differences (Calvert et al. 2004; King 2005; Levitin et al. 2000; Spence and Driver 2004; Spence and Squire 2003). Here, we review several key issues about intersensory timing.