Special Colloquium Multisensory interactions and plasticity – shooting hidden assumptions, revealing postdictive aspects

Abstract - Multisensory psychophysics and neuroscience have bloomed in the last several decades. In this talk, I first aim to give a critical review of the field by a list of commonly-accepted propositions/beliefs and provide (at least partially) counterevidence from our home-based laboratory. Some of such 'commonsensical' beliefs and counterevidence are as the following:

1) Vision is dominant, affecting other modalities, not vice versa, (as seemingly indicated by well-known illusions in the field, such as the McGurk and the Ventriloquism effects). 

Our “double flash illusion” may be a prominent exception to this principle, where the auditory input determines the content of visual percept (Shams, et al., 2000). A more flexible view may be needed, beyond the classical theories such as vision dominance or the modality specificity. 

2) Concurrent crossmodal stimulation is needed for multisensory interactions. That is, a percept is determined exclusively by concurrent stimuli in the other modalities.

3) Multisensory integration follows the Bayesian/Maximum Likelihood predictions.

Our study on crossmodal, temporal frequency adaptation seemingly offers a counterexample to both these principles above (i.e., the necessity of concurrent stimulation, and the Maximum Likelihood) (Levitan, et al., 2015). The pattern of results, including crossmodal transfer of adaptation, does not seem to match the straightforward predictions from Maximum Likelihood, at least. 

4) There are no intrinsic mappings across modalities, other than associatively learned from experience.

 “Intrinsic correspondence” (C. Spence), or synesthesia-like associations are known. On top of that, our study of multisensory associations in usage of a sensory-substitution device (vOICe) provides additional evidence of such intrinsic associations (Stiles & Shimojo, 2015).

5) The psychological body is restricted to the physical body, not having much flexibility based on experience.

The classical “inverted vision goggles” experiments (G. Stratton, I. Kohler, etc.) indicate rapid recalibration of multisensory body schema. In addition, the latest “Visual-Tactile Rabbit” demonstration in the VR environment points to dynamic flexibility of the multisensory body (Berger & Gonzalez-Franco, 2018). 

6) Conscious experiences of multisensory perception are governed by predictive processes. 

We carefully examined how our own “Auditory-Visual Rabbit” illusion depends on temporal parameters. The results revealed not only predictive but also postdictive aspects, where a stimulus presented in the other modality later in physical time still can affect the integrated event perception (Shimojo, 2014; Stiles, et al., 2018).

Altogether, these findings guide us to a more dynamic, and flexible view of multisensory integration. 

References

Berger, C.C. and Gonzalez-Franco, M. Expanding the sense of touch outside the body. Proc 15th ACM Symp Appl Percept - SAP ’18:1–9, 2018.

Levitan, C.A., Ban, Y-H.A., Noelle R. B. Stiles, N.R.B. and Shimojo, S.  Rate perception adapts across the senses: evidence for a unified timing mechanism. Sci. Reports, 5:8857,  doi:10.1038/srep08857, 2015. 

Shams, L., Kamitani, Y. and Shimojo, S. What you see is what you hear. Nature, 408, 788, 2000.

Shimojo, S. Postdiction: its implications on visual awareness, hindsight, and sense of agency. Frontiers in Psychology, 196, 1-19, 2014. doi: 10.3389/fpsyg.2014.00196, 2014.

Stiles, N. R. B. and Shimojo, S. Auditory Sensory Substitution is Intuitive and Automatic with Texture Stimuli. Scientific Reports, 5:15628, DOI: 10.1038/srep15628, 2015. 

Stiles, N.R.B., Li, M., Levitan, C.A., Kamitani, Y., Shimojo, S.What you saw is what you will hear: two new illusions with audiovisual postdictive effects. PLOS ONE, 13(10): e0204217.https://doi.org/10.1371/journal.pone.0204217, 2018.