Personal tools
You are here: McMaster Institute for Music and the Mind > Publications > Musical use of visual gestures: the importance of contextual information in sensory integration

Michael Schutz and Michael Kubovy (2007)

Musical use of visual gestures: the importance of contextual information in sensory integration

Journal of Vision, 7(9):306.

At VSS 2005 we reported that percussionists use visual gestures to alter audience perception of musical note length (Schutz & Kubovy). This conflicts with previous research suggesting vision does not alter auditory judgments of tone duration (Walker & Scott, 1981). In order to determine why our results differ from prior research, we built a point light simulation of a percussionist's movements tracking the long and short gestures used to strike percussion instruments. After pilot studies demonstrated the simulation to be successful, a series of four experiments revealed:

  1. When using alternate sounds, the visual influence:

    1. Persists with percussive sounds (piano).

    2. Does not persist with non-percussive sounds (clarinet, trumpet, voice).

  2. Visual influence is lost when the moving dot pauses at the moment of impact before rebounding.

  3. Influence is lost when the motion path is rotated 180 degrees after impact to simulate moving “through the bar” rather than rebounding off of it.

  4. A static dot presented simultaneously with the percussive sound produces effects similar to those observed with gestures.


We conclude our initial results were driven by the perceived causal relationship between modalities. Only sounds caused by impacts (e.g. percussive sounds) are influenced by visual impact gestures (Experiment 1). Manipulations which obscure the perception of a visual impact have no effect on auditory judgments (Experiments 2). The most important aspect of the gesture is the sudden change of direction at the moment of impact (Experiment 3). However, the motion itself is important only in that it depicts a change. A spatially static visual change (e.g. the appearance of a dot on a previously blank screen) alters perception as reliably as the original gestures (Experiment 4). Together these studies show the importance of the relationship between the auditory and visual information in understanding the nature of sensory integration.

visual, music, sensory integration

Document Actions