HOME

ARTICLES

NET GALLERY

---

---

 

Sensory Substitution

 

judsoN

 

There is some debate whether cases of sensory substitution (1) are the results of imaginativeness, psychological effects or neurological (mis-)wiring. For our purposes, this distinction need not be relevant. We have evidence such substitutions do happen and that trumps any theory answering why. Whether or not Wagner, Klee or Kandinsky actually had synesthesia (2), there is a rich history of people equating one type of sensory stimuli for another. Most often we hear about organs, such as Scriabin's, that emitted colored light shows with each note. There has been a lot of experimentation over the years in search of correspondences between absolute frequencies of light and absolute frequencies in sound as well as in the cyclical orbit of planets and chakras (3). We can chalk it up to over-eager spiritualism, but who can really argue if in the end this proves inspirational to the composers.

It does seem peculiar that relative frequencies (for example musical intervals) are rather rare. Joseph Albers famous color studies (4) touch on relativistic issues, but the book is certainly not about music. (Or does that matter?) "A 'transformation' of 4 reds to 4 blues of a slightly lower key seems comparable to a transformation of a tetrachord in music from one instrument to another. Therefore we are also concerned with 'intervals.'" (5) Though not altogether literally true or even a very well articulated point, the analogy is a telling one. Simply it reveals that there is a metaphorical correlation we tend to make in language (for instance, extension of the metaphor we use to say "a brighter hue" can readily be extended to describe a tone). Extension of metaphors that act as building blocks for comprehension is hardly a radical notion (6). If we understand how a piece of string works, we can use this concept as a metaphor when imagining a number line. In a similar way, both traditional musical and/or visual art students will recognize the idea of "intervals".

As for music, the relativistic case is far more formalized thanks to Western music theory. Within a key, obviously the root note has a distinct sound from the minor second above it. But C in the key of G is furthermore perceived differently than the same C in the key of F#. Even accounting for perfect pitch or the ability to identify the name of the note, the context is the over all music. Naming the notes doesn't make them sound any more like music than noise. To detect musical-ness requires recognition of a context. This doesn't only apply to the pitches but also rhythms. "In addition, some experimenters have noted in passing that changing the metrical context of a melody by presenting it with a different accompaniment, can make it sound totally different." (7). Interestingly that context is simply a cluster of more notes, symbiotically dependant on the context, just as individual cars both part of and a delineate traffic. It is a theme that recurs in studies of networking.

Investigations on the perception of color are also enlightening. For instance, though languages vary in the amount of vocabulary devoted to naming colors and the number of them named even in cultures that only have words for "black" and "white", there is a remarkable agreement as to what the archetypal red is and the bluest blue. There is also a peculiar agreement as to what the basic colors are. The rainbow could be segmented an infinite number of ways, but the divisions tend toward the same number and uneven spacing on the spectrum.

 

"Indeed, Rosch's predecessors in this work, Brent Berlin and Paul Lay, had previously discovered a universality to the order in which primitive cultures come to assign color names, with very primitive cultures only having words for black and white. More advanced cultures sequentially add red, then yellow and green,(in either order), then blue, then brown, and finally pink, orange and grey (these last four in any order).


… There is, it appears, a natural way to divide the spectrum into color categories, a division that appears to have more to do with the color itself and our human mechanism for experiencing it (our three types of cones and how they are wired to our neurons) than it has to do with any particular culture or language. Be reassured: members of the Navajo Nation experience blue and green just like everyone else who isn't color blind.

… These biological facts of our cognitive life are completely consistent with Wittgenstein's insight that our language categories have such prototypical 'centers' (a 'good red', a 'good chair') but fuzzy boundaries (reddish-orange, a piano stool)." (8)

 

One aim of psychology and Cognitive Science is to discover precisely why we so often do understand such a correlation. There clearly is no physical reason that we are so influenced by this adopted linguistic metaphor. We conceive of pitches as having locations on a line, though neither sound nor light behave linearly at all. These lines are merely conventions of convenience, but ultimately imaginary abstractions.

Note that many cultures recognize and extract scales of seven notes. They may choose 5 (often in Africa) or even less. When cultures employ quarter notes, pitches between the notes of our twelve note chromatic scale (as in Middle Eastern music), these pitches are not treated like the others but lead the listener to more "stable" notes (9). Hence, they are not precisely what we would call a chromatic scale (all the notes). The vast majority of music throughout the world then employs a subset of 5-7 of usually 12. The construction of musical theories across the globe, whether explicitly formalized or unconsciously, like color perception, tend to delineate the perceptible spectrum using a very similar strategy, though the physical events that trigger these stimuli differ enormously.

Aesthetic and subjective issues aside, it is hard to top the impressive results obtained by Paul Bach-y-Rita and Carter Collins in the '70s. Theirs was a very different approach. Rather than look for a correlation that may or may not exist, they created conditions so that brains, with their impulsive organization of incoming information, would be forced to create/customize functions (and physical brain modules) in order to use the new information. In both cases, they substituted sight for touch using the brain's somatosensory map (basically the sense of touch). There is a map of our bodies in our brains deformed as larger neural territory that is necessary to correspond to more sensitive body parts. The "somatosensory homonculous" has enormous lips and very small ear lobes. Their idea allows for individuals to "remap" touch stimuli to visial stimuli by means of neuroplasticity (10), and absolutely blind people as well as blind folded subjects could identify objects and play "rock, paper, scissors.

Bach-y-Rita created a contraption that "displays" a 7 x 7 pixel image in a configuration of tiny jolts from a grid of electrodes placed on the tongue (11). Though Bach-y-Rita claimed that the low resolution was insignificant (12), even a cursory brush with computer graphics will show this is hardly the case (13). Nonetheless, his boast that "I can connect anything to anything." (14) is impressively justified.

Carter Collins faced the opposite obstacle. Though his invention was also an impressive one by any standard (15), it was less limited by a 100 x 100 pixel resolution. A grid of approximately 1/2 square inch covered the back. This is about as densely as the nerves on the back can locate though (16). There seems to be no ideal location on the body where the nerves are distributed densely enough over a wide enough area. Though the back offered a convenient spread of skin, especially given the size, this proves unwieldy. Subjects had to remain seated and stationary to accommodate the larger cameras and stand back then. The hands and lips offer far more information, but then we are back to a physical limitation of the electronics. These 2 legendary researchers have since worked together, though Bach-y-Rita's work has been far better publicized.

It should be noted though that there really is nothing like "visual stimuli" once inside the brain. The brain doesn't actually ever deal in images. It sends information of all kinds to the prefrontal cortex (among other places, but this is generally what where the action happens that we are talking about). The prefrontal cortex is a black box that does it's magic (we don't know much about how, but don't need to at this point) and assembles an updated mental model of the world from the stream of incoming bits of information (17). We tend to picture this world, and thus imagine that what we see, is a world "out there" that is intrinsically visible. The visable-ness is merely a byproduct of our brains, not something we can verify in the a priori universe. The same is true for sound and the other senses. Thus these experiments are able to exploit this scheme, all they need to do is fool the brain into thinking the incoming stimulus is something to picture in the mental model.

The drive to limit color palettes and musical scales in similar ways is not likely to be coincidental. Just as phone numbers cannot be any number of digits or we tend to name our children a number of syllables relatively normal for our culture, there is a concept called chunking (18). Humans tend to remember around 7 things. This number can be pushed or pulled in a variety of ways, the most important being culture.

While there is absolutely nothing similar about color and sound in the real physical world (even the concept of frequencies is a bit imprecise, albeit useful in conversing), the peculiar limitations in the way human brains process these has features which we can use. There is even debate if anything like qualia (the feeling that red-ness or the smell of burnt toast evokes in the mind) actually occurs, and is not simply a theoretical concept (despite Wittgenstein's insightful theorizing about this). However it may not be intuitively obvious, we can easily brush these notions aside. We can successfully draw parallels between the ways the mind receives a chosen sensory stimulus and another. Not the way they occur in the physical world nor the way the brain processes them, but the way they are subjectively felt.

No computer can predict what you will feel from looking at a painting by Monet as opposed to a painting by Lucian Freud obviously. Nor can it analyze a Beach Boys tune and a piece by Bartok and come up with rules to generate create a happier mood. But a computer can compare things like warmth, brightness, shapes/textures (the number of hard edges and softer edges). These things are likely to contribute to the mood, though we don't need to know what that specific mood is. Likewise, we can guess, employing the concepts of sensory substitution outlined above, that hues are comparable to scale degrees, how distant or usual a color is when compared to the general color scheme correlates to how far from the root of a scale along that scale is a given note. The result is neither a better or more emotive image or sound but one that is likely to trigger emotions in the same ways. The emotions need not be the same, but the audience is then able to compare the way one stimulus feels with another.

 

 

 

REFERENCES
(1) Bavelier, D. & Neville, H.: Cross-Modal Plasticity: Where and How. Department of Brain and Cognitive Sciences, University of Rochester (2002)
(2) Cytowic, R.: The Man Who Tasted Shapes. MIT Press, Cambridge (2003)
(3) Sacks, O: Musicophelia: Tales of Music and the Brain. p 171n, Alfred A Kniopf, New York (2007)
Godwin, J: Music, Mysticism and Magic A Sourcebook. pp 216—229, Penguin Group, New York (1986)
(4) Albers, J.: The Interaction of Color. Yale University Press, New Haven (1963)
(5) ibid, p. 80
(6) Lakoff, G & Nunez, R: Where Mathematics Comes from. ch. 2, Basic Books, New York (2000)
(7) Temperley, D.: The Cognition of Basic Musical Structures. MIT Press, Cambridge (2004) refereeing to referring to Pavel & Essens, p. 432. (1985))
(8) Hundert, E. Lessons from an Optical Illusion. pp. 169-170. Harvard University Press, Cambridge (1995)
(9) May, E. (Ed.): Music of Many Cultures. Chapters 14, 15. University of California Press, Berkeley (1980)
Levitin, D.: This Is your Brain on Music. p. 61. Penguin Books, New York (2006)
(10) Doidge, N.: The Brain that Changes Itself. chapter 1. Viking, New York (2007)
(11) Bach-y-Rita, P.: Brain Mechanisms in Sensory Substitution. Academic Press, New York (1972)
Bach-y-Rita, P. & Kaczmarek, K. & Tyler, M. & Garcia-Lara, J.: Form perception with a 49-point electrotactile stimulus array on the tongue: A technical note. Journal of Rehabilitation Research. (35) pp. 427-430. (1998)
(12) #10, p. 12
(13) Myler, H. & Weeks, A.: The Pocket Handbook of Image Processing Algorithms in C. Prentice Hall, Upper Saddle River (1993)
Gonzales, R. & Wintz, P.: Graphics Programming. Addison Wesley, Reading (1977)
(14) #10, p. 15
(15) Ornstein, R.: Psychology. pp. 255-256. Harcourt Brace Javonovich Publishers, USA (1985)
(16) Stafford, T. & Webb, M.: Mind Hacks: Tips and Tools for Using your Brain. pp. 27-31. O'Reilly, Sebastopol (2005)
(17) Solso, R.: The Psychology of Art and the Evolution of the Human Brain. pp. 254—259. MIT Press, Cambridge, MA (2003)
(18) Calvin, W.: How Brains Think. pp. 92—93. Basic Books, New York, NY (1999)

 

 

In 2007, judsoN completed a Faculty Fellowship at ITP (Interactive Telecommunications Program) at New York University/Tisch, graduated from Brown University in 1992, having studied under a former student of Stockhausen's. He programmed interactive artwork in 1996 with one of the first online galleries Ädaweb, relating sounds to visuals. Since then, roughly half of his artwork involves music, often sensory substitution. Multi-sensory works include a piece for the Arts Council of Mildura Australia, for the 809 Art District in China, at the Kennedy Center in Washington DC, with the Brooklyn Philharmonic and composer Eve Beglarian, for IMEB (the Institut International de Musique Electroacoustique de Bourges). He also contributed a chapter published in the Handbook on Computational Arts and Creative Informatics.

 

 

 

HOME

ARTICLES

NET GALLERY

---

---

 

FYLKINGEN'S NET JOURNAL

- © 2009 all rights reserved -