2 really nice pieces that are interested in the visualisation of invisible flows of information. These two aesthetic and sensory observatories for the perception of this parallel reality have been developed by Golan Levin and Zachary Lieberman in collaboration with the Ars Electronica Futurelab in Linz, Austria.

The Hidden Worlds of Noise and Voice is an interactive audiovisual installation, or, alternatively, an augmented-reality speech-visualization system. Its central theme is the magical relationship of speech to the ethereal medium which conveys it. Participants in Hidden Worlds are able to “see” each others’ voices, which are made visible in the form of animated graphic figurations that appear to emerge from the participants’ mouths while they speak. In the installation, visitors wear special see-through data glasses, which register and superimpose 3D graphics into the real world. When one of the users speaks or sings, colorful abstract forms appear to emerge from his or her mouth. The graphics representing these utterances assume a wide variety of shapes and behaviors that are tightly coupled to the unique qualities of the vocalist’s volume, pitch and timbre.

RE:MARK is an installation for two participants which likewise presents an interactive visualization of its users’ speech. Unlike Hidden Worlds, which primarily attends to the noiselike aspects of vocal sounds, RE:MARK shifts this inquiry towards the more symbolic domain of the spoken and written word.

In RE:MARK, sounds spoken into a pair of microphones are analyzed and classified by a phoneme recognition system. When a phoneme is recognized with sufficient confidence, the written name of the phoneme (for example, oh, ee, ah, etc.) is projected on the installation’s display. If the user’s sound is not recognized by the system’s classifier, then an abstract shape is generated instead, based directly on the timbral characteristics of the vocalization.

As the visitor speaks, the corresponding written phonemes and abstract forms are rendered as silhouettes, and appear to emerge from the shadow of the speaker’s head. A computer-vision system permits the visitors to sweep these forms across the screen with the shadow of their body. The result is a playful and revelatory illusion, in which the installation’s visitors become actors in a shadow world of reactive cartoon language.