“Our body applies itself to space like a hand to an instrument, and when we wish to move about we do not move the body as we move an object. We transport it without instruments as if by magic, since it is ours and because through it we have direct access to space.”

Maurice Merleau-Ponty

Since the 1970s, artists and filmmakers around the world have been experimenting with full-body interactions that go beyond GUI metaphors and relate more to the visceral experience. Predecessors such as Peter Campus (Featured image) and Scott Snibbe (Figure 1) got motivations from the philosophy of phenomenology and the desire to understand the complexity of identity-building. In Snibbe’s words, his works represent one extreme of this development which he defines as “social immersive media” (the other extreme being the VR system): “immersive media that favors interaction in a shared social space using a person’s entire body as the ‘input device,’ unencumbered by electronics or props.”¹ Such experience is considered a subset of augmented reality that is “emotional, social and physical.”²

Our group believes that it is time now to rethink this long-existed and yet immature media. The contemporary society has witnessed a growing concern of alienation, and the pandemic worsens the situation. How can our embodied interaction design respond to this issue and influence people’s behaviours and experiences? In Prototype 1 exhibited at Project Fair, we marked out the proximity of people in public space visually and aurally to study the everyday social behaviour (Figure 2). With the keyword “distancing,” we then quickly sketched a series of design ideas on how we may engage in the process (Figure 3).

Early examples such as Boundary Functions (Figure 1) and Braincoat (Figure 4) require multiple users to be physically present in the same space for interactions; what about extending the scale of interaction to urban level and do it in a crowd-sourcing way? Relevant examples include Do Not Touch by Studio Moniker (Figure 5) and Exhausting A Crowd by Kyle McDonald (Figure 6).

This week we realised the functionality of transporting our body skeleton captured by webcam + pose-estimating ML model to Youtube live streaming video (Figure 7). Later we would like to have all the users’ body movements recorded and overlaid with the live video (Figure 8). Following our pervious explorations on motion-sound translation, the playful interaction is expected to be audible. In this manner, the users are not merely observing but actively engaging in the life elsewhere and physically interacting with other strangers remotely.
Launching soon…

1. Scott S. Snibbe and Hayes S. Raffle, “Social immersive media: pursuing best practices for multi-user interactive camera/projector exhibits,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’09) (New York, USA: Association for Computing Machinery, 2009), 1447.
2. Ibid.

---

Reference

Snibbe, Scott S. and Raffle, Hayes S. “Social immersive media: pursuing best practices for multi-user interactive camera/projector exhibits.” In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’09), 1447–1456. New York, USA: Association for Computing Machinery, 2009. DOI:https://doi.org/10.1145/1518701.1518920

List of Figures

Featured Image: Peter Campus, Interface, 1972.
Figure 1: Scott Snibbe, Boundary Functions, 1998.
Figure 2: Project prototype one – 03:42:58.
Figure 3: Sketchy design ideas on “distancing.”
Figure 4: Diller Scofidio+Renfro, Braincoat, 2002.
Figure 5: Studio Moniker, Do not Touch, 2013.
Figure 6: Kyle McDonald, Exhausting A Crowd, 2015.
Figure 7: Current working progress – transport and overlay.
Figure 8: Sketchy ideas on future developments.