Introduction

Subjective experiences in everyday phenomena are new encounters of space, objects, and memories in the real world. Therefore, seeing something entails an internal perception in terms of a phenomenal dimension. Images have the possibility of various transformations through conceptual and formative imagination. Through intimate interaction, the individual has a new relationship with the object he is looking at. According to Bachelard (1964), by the material of the object presented to us, we can create an infinite amorphous image through thought, which is not instantaneous and slowly changes in our consciousness and approaches its essential concepts. It can be said to collect the characteristics of time perceived in phenomena and movements and to recognize objects in subjective time that adds imagination to the time.

Compared to the past, space, objects, and time we encounter become more and more complicated. The edge between the real space and the virtual space is getting vague. Time in both spaces is intertwined. As the virtual space is getting real, an avatar in the virtual space might be perceived as the self. For some people, the self in virtual space is considered more important than the self in reality. Kwinter (1996) warns of the possible negative consequences of virtual reality, which blurs the boundary between reality and virtuality because it is not yet a fully simulated environment. Looking at the current situation, this prediction is somewhat realistic. Furthermore, since objects are getting wiser, they are likely to be treated as being with consciousness rather than the object itself. The concept of the self and the being is getting confused. This situation naturally leads to anxiety about existence and self-existence. From these contexts, several questions about existence came up to me. Especially, one of the questions has arisen in my mind.

“Can people feel the self through the everyday object or natural medium in physical space?”

It was started with the intention which leads people to rethink about themselves in real space. In this paper, this question extends to the concept of self-presence and memory. As a way of answering the question, the purpose of this paper is to explore the theoretical foundation for experiments and describe the process of creating experiences about the concept of self-presence and memory in real space by using the method of ‘installation’.

Process

As mentioned above, this paper focuses on experiments to create experiences that could encourage people to reconsider the relationships between the self, the other, object, space and time. As the process of the experiments, the concept of self-presence being used in academic fields will be the basis. Since this concept deals with relationships between the self and perceptual experiences based on body schema, it might support relevant theoretical foundations for delivering various kinds of phenomenological experiences that can be induced during the transition of perception through body movements over time. Also, the thought of memory will be referred to.

In this experiment, the body is used as the fundamental sensory element. In the book ‘An architecture of seven senses’, Pallasmaa(no date) refers to body as a significant sense with five senses such as vision, sound, scent, touch, and taste. He argues that, through body schema, a person can feel the self in the environment as a whole, through bodily identification, a person can feel the being and do interaction with the being. According to Merleau-Ponty (1945), who insisted on phenomenology, humans compose a space in a way in which they perceive objects based on the body and add creative meaning to them. Thus, the approach to space through the body seen from phenomenology means that a person can create rich emotional experiences in space by inducing the activation of various senses and perceptions of human beings. Merleau-Ponty emphasizes the fact that staying in the world is the self as a living body not the self as a spirit. Perception through the body is not a simple linear perception but a perception through interaction with space or objects. This perception of space through physical perception presupposes experience. The objects in space and the environment interacting with them are intertwined to create new meanings and open possibilities of various interpretations.

Another primary element in this experiment is time. Merleau-Ponty notes that the time captured by perception has abundant depth, and that depth is formed from the body’s experience. In other words, this depth comes from the flow of phenomena and body movements over time. The time in art is not the time as a physical concept without the subject, but the time of the psychological concept in which the subject is included. It could be an expression of time moving according to the change of perception. Based on this concept of time, today’s contemporary art is interpreted and developed in various ways through its own domain beyond the representational characteristics. In addition, the concept and expression of the space are also interpreted and made in various ways. Memory is the first feeling in the process of perception, and at the same time, it is transformed by the moment of cognition. The individual understands the artwork based on his experience and memory and produces another transformed memory. This memory is transformed again in time continuously.

Light and sound are dealt with a significant natural medium in this experiment. However, this experiment does not focus on the exploration of nature in light or sound but takes notice of them as fundamental mediums for projecting the existence of the self and interacting with others. Especially, moving light(head movement stage light) was chosen as a fundamental medium. There could be two ways of seeing this light(head movement stage light). The one is to look at this light as a physical object which is a robot with light. It is because the motor in light can serve basic robotic(mechanical) movements in physical space. The other is to see as a physical object which is the specific part of physical surface identified by traits of light. It is because light can serve basic physical traits(geometry, brightness, color) on a certain area of the surface with texture. As another medium, the sound is used in experiments. Sound could be considered as an obscure medium like a light. This ambiguity requires the concentration of a person and could help to create a variation.

In terms of experiments, based on the concept of self-presence, the ‘installation’ method of using light and sound is used for the phenomenological reproduction of the self and incomplete memory. This could lead to the experience of the self-presence and the memory of space through the transformation and expansion of the body.

To put it simply, firstly, discussion including literature review about the concept of presence, self-presence, and memory has proceeded. Secondly, keywords for experiments are derived and case studies relating to these keywords for experiments are conducted. As the final phase, practical experiments through light and sound for exploring experiences based on conceptual layers of self-presence have proceeded.

 Discussion

_Presence

When a person faces “The Seven Heavenly Palaces” (Fig 1) by Anselm Kiefer (1945-), a concrete building can be an object that brings the past into the present rather than a simple object. The individual who exists in the present can feel the time of the past and the pain in the past through the object. In other words, people could feel the presence of the past through objects. As such, the existence of space or objects can cause subjective experience and another memory with special meanings based on the individual’s memory and experience. This experience might be understood as a poetic experience. In other words, the poetic experience through space or object can be regarded as an aesthetic experience that can empathize and feel meaning beyond time and space through the subjective perception of space or object. In art, minimalism uses an approach that induces a poetic experience using fundamental objectivity in order to make people face reality from a subjective perspective. It is a way of presenting an essential being and suggesting subjective interpretations, feelings, and experiences based on the experiences and memories of each individual.

Fig 1. The Seven Heavenly Palaces (Anselm Kiefer, 2004-2015)

This chapter examines how the term presence is defined and handled in various disciplines. The concept of presence is being used in the need to approach the concept of physical objects or virtual objects that can replace or mimic humans in academic fields such as robotics and virtual reality. The term ‘telepresence’ referring to the experience of being there in the mediated environment was mentioned by Steuer (1992). The term ‘presence’ was defined as the sense of being in an environment and telepresence was described as the experience of presence in an environment by means of a communication medium. Biocca (1997) refers to this term as the sense of being there in a physical, mediated, or imaginary space. He describes how virtual reality technology tries to induce a feeling of physical presence through sensory engagement, motor engagement, and sensorimotor coordination. Lombard & Ditton (1997) define this term as the perceptual illusion of non-mediation. They limit this term to the media environment and try to explain the causes and effects of presence. Lee (2004) defines this term as a psychological state in which a virtual object, a social actor, or a virtual self is experienced in a sensory or non-sensory way. He tries to describe this term as artificial objects that are related to the real world or not related to the real world. Even if the definition of this term varies and utilized for a disparate purpose, it is commonly mentioned that the feeling of presence occurs when the mediated experience is treated as a real experience.

In addition to robotics and virtual reality, this term is also being used in the neuroscience field. Damasio (1999) provides a framework for the concept of the self which integrates the interrelationships between body and consciousness. This framework consists of two levels of consciousness that affect an experience. Core consciousness in the first level appears through interaction with objects. The second level, extended consciousness, occurs when past experiences and anticipated futures are recorded in memory and reproduced for generating a sense of self-discovery. According to him, while core consciousness can govern the action of the current moment, extended consciousness can affect all information related to future actions, including identity and individual perspective. Waterworth and Waterworth (2003) use this framework to describe the concept of presence as an evolutionary mechanism that can help humans distinguish the inner world and the outer world. While the inner world can be facilitated by an expanded consciousness, the external world is facilitated by core consciousness. On the basis of this thought, they argue that the state of presence can be induced by deceiving the core consciousness in such a way as to interact with the object in the mediated environment as if it were a physical object in the real space.

Damasio (1999) describes the relationship between core consciousness and extended consciousness by extending as three distinct levels of self. According to him, proto-self senses the physical existence of an individual by mapping and recognizing the physical structure that he is not conscious of. Core-self refers to an emotional state that can be created and conscious when encountering the self or object in the environment. Lastly, Extended-self means that an individual is conscious of the self or identity based on personal characteristics or memories of past experience. Riva, Waterworth, & Waterworth (2004) use the three levels of self in Damasio’s (1999) to define the three levels of presence. These applications of the framework in Damasio (1999) generally provides a crucial viewpoint on the concept of presence related to the experience of the mediated environment. Since this framework focuses on the concept of self, it has the potential to contribute to the understanding of self-expression in the mediated environment. Likewise, this framework will be the foundation for the development of ideas and experiments.

_Self-presence

Bachelard (1964) notes that poetic images are created through the subjective consciousness of individuals based on phenomenological objects such as houses. It is mentioned that a house is created through memory and experience and that people can understand the individual through a phenomenological understanding of the house in which the individual lives. Space or object can have an identity through personal experience and memory. When a person encounters “The House of the Suicide and The House of the Mother of the Suicide” (Fig 2) designed by John Hejduk (1929 -2000), the individual can feel the pain of the person who committed suicide and the mother who has the child who committed suicide through the object. It could be a moment when empathy with the object occurs. The memory or experience of an individual meets an object, and the poetic image is projected onto the object, and emotion is evoked through the object. As such, an object could exist as the other or the self on which identity is projected like looking at myself in the mirror, rather than a simple object.

Fig 2. The House of the Suicide and The House of the Mother of the Suicide (John Hejduk, 1991)

In academic fields, the concept of self-presence and presence share viewpoints. Biocca (1997) and Lee (2004) illustrate this concept as a psychological state in which the imagination is treated to some extent as the actual experience. Presence is more related to the environment, and self-presence is more related to self-expression. Biocca (1997) uses the term ‘self-presence’ to describe mental model, psychological, emotional states, identity in the mediated environment. Lee (2004) uses this term as a psychological state which is experienced as a real self through the virtual self with referring to body schema as well as identity. While the experience of presence is related to feeling existence in the mediated environment, the experience of self-presence is related to self-expression in the mediated environment. Some empirical studies deal with this concept in a divergent direction. As one of these research, Ratan, Santa Cruz, and Vorderer (2007) investigated self-presence as a concept related to identity. The process of this research is based on the definitions introduced by Biocca (1997) and Lee (2004) and the framework of Damasio’s (1999).

Damasio’s (1999) framework is easily mapped to the concept of self-presence. Proto-self is related to body schema, core-self is related to emotional state, and extended-self is related to perceived traits and identity. Thus, self-presence is defined as the degree of involvement in the mediated environment. The separated concept as three levels of self-presence could be more significant than the concept of self-presence as a whole. Table 1 summarizes this framework.

Table 1. Layers of self-presence(Damasio, A. R., 1999)

When integration with the self-expression occurs, the individual interacts with the object in the mediated environment without being aware that the self-expression is not real. The experience of proto self-presence can be evoked by controlling mediated objects as bodily extensions. In order to do this, the interface that can control the object as the self is needed. During controlling the object projecting the self, the person could feel that it is a part of the body schema. This kind of experience could be related to proto self-presence. Interactions between an individual’s body and objects in the environment yield emotional responses (Damasio, 1999). Armel, & Ramachandran (2003) found that when a fake body part associated with a body schema was harmed, the individual responded emotionally. Thus, if self-expression is integrated into a body schema, the interaction between self-expression and objects also leads to emotional responses. This kind of experience could be connected with core self-presence. In the case of women who choose male characters in online, they are more likely to be more masculine in online behavior than the same women (Huh & Williams, 2010). Through self-expression and behaviors, people could show their identity. This could represent the experience of extended self-presence.

Three levels of self-presence are necessary to be considered separately. However, Damasio (1994) mentions that the dualism of body and consciousness could be dangerous and need to be an integrated experience. It means that the three levels need to be hierarchically linked. Also, the temporal aspect of self-presence needs to be considered. Proto self-presence occurs while controlling the self-expression. The self-expression of body parts should be appropriately positioned for each individual to integrate into the body schema (Armel & Ramachandran, 2003; Tsakiris & Haggard, 2005). In order to integrate the self-expression into the body schema, the part of the body must be integrated with movements of self-expression. Thus, if the person loses control of the self-expression, the proto self-presence pauses within a few seconds if not immediately. At the same time, core self-presence appears during the interaction between the self and the mediated objects or environment. However, the feeling of core self-presence may continue even after the person stops controlling the self-expression. For example, a person experiencing core self-presence may be able to feel the emotion to positive or negative events that occur to the self-expression, in spite of exiting from the mediated environment. Thus, core self-presence is likely to last longer than proto self-presence. If someone creates the self-expression with a specific identity, the created identity could be persistent. Because this image could be saved in long-term memory. Therefore, we can say that extended self-presence can last longer than two other levels.

To sum up, proto self-presence is related to the representation and control of body schema. Core self-presence refers to the interaction between the self and the other(objects or people). And extended self-presence is related to creating or assigning a personal identity based on memories and past experiences. Proto self-presence is the hardest to reach and has the shortest duration.

_Memory

The facade of “Louvre Abu Dhabi” (Fig 3), designed by Jean Nouvel (1945 -), was constructed through a superposition of geometric shapes that allow light to pass through. The light reaches the interior through the overlapping facades and appears as a distorted form of light, which changes continuously over time. As such, memories, the first images accepted by the senses, could be transformed throughout layers of time and generate new images continuously.

Fig 3. Louvre Abu Dhabi (Jean Nouvel, 2017)

Memory is the storage of sensory experiences obtained by observing or interacting with the object or the environment. The acquired experience forms the memory affected by the imagination being based in part on past experiences and memories. Individual sensations could affect the experience. Since sensory interventions are made by imagination, it could generate the recreated memory completely inconsistent with the real phenomenon. The memory used as the theme in this experiment is regenerated by visual and acoustic images based on the body. In other words, it focuses on reproducing deformed states that differ from the original memories over time.

According to Bergson (1889), time is not able to reveal its exact position on the space, and the substantial concept of time is the freedom and creativity to enable newness and change continuously. He names this understanding about time as the term ‘duration’. The term ‘duration’ is to understand time as an experience, not a unit of measure. It means that there are traits of time that cannot be dealt with analytically and it could be considered more critical. First of all, he emphasizes the characteristics of personal, subjective, organic and psychological time. In terms of memory, Bergson (1896) considers consciousness itself as memory. According to him, memory is divided into two parts. One is the habit-memory, which is the memory that habitually regenerates the experience gained by the continuous movements of the past. The other is the image-memory. image-memory means memory in which all past experiences are uniquely and physically preserved as the form of images without repetitive efforts. This experience can never be repeated the same and is represented by images, not actions. Image-memory is created by activating the memory of the past. He names the potentially remaining aggregate of the past as pure memory. He argues that the perception of the present is conceptual and fluid, and the memory of the past is conceptual. Perception by the movement of the body includes not only the perception getting from the external world but also the perceptual emotions obtained by the body. So, he insists that memory is embedded in all perceptions.

The reproduction of memory can be understood as the re-expression of the past situation that is stored. According to Deleuze (2005), the representation of memory blurs the distinction between subjective and objective. This ambiguity confuses what is the imagination and what is the real image because of the uncertainty and the unidentifiable. The memory by sensory experience is involute and independent, and the representation of memory is affected by time.

Case studies

From the discussion about self-presence and memory, three keywords for experiments were driven. The first keyword for the layer of proto self-presence is self-reflection or self-representation. The second keyword for the layer of core self-presence is interaction with memory. The last keyword for the layer of extended self-presence is multiple overlaps as a concept of accumulated memories. Therefore, the works analyzed in this paper are classified into three categories, ‘Self-reflection’, ‘Interaction with memory’ and ‘Multiple overlaps’ according to the derived keywords. ‘Self-reflection’ examines installations in which movements are expressed based on the body. Through this, the expression methods in which the body is physically reconstructed in the real space are explored. ‘Interaction with memory’ focuses on temporality by investigating installations that attempt to combine the past with the present. ‘Multiple overlaps’ explores installations having the potential of creating and interpreting new images through the overlapping of space, objects and time.

_Self-reflection

In this chapter, except for works using general mirror reflection effect or shadow effect or digital screen, mechanical installations dealing with the concept of self-reflection through objects are mainly investigated.

Fig 4. Wooden Mirror(Daiel Rozin, 1999)

Daniel Rozin (1961-) is an interactive media artist who created Wooden Mirror (1999) (Fig 4). Based on the concept of self-awareness, his works explore the boundary between analog and digital. It reflects the individual through the physical form rather than the reflection effect of the mirror. Since the figure of the individual is projected on the work in real-time, it actively induces the movement of the body. The person’s presence completes the work, and the formative change is also made by the body. Representing the person in real-time, if there is no movement of the body, such self-reflection will not be achieved, and the illusion which there is the self in the artwork cannot be reached. The formative expression by temporality appears in rhythm and illusion. In this work, there is a time delay in the process of expressing the movement of the person. Due to the shape of the body moving at a slower speed than the actual movement, a calm rhythm is created and arouses various imaginations. After this work, he created Trash Mirror (2001) (Fig 5) and Pom Pom Mirror (2015) (Fig 6) as works of a similar concept.

Fig 5. Trash Mirror(Daiel Rozin, 2001)

Fig 6. Pom Pom Mirror(Daiel Rozin, 2015)

Random International (2005-) is a studio that is proceeding various experimental works within the boundaries of contemporary art. The conceptual experiments including social issues are expressed through various media. Fragments (2016) (Fig 7) is a frame consisting of about 200 pieces of small mirrors that act as normal mirrors in the static state. If it recognizes the presence of a person, the angles of the small mirrors are changed by the motor. It distorts the person’s shape and creates another self on the frame. In this way, since it makes it hinder to perceive the self completely, it could lead the person to rethink the real self.

Fig 7. Fragments (Random International, 2016)

Todd Bracher is operating a studio in NewYork as a product designer and strategist. He commissioned by Humanscale and progressed Bodies in Motion (2019) (Fig 8) for branding the company in Milan. Since this company is a furniture company that value ergonomics, this project is based on the concept of body and movement. Sixteen moving-head-lights were used for tracking each joint in the human body. The technical approach of this installation is partially similar to the interactive system developed in this experiment.

Fig 8. Bodies In Motion(Todd Bracher, 2019)

_Interaction with memory

In this chapter, the installations about the coexistence of the past and the present relating to self-reflection are primarily investigated.

Daniel Dan Graham (1942-) is an American artist, writer, and curator. As an artist, for his works, he uses media that integrates photography, video, performance, glass and mirror structures. In Present Continuous Past(s) (1974) (Fig 9), the past is brought to the present through the time delay of recorded video and the reflected image in the mirror. The mirror reflects the present. The camera records the space in front of the mirror and the reflected image of the mirror. The monitor on the opposite side of the mirror displays the recorded video after 8 seconds. Therefore, the person can see the recorded video 8 seconds ago and the image reflected from the mirror. The image reflected in the mirror shows the monitor again, and the video of this monitor is the video 8 seconds ago again. In this way, a series of the time of the past is brought into the present.

Fig 9. Present Continuous Past(s) (Dan Graham, 1974)

Rafael Lozano-Hemmer (1967-) is a Mexican-Canadian electronic artist who explores the relationship between humans, time and space based on various media. People on People (2010) (Fig 10) is one of the works dealing with the concept of existence and time. People’s shadows are projected onto the wall, and the images of the past are projected inside the shadows so that shadows of the present and the images of the past coexist. Therefore, people can see other people’s images of the past represented in shadows, and their images are also recorded and represented in someone else’s shadows.

Fig 10. People on People(Rafael Lozano-Hemmer, 2010)

Tetsuro Nagata’s Computing an Identity (2009) (Fig 11) also deals with a similar concept. It consists of three works, ‘Peter Pan’s Shadow’ is an interactive floor that can interact with the shadows of people in the past by delaying the movement of the shadows. ‘Palimpsest’ is a mirror that merges the image having time delay with the present image. ‘Rose Window’ displays the person who participates as a present being and the person who participate as a shadow of the past.

Fig 11. Computing an Identity (Tetsuro Nagata, 2009)

Self and other (2016) (Fig 12), another project of Random International (2005-), is an installation with a three-dimensional structure of lights that expresses the shape of the body with minimal lighting. The light is expressed with time delay, not in real-time, which leads a person to interact with the self of the past being represented by lights.

Fig 12. Self and other (Random International, 2016)

Since my experiment for core self-presence focuses on the interaction between the past and the present, the temporality is critical. In my experiment, the past and the present coexist through light and interact through sound. If the stimulation and reactions of a person are provoked through the way of bringing the memory to the present, even if there is no active interaction, it could be considered as an interactive activity.

_Multiple overlaps

In this chapter, installations applying the method of multiple overlaps through light are principally investigated.

Laszlo Moholy-Nagy (1895-1946) is an artist who pursued a new perception of space and light. The artwork he pursued was to experiment with the intrinsic power of the light that creates the form. He began to recognize light as an object and treated it as a formative material like color and movement. He constructed an imaginative space through the movement of light and transparency. In the plane work, he focused on the effect of transparency by the overlap of shapes and expressed the rhythm of the interaction of the form and color floating in space. In terms of spacial work, the Light-Space Modulator (1922-1930) (Fig 13) creates a fluid and overlapping spatial dimension by combining light and space. It could generate subjective experience through intervening human senses in the phenomenological experience. In other words, it shows the transformation and reinterpretation of space through experience. He has noticed the immaterial light as a potential medium in expressing perceptual experience. He emphasizes that light could extend our perception of time and space.

Fig 13. Light-Space Modulator(Laszlo Moholy-Nagy, 1922-1930)

Robert Irwin (1928-) is interested in the perceptual experience of light and space. Based on the phenomenological minimalism, he forms a concept about the person’s body movements, the change of vision and the perception of space. His works awaken the sense which is inherent in the person and makes them feel illusory space. One of his masterpieces, Untitled (1965-67) (Fig 14), the disc series is manipulated to generate four shadows on one disc through two lights on the ceiling and two lights on the floor. The overlapping of the real shape and shadow shows a floating effect in the space. He tried to extend the visual perception to the whole space by experimenting with subtle phenomena of light and shadow.

Fig 14. Untitled(Robert Irwin, 1965-1967)

Julio Le Parc (1928-) is a pioneer in the field of op-art and kinetic art and has continued to create works using the overlay of light leading people to feel illusion. One of his works, Lumière envibration (1968) (Fig 15), uses the moiré effect of overlapping lines to deliver a feeling of vibration, although it is actually a fixed installation. This effect allows the individual to experience the illusory image obtained through the process of subjective perception in the real space.

Fig 15. Lumière envibration(Julio Le Parc, 1968)

Olafur Eliasson (1967-) is an artist who creates light and color based on time and space. Your Museum Primer (2014) (Fig 16) is expressed as myriad shapes and colors through overlapping light by a sculpture rotating at irregular speed and direction. The sculpture has a fluid motion and shows a phenomenological variability that expresses the shape of light. The versatile forms of light in the continuity of scenes are collected through individual perception, it could lead to personal experience. It is the subjective spatial perception of the individual based on the conjunctive activity of the consciousness perceived through the transition of abstract form by lights. The movement of light over time is the basis for discovering metaphorical languages and meanings. Since metaphysical and immaterial light is fluid, it can change the unique form of the object and help to create a subjective interpretation. In this way, light has the role of the medium which redefines the relationship between space and the individual and delivers personal experiential value.

Fig 16. Your Museum Primer (Olafur Eliasson, 2014)

The direction in my experiment is to create a visual and acoustic image representing identity by accumulating time. Light is considered as the medium to do the role of forming the relationship of time with the self in order to help to create a unique identity and meaning for the individual. The fluid flows of light which express the self can connect space, it creates new images and forms myriad variations. Light has transparency that can simultaneously perceive that it exists in another dimension beyond the actual space. The image of light expands through imagination and memory and is located on the inside of consciousness. In this way, it creates a unique meaning for the self. Based on the subjective interpretation, the shape and time of light are reconstructed and reinterpreted. In this way, aesthetic experience can be generated by combination, movement, and transformation of space and light over time. This space might be an interactive space having the potential to be interpreted in various meanings for the individual. Table 2 shows a list and characteristics of cases investigated.

Table 2. List of cases

Experiments

Before starting experiments in real space, in order to refine the direction, visual experiments in virtual space relating to the keywords derived from the framework of self-presence have proceeded. The [Chair] series is visual experiments about the interaction of everyday objects with humans, space and time. [Chair] (Fig 17) is a visual experiment of proto self-presence through the movement of everyday objects over time. [Chair] (Fig 18) is a visual experiment about the interaction between proto self-presence and other objects or spaces. [Chair] (Fig 19) is an experiment about the fluid image generated by creating and overlaying a new layer over time.

Fig 17. Chair(Visual experiment for proto self-presence)

Fig 18. Chair(Visual experiment for core self-presence)

Fig 19. Chair(Visual experiment for extended self-presence)

The first visual experiment is about whether an individual can recognize the self through the object applying the body schema. The second visual experiment is about the interaction between the virtual self and other objects or environments. In the last experiment, the potential of expression is explored through the composition of multiple layers based on the body schema over time. The similarity of the expression can be found in the direction and method of constructive spatial exploration of El Lissitzky (1890 – 1941). In particular, the composition of the space through his continuity of image is related to the experiment that expresses the fluid object and space through the overlaps of time.

In addition, since experiments apply the way of approaching the essence based on the minimal shape and basic movement like investigated some cases, I can say that these experiments were influenced by minimalism for finding the essential quality. The methods pursued by minimalism might be as follows. Firstly, in order to approach the essence of beings, the way of constructing the existence itself is pursued. It has evolved into a formative method of constructing the art with the minimum requirements for expressing the existence of objects or spaces. Secondly, an attempt was made to use the least amount of material in order to represent the essence. As a result, the object itself, which revealed the physical properties of the material, was presented as a completed artwork. Lastly, minimalism has been tried to find essence by presenting actual space, highlighting spatial concepts as important topics in contemporary art.

The experiments could be connected to the experiments of constructivism and minimalism in art. However, these experiments are focusing on the medium that does not have a specific form such as light and sound rather than materiality or objectivity. In other words, compared to cases investigated, the experiments aim to elicit visual and acoustic experiences based on subjective perception through more obscure media.

_Experience for proto self-presence

Firstly, the development of the lighting system for experiments of the concept of proto self-presence was progressed. The first question for the experiment about proto self-presence is whether people could feel the self through lights in real space. The level of body representation through light was chosen as a factor for this question. It is because people could start to feel themselves from objects through their body schema. Body schema is about the representation of the positions of body parts of myself in space. And, it is updated during body movement. And elements to express the level of body representation(Fig 20) in this lighting system were defined. It includes body tracking method, expression method, type of light, number of mapping positions or degrees of body parts and number of lights.

Fig 20. Elements for the level of expression

In terms of tracking methods, there could be two ways for the physical expression of the body schema. The one is to track the positions of joints of the body. The other is to calculate the degree between joints of the body. Fig 21 and Fig 22 show processed simulations with different methods for tracking body movements in three-dimensional space. In terms of economical aspect in the robotic system, the method tracking degrees could be more efficient. It is because fewer motors are needed to express body movements.

Fig 21. Simulation for tracking position in 3d space

Fig 22. Simulation for tracking degrees in 3d space

As the experiment, an installation was developed. [There] is an interactive light installation for exploring an experience about the concept of proto self-presence. Basically, this installation started out as a question of whether people could recognize themselves from the light. The various attempts have been made from the abstract expression to the representational expression using the characteristics of light. On a certain level of expression, it could induce an uncanny experience of finding themselves in unexpected objects. Fig 23, Fig 24 shows the diagram of the lighting system for experiments about feeling proto self-presence.

Fig 23. Technical diagram for [There]

Fig 24. System diagram for [There]

And, simulations based on a different level of representation and elements affecting these levels have proceeded. Fig 25 shows one of the simulations.

Fig 25. Simulation for high-level representation in [There]

For example, the upper simulation (Fig 25) shows a high level of representation of body schema. And below simulations (Fig 26, Fig 27) show a low level. In these simulations, even though both levels are low, separate mapping logic is applied to the system. In the second simulation, four number of body parts are mapped to one light depending on body movements.

Fig 26. The first simulation for low-level representation in [There]

Fig 27. The second simulation for low-level representation in [There]

From this experiment, some ideas could be driven. The first one is that movements of body parts could be more crucial than body shape to reach self-presence. The second one is that movements of the arm or leg are critical. The last one is that, even at a low level of representation, there could be a way to reach self-presence.

_Experience for core self-presence

Recognizing time is a critical point of this experience. Therefore, the next question for exploring the concept of core self-presence is about time. The first question about temporality is whether people could feel the past through lights in real space. As a factor for this question, the length of the recall period was chosen. Because of the capacity of short-term memory, people have a limitation of memory to be able to remember at the present. Therefore, the length of time recalling body schema could affect the perception of the past. It could be critical to figure out the appropriate length of recalling time for people to be able to feel the past through light. Fig 28 shows the system diagram for experiments about feeling the past.

Fig 28. System diagram for an experiment about feeling the past

Based on this system, different lengths of time recalling body schema were simulated. Fig 29 shows one of the simulations.

Fig 29. Simulation about feeling the past

People might start to feel the past by remembering the past. But, when people look at their past, if people cannot recognize it as their past, even though it is their past, it might be difficult to feel the past. The longer the length of the recall period is, the harder it is to recognize the past. On the contrary, if the time is too short, it could be also difficult to recognize the past.

Another question about time is whether people could feel the present through lights. As a factor for this question, the perception of the past was chosen. It is because it might be hard to feel the present without consciousness of time. Even when people look at the self in the present, it might be difficult to feel the present. However, if people look at the self in the present with the consciousness of time, it might be helpful to lead people to feel the present. The one way evoking us to feel time might be looking at the self from the past in the present. Fig 30 shows the system diagram for experiments about feeling the present.

Fig 30. System diagram for an experiment about feeling the present

These simulations in this lighting system are quite simple. One simulation is to make a person to look at the self in the present without any clue of time. Another simulation is to make a person to look at the self in the present during looking at the self from the past. Fig 31 shows one of the simulations.

Fig 31. Simulation about feeling the present

If people can perceive the self from the past, it means that people could feel the past and time. In this way, if people can perceive the self in the present with the consciousness of time, it could drive people to feel the present. And it could be the starting point for the experiment about the interaction between the present and the past, namely interaction with memory.

As a final experiment about core self-presence, another installation was developed. [The here] is an interactive light and sound installation for exploring an experience about the concept of core self-presence. It focuses on leading to uncanny experience which could be induced by the interaction between the self in the present and the other on a different timeline through lights and sound. As a person approaches, lights begin to project the person’s figure and the person can see the self in lights. And it generates the sound based on the movement of body parts. The lights remember the person’s movements and the sound until the person leaves. After leaving, lights start to project the movements of the former person and play the sound generated by the former person like recalling. It could be the moment that people can encounter with the other from the past through lights and sound.

As the next person approaches the figure of the former person being expressed by lights, the person’s figure in the present is represented by a different type of lights and it is overlaid with the figure of the former person from the past. The person who sees the self-expression by lights could be affected by lights and sound representing the former person from the past. The next person in the present can modulate the sound generated by the former person in the past. In this way, the self who is in the present can interact with the other who was being in the past through lights and sound. Fig 32, Fig 33, Fig 34 show diagrams relating to this installation.

Fig 32. Technical diagram for [The here]

Fig 33. System diagram for [The here]

Fig 34. Interaction diagram for [The here]

In this installation, the sound is considered as another significant medium for interaction. As the sound is generated by the movement of the body, it could drive us to feel the self through the sound. Also, the sound could be a role as a medium for interacting with the past in a way that modulates the sound generated in the past through body movements.

Fig 35. The logic for generating sound in [The here]

Fig 36. The logic for interaction in [The here]

Fig 35, Fig 36 shows how the sound is generated through the movement of the body. After checking the motion of each body part, the largest motion among the motions is selected and a specific note is assigned according to the angle of the motion. Fig 37 shows how the movement of the body in the present modulates the sound that was generated in the past. The current movement could affect the past movement in such a way that the movement of the body in the present is added to the movement of the body in the past. This logic is applied to the sound as well.

Fig 37. Simulation for Interaction with the sound of the past in [The here]

Fig 38. Interaction with the self of the past                    Fig 39. Interaction with the other of the past

Fig 38 shows a person looking at the self in the present who is encountering with the self in the past. When a person leaves the place, lights recall the movements of the past with red wash lights. Fig 39 Shows the next person looking at the self in the present who is encountering the other in the past.

_Experience for extended self-presence

Lastly, as the final experiment, an installation relating to the concept of extended self-presence was developed. In this installation, the idea of multiple overlaps is central.

[The variation of memory] is an interactive light and sound installation for exploring an experience about the concept of extended self-presence. This installation focuses on delivering an experience of creating a personal identity through lights and sound based on past movements and memories.

It consists of multi-layered lights, speaker and the infrared sensor which can detect the movements of human body parts. Each-layer represents the specific depth of the timeline. Each-layer of the timeline is designed to remember lighting movements by body-parts and sounds generated by body movements during the period of time in each depth of timeline. Thus, the movements of the body at present accumulate as dynamic lighting images and sounds through depths of timeline. These accumulated dynamic lighting images and sounds are reproduced in the present at the same time. Therefore, a multi-layered timeline that has their own memories can create an over-layered dynamic lighting image and sound through time. In this way, the person can create the self’s own visual and acoustic output at superimposing the past memories. Fig 40, Fig 41, Fig 42 show diagrams relating to this installation.

Fig 40. Technical diagram for [The variation of memory]

Fig 41. System diagram for [The variation of memory]

Fig 42. Interaction diagram for [The variation of memory]

In this installation, the pitch was used as the method for sound generation. The way to modulate the pitch according to the degree of movement was used in order to induce people to feel the speed of movement. Fig 43 shows the logic for generating sound.

Fig 43. The logic for generating sound by body movements in [The variation of memory]

In order to test multi over-layered sound output, various kinds of sound waves were simulated. Fig 44 shows the simulation of multi over-layered sound generated based on the different sound waveforms.

Fig 44. Simulation for multi over-layered sound on the timeline in [The variation of memory]

Fig 45. Creating visual and acoustic output by the way of accumulating the past in [The variation of memory]

Fig 45 shows a person creating his own visual and acoustic output through overlaps of the past. It could be explained that this installation could help an individual create his own unique image and sound over time as his identity.

Conclusion and potential

In this paper, based on Merleau-Ponty’s thoughts about the body and the idea of ‘depth’ in time and Bergson’s concept of the term ‘duration’ relating to time, the direction of the experiment on body and time was explored. Especially, the concept of self-presence and memory was investigated as the theoretical foundation of the experiment’s progress. In order to refine the basis of the experiment, through case studies, installations relating to the object or space constructed based on the concept relating to self-expression, memory and multiple overlaps were explored. As experiments based on theoretical research, an interactive system using light and sound was developed in order to deliver the experiences about self-presence and time. These experiences from experiments are not an absolute experience, but subjective and imperfect state that cannot be completed by itself. The characteristics that constantly change in time and space can be summarized as ambiguity. In these experiments, through the transformation of the self over time, the process of looking for the subjective meaning of the self and memory that constantly change is expressed based on the body movements. The intention of experiences pursued through the experiments is to support an individual to reveal the self through the combination with the inner view and rethink about the self.

As the next phase, based on the concept of self-presence and memory, it is planned to proceed with the experiment through a physical object or natural medium merging with body movement and sound representing distorted memory.

Considering potential, for industry, since the user interface of the system which can control multiple lights, motors, and sound is developed for the experiments, if it is modified for users considering usability, it is expected to be able to be a role for interactive media platform for creative works dealing with body movements or the concept of the self and time for artists or companies. Fig 46 shows the user interface of an interactive system developed for this experiment.

Fig 46. User Interface of the interactive system for experiments

As another potential, I expect that experience from these experiments will be connected with each individual’s life and lead them to the inner space of self-discovery.

Bibliography

Armel, K., & Ramachandran, V. (2003). Projecting sensations to external objects: Evidence from skin conductance response.

Bachelard, G. (1964). The Poetics of Space. Beacon Press.

Bergson, H. (1896). Matter and Memory. translated by Nancy Margaret Paul and W. Scott Palmer. London: George Allen and Unwin (1911).

Bergson, H. (1889). Time and Free Will. translated by F. L. POGSON. London: George Allen and Unwin (1910).

Biocca, F. (1997). The cyborg’s dilemma: Embodiment in virtual environments. Journal of Computer-Mediated Communication, 3(2), 12—26.

Bishop, C. (2005). Reconstruction Era: The Anachronic Time of Installation Art.

Damasio, A. R. (1999). The feeling of what happens: Body and emotion in the making of consciousness (1st ed.). New York, NY: Harcourt Brace.

Deleuze, G. (2005). Cinema 2: the time image. London: Continuum.

Huh, S., & Williams, D. (2010). Dude looks like a lady: Online game gender swapping. In Bainbridge, W. (Ed.), Online worlds: Convergence of the real and the virtual. New York, NY: Springer. doi:10.1007/978-1-84882-825-4_13.

Kirkpatrick, D. (1988). Time and Space in the Work of Laszlo Moholy-Nagy Hungarian Studies Review – Vol. 15. No 1.

Kwinter, S. (1996). Virtual City, or the Wiring and Waning of the World. Assemblage No. 29, pp. 86-101 (16 pages). The MIT Press.

Lee, K. M. (2004). Presence, explicated. Communication Theory, 14(1), 27—50. doi:10.1111/j.1468-2885.2004.tb00302.x

Lombard, M., & Ditton, T. (1997). At the heart of it all: The concept of presence. Journal of Computer-Mediated Communication, 3(2), 20. https://doi.org/10.1111/j.1083-6101.1997.tb00072.x

Merleau-Ponty, M. (1945). Phenomenology of perception. translated by Routledge & Kegan Paul. Taylor and Francis e-Library (2005).

Pallasmaa, J. (no date) An architecture of seven senses. Available at: https://marywoodthesisresearch.files.wordpress.com/2014/03/an-architecture-of-the-seven-senses_pallasmaa.pdf (Accessed 09/08/2019)

Ratan, R. A., Santa-Cruz, M., & Vorderer, P. (2007). Multitasking, presence & self-presence on the Wii. Proceedings of the 10th Annual International Workshop on Presence.

Ratan, Rabindra. (2012). Self-presence, explicated: Body, emotion, and identity extension into the virtual self. Handbook of Research on Technoself. New York, NY: IGI Global. 10.4018/978-1-4666-2211-1.ch018

Riva, G., Waterworth, J., & Waterworth, E. (2004). The layers of presence: A bio-cultural approach to understanding presence in natural and mediated environments. Cyberpsychology & Behavior, 7(4), 402—416. doi:10.1089/cpb.2004.7.402

Steuer, J. (1992). Defining virtual reality: Factors determining telepresence. The Journal of Communication, 42(4), 73—93. doi:10.1111/j.1460-2466.1992.tb00812.x

Tsakiris, M., & Haggard, P. (2005). The rubber hand illusion revisited: Visuotactile integration and self-attribution. Journal of Experimental Psychology. Human Perception and Performance, 31(1), 80—91. doi:10.1037/0096-1523.31.1.80

Waterworth, J., & Waterworth, E. (2003). The core of presence: Presence as perceptual illusion. Presence Connect, 3(3).

https://olafureliasson.net/archive/artwork/WEK108815/your-museum-primer [Accessed 06/08/2019]

https://www.seditionart.com/magazine/singing-the-body-electric-at-bian [Accessed 07/08/2019]

https://www.dezeen.com/2014/02/21/mechanical-mirrors-by-daniel-rozin-replicate-images-using-everyday-objects/ [Accessed 07/08/2019]

https://www.dezeen.com/2015/05/29/daniel-rozin-pompom-mirror-interactive-silhouettes-fur-tufts-descent-with-modificationbitforms-gallery-new-york/ [Accessed 07/08/2019]

https://www.dezeen.com/2019/04/10/humanscale-todd-bracher-milan-installation-video/?li_source=LI&li_medium=bottom_block_1 [Accessed 07/08/2019]

https://inmotion.humanscale.com/ [Accessed 07/08/2019]

http://tetsuronagata.co.uk/memory.html [Accessed 07/08/2019]

https://en.wikipedia.org/wiki/Rafael_Lozano-Hemmer [Accessed 07/08/2019]

http://www.julioleparc.org/ [Accessed 07/08/2019]

http://toddbracher.com/work / [Accessed 07/08/2019]

https://en.wikipedia.org/wiki/Dan_Graham [Accessed 07/08/2019]

https://www.random-international.com/ [Accessed 07/08/2019]

https://www.artsy.net/article/artsy-editorial-rain-room-creators-return-with-interactive-robotic-sculptures [Accessed 07/08/2019]

https://archpaper.com/2016/01/hejduks-house-suicide-structures-get-new-life-prague/ [Accessed 07/08/2019]

https://www.tate.org.uk/art/artists/el-lissitzky-1519 [Accessed 07/08/2019]

List of Figures

Fig 1. The Seven Heavenly Palaces (Anselm Kiefer, 2004-2015) (Lim.J.H, 2017)
Fig 2. The House of the Suicide and The House of the Mother of the Suicide (John Hejduk, 1991) Available at: https://archpaper.com/2016/01/hejduks-house-suicide-structures-get-new-life-prague/ [Accessed 07/08/2019]
Fig 3. Louvre Abu Dhabi (Jean Nouvel, 2017) (Lim.J.H, 2019)
Fig 4. Wooden Mirror(Daiel Rozin, 1999) Available at: https://www.seditionart.com/magazine/singing-the-body-electric-at-bian/ [Accessed 01/09/2019]
Fig 5. Trash Mirror(Daiel Rozin, 2001) Available at: https://www.dezeen.com/2014/02/21/mechanical-mirrors-by-daniel-rozin-replicate-images-using-everyday-objects/ [Accessed 01/09/2019]
Fig 6. Pom Pom Mirror(Daiel Rozin, 2015) Available at: https://www.artsy.net/artwork/daniel-rozin-pompom-mirror [Accessed 01/09/2019]
Fig 7. Fragments (Random International, 2016) Available at: https://www.wallpaper.com/art/corporeal-company-random-international-explores-movement-through-robots-and-mirrors [Accessed 01/09/2019]
Fig 8. Bodies In Motion(Todd Bracher, 2019) Available at: http://www.thegreeneyl.com/bodies-motion [Accessed 01/09/2019]
Fig 9. Present Continuous Past(s) (Dan Graham, 1974) Available at: http://www.newmedia-art.org/cgi-bin/show-oeu.asp?ID=150000000020624&lg=GBR [Accessed 01/09/2019]
Fig 10. People on People(Rafael Lozano-Hemmer, 2010) Available at: http://www.lozano-hemmer.com/people_on_people.php [Accessed 01/09/2019]
Fig 11. Computing an Identity (Tetsuro Nagata, 2009) Available at: http://two.wordpress.test/computing-an-identity-tetsuro-nagata.html [Accessed 01/09/2019]
Fig 12. Self and other (Random International, 2016) Available at: https://www.artsy.net/article/artsy-editorial-rain-room-creators-return-with-interactive-robotic-sculptures [Accessed 01/09/2019]
Fig 13. Light-Space Modulator(Laszlo Moholy-Nagy, 1922-1930) Available at: https://www.theatlantic.com/photo/2018/03/bike-share-oversupply-in-china-huge-piles-of-abandoned-and-broken-bicycles/556268/
Fig 14. Untitled(Robert Irwin, 1965-1967) (Lim.J.H, 2019)
Fig 15. Lumière envibration(Julio Le Parc, 1968) Available at: http://julioleparc.org/tablet/lumi%C3%A8res.html [Accessed 01/09/2019]
Fig 16. Your Museum Primer (Olafur Eliasson, 2014) Available at: https://olafureliasson.net/archive/artwork/WEK108815/your-museum-primer [Accessed 01/09/2019]
Fig 17. Chair(Visual experiment for proto self-presence) (Lim.J.H, 2019)
Fig 18. Chair(Visual experiment for core self-presence) (Lim.J.H, 2019)
Fig 19. Chair(Visual experiment for extended self-presence) (Lim.J.H, 2019)
Fig 20. Elements for the level of expression (Lim.J.H, 2019)
Fig 21. Simulation for tracking position in 3d space (Lim.J.H, 2019)
Fig 22. Simulation for tracking degrees in 3d space (Lim.J.H, 2019)
Fig 23. Technical diagram for [There] (Lim.J.H, 2019) (Lim.J.H, 2019)
Fig 24. System diagram for [There] (Lim.J.H, 2019)
Fig 25. Simulation for high-level representation in [There] (Lim.J.H, 2019)
Fig 26. The first simulation for low-level representation in [There] (Lim.J.H, 2019)
Fig 27. The second simulation for low-level representation in [There] (Lim.J.H, 2019)
Fig 28. System diagram for an experiment about feeling the past (Lim.J.H, 2019)
Fig 29. Simulation about feeling the past (Lim.J.H, 2019)
Fig 30. System diagram for an experiment about feeling the present (Lim.J.H, 2019)
Fig 31. Simulation about feeling the present (Lim.J.H, 2019)
Fig 32. Technical diagram for [The here] (Lim.J.H, 2019)
Fig 33. System diagram for [The here] (Lim.J.H, 2019)
Fig 34. Interaction diagram for [The here] (Lim.J.H, 2019)
Fig 35. The logic for generating sound in [The here] (Lim.J.H, 2019)
Fig 36. The logic for interaction in [The here] (Lim.J.H, 2019)
Fig 37. Simulation for Interaction with the sound of the past in [The here] (Lim.J.H, 2019)
Fig 38. Interaction with the self of the past in [The here] (Lim.J.H, 2019)
Fig 39. Interaction with the other of the past in [The here] (Lim.J.H, 2019)
Fig 40. Technical diagram for [The variation of memory] (Lim.J.H, 2019)
Fig 41. System diagram for [The variation of memory] (Lim.J.H, 2019)
Fig 42. Interaction diagram for [The variation of memory] (Lim.J.H, 2019)
Fig 43. The logic for generating sound by body movements in [The variation of memory] (Lim.J.H, 2019)
Fig 44. Simulation for multi over-layered sound on the timeline in [The variation of memory] (Lim.J.H, 2019)
Fig 45. Creating visual and acoustic output by the way of accumulating the past in [The variation of memory] (Lim.J.H, 2019)
Fig 46. User Interface of the interactive system for experiments (Lim.J.H, 2019)

List of Tables

Table 1. Layers of self-presence(Damasio, A. R., 1999) (Lim.J.H, 2019)
Table 2. List of cases (Lim.J.H, 2019)