The feeling of presence (FoP) is a fleeting sensation often associated with neurological disorders or spiritual encounters, and yet its experience among healthy individuals are scarcely researched. Upon constructing a media-rich installation with diverse sensory stimuli, I hope to elicit the experience of FoP in normal subjects. Two key areas to explore include altering one’s bodily self-consciousness and supplying ambiguous sensory cues in the environment. A temporary mismatch between the sense of agency and the sense of ownership is suspected to be a potential method to induce FoP from participating visitors. After a critical evaluation of the preliminary prototypes, I shall propose an installation art piece that relies on the latest computer vision and video projection technologies to enable the sensorimotor conflicts that encourage the FoP experience based on cast-body shadows.

Background and Objectives

The feeling of presence (FoP), sometimes referred to as “sensed presence”, and contentiously defined as “extracampine hallucinations” (Chan & Rossor, 2002) or “leibhaftige Bewusstheit” (Jasper, 1913; Sato & Berrios, 2003), is a sensation far more common than the general public take notice of. This phenomenon is most frequently associated with hallucinations accompanying sleep paralysis attacks (Cheyne & Girard, 2007), illusory sensations arise in Parkinson’s disease and dementia with Lewy bodies (Fénelon et al., 2011; Wood et al., 2015; Chan & Rossor, 2002), the sensing of a deceased one following bereavement (Steffen & Coyle, 2011), and hallucinatory experiences in extreme survival scenarios (Brugger et al., 1999; Geiger, 2010). The rapid development in the field of cognitive neuroscience has brought renewed attention to FoP in the past decades, and yet, the lack of a consistent body of literature indicates the need of a systematic investigation on the topic.

Despite the variety of experiences associated with FoP, a few salient features emerge across the board. First, similar to other forms of hallucination, FoP occurs convincingly without identifiable external stimuli (Nielsen, 2007; Solomonova, Frantova & Nielsen, 2011; Alderson-Day, 2016; Barnby & Bell, 2017). Any visual, auditory or tactile cues perceived from the environment are only interpreted “in subordination to the presence” (Solomonova, Frantova & Nielsen, 2011), as they contribute to the construction of a coherent narrative. Second, the presence is of “a purely spatial nature” (Nielsen, 2007), which means its emergence does not rely on mental representations we abstract from sensory channels such as vision, hearing or touch. Instead, it contains “the most basic of all invariant perceptual information available in the environment” (Gibson, 1966) such as that of “spatial location, orientation, mass, volume, extent, movement, trajectory”(Nielsen, 2007). Third, FoP is always experienced through the perception of a human, contrary to a non-human figure (Chan & Rossor, 2002). Some further propose that the presence carries a sense of intentionality and that they “seem to have goals or intentions with respect to the self” (Nielsen, 2007).

In the past, a few theories have been brought forth to explain how the anomalous phenomenon of FoP came into being, which include the “illusory reduplications” of one’s body projected into the extrapersonal space (Brugger, Regard & Landis, 1996; Blanke, Arzy & Landis, 2008), the generation of “hallucinatory variants of social imagery” (Nielsen, 2007; Solomonova et al., 2008; Fénelon et al., 2011), or a threat activated “hyper-vigilant and biased attentive state” (Cheyne, 2001; Cheyne & Girard, 2007). Unfortunately, there has yet to be a tried and tested method that replicates the sensation of FoP in the lab to unveil the fundamental sensory elements that trigger such an experience. The only experimental elicitation of FoP by transcranial magnetic stimulation (Cook & Persinger, 1997; Persinger & Healey, 2002) was quickly debunked through a double-blind trial in later years (Granqvist et al., 2005). On the other hand, a single case targeting electrical stimulation towards the temporoparietal junction of the brain was reported successful to induce the sensation of FoP (Arzy et al., 2006). The scholars called attention to the functions of the temporoparietal junction involved in “self-processing, self—other distinction, the integration of multisensory body-related information, and other illusory own-body perceptions”. This has motivated the suspicion that a disturbance in “multisensory (proprioceptive and tactile) and sensorimotor integration of information” with respect to the body can lead to a bodily illusion that promotes FoP.

In the years to follow, a series of research led by Blanke proceeded in the direction that sees FoP as “an illusory own-body perception with well-defined characteristics that is associated with sensorimotor loss” (Blanke, 2014). He and his colleagues designed a master-servant robotic system to produce specific sensorimotor conflicts in the human body, aiming to verify FoP “caused by misperceiving the source and identity of sensorimotor (tactile, proprioceptive, and motor) signals of one’s own body”. The results were most interesting when asynchronous feedback stimulation was introduced, which caused a shift from illusory self-touch to the feeling of being touched by other. The test was found successful for inducing FoP on normal participants and gave profound insight into atypical sensorimotor signal integration leading to the phenomenon.

This particular finding has helped focus my research and revealed the significance of two critical aspects that contribute to the FoP experiences: the fluctuation of self-attribution and the ambiguity of sensory stimulants. In this light, it might be valuable to first gain an understanding of the human bodily self-consciousness and then work towards altering the phenomenal selfhood through a digital environment. When it comes to the self, there has been such an abundance of research conceptualising its definition all across the spectrum that any attempt to forge a comprehensive review would appear but inadequate. In later sections of this paper, I shall organise the most relevant findings taken from cognitive science and establish a theoretical foundation for my design decisions.

The primary objective of my research is not to agonise over an infallible explanation for the occurrence of FoP, but to test out existing psychiatric theories through experiments and hopefully extend their application to the realm of installation art. The final design project would aim to push the boundary of bodily self-consciousness via implementing a series of computer-mediated sensory cues in the environment. More than sixty years after Ryle has abolished the “ghost in the machine” argument (Ryle, 1949), I hope to further the conversation and to discover if the computer manipulation of external conditions – corresponding to the human body – would result in the human mind expanding its self-identity. During the design research, I will speculate the minimum multi-sensory stimuli it requires to induce the FoP experience. A range of mechanisms and materials will be examined in the hope that the eventual outcome is a rewarding one.

Methods

As a remarkably intriguing theory goes, “no such things as selves exist in the world” (Metzinger, 2003). In a seemingly absurd fashion, Metzinger stresses that the construction of the self is an ongoing and dynamic process, where “the subjective experience of being someone emerges if a conscious information-processing system operates under a transparent self-model.” He introduces a distinctive model of such cognitive operation called the “phenomenal self-model”, which corresponds to what Gallagher calls the “minimal self” – as opposing to the “narrative self” (Gallagher, 2000). The minimal self-awareness contains two pivotal concepts: the “sense of agency” and the “sense of ownership”.

Sense of agency: The sense that I am the one who is causing or generating an action. For example, the sense that I am the one who is causing something to move, or that I am the one who is generating a certain thought in my stream of consciousness.

Sense of ownership: The sense that I am the one who is undergoing an experience. For example, the sense that my body is moving regardless of whether the movement is voluntary or involuntary.

These two aspects are vital to the moment-to-moment motor planning process as well as to the creation of the immediate self-consciousness. A computational model proposed to illustrate the sensorimotor integration cycle in the body (Wolpert, Ghahramani & Jordan, 1995; Wolpert & Ghahramani, 2000) could also be elevated to the phenomenal level of selfhood. As shown in Figure 1, Gallagher demonstrates the two comparators involved in executing an intended action, with the match in the forward model providing a sense of agency and the match in the feedback model yielding a sense of ownership. Upon confirming both matches, a sensorimotor loop would then be rendered complete. We will find this procedure equally beneficial for constructing the phenomenal level of representation that constitutes the fundamental experience of our conscious existence, although not all agree on the matter. Unlike many scholars who thought agency to be a necessity in achieving self-consciousness (Gallagher, 2000; Metzinger, 2003; Pacherie, 2007), Blanke and Metzinger maintain that “bodily agency is a causally enabling, but not a constitutive condition, for phenomenal selfhood” (Blanke & Metzinger, 2009).

Figure 1: The forward and feedback comparators (Gallagher, 2000)

Figure 2: Neuropsychological model of inserted thought (Gallagher, 2000)

On the contrary, as discovered in the master-servant robot experiment, a reduction in the sense of agency is likely to result in sensorimotor conflicts leading to the generation of FoP (Blanke, 2014). It should come as no surprise once we compare this with symptoms found in schizophrenia, such as thought insertion and delusions of control. Figure 2 shows the sense of agency being lost through the lack of a match in the forward model, while the cognitive feedback model completes its regular loop. As a result, the schizophrenic patients would fail to integrate the reafferent feedback signals into their intended state and therefore feel their thought or action being controlled by an external agent (Frith, 1987).

It is important to note that these symptoms differ from another abnormality associated with the deficiency of agency, the anarchic hand sign (Blakemore, Wolpert & Frith, 2002; Pacherie, 2007), which is when patients report complex movements from an upper limb noncompliant to their intention. The intriguing difference here is the fact that the anarchic hand sign patients never express a sensation of alienation from their limb, whereas the experience from delusions of control is always attributed to an alien force. The schizophrenic patients are recorded making statements such as “feeling like an automaton”, “guided by a female spirit who had entered me”, or “spirits moving my shoulder” (Spence et al., 1997). Scholars discovered that the anarchic hand sign only offends the agentive self-awareness in a higher-order narrative self, but not the low-level mechanisms of sensorimotor integration pertaining to the minimal self (Blakemore, Wolpert & Frith, 2002; Pacherie, 2007). Such discrepancy draws attention to an important feature of the alien presence in FoP: it is a hallucination bearing the “most basic type of perceptive information” (Nielsen, 2007). Cheyne poetically describes it as “a feeling of raw otherness present-at-hand, its only quale an ineffable sense of ‘thereness’” (Cheyne, 2001). Regardless of the impact FoP brings towards our narrative self-understanding, it is foremost a disturbance to the most fundamental perception of existence. This notion should inform my depiction of FoP as an experience that challenges our utmost basic instincts about the selfhood.

Meanwhile, it is worth considering the three popular hypotheses of FoP: external body mapping, social imagery, and threat activation. On sufficient theoretical grounds, a focused effort will undoubtedly enhance the effectiveness of my design outcome.

To start with, FoP proclaims a shift of the bodily proprioceptive localisation outwards to one’s extrapersonal space. The occurrence of proprioceptive drift was first observed in the rubber hand illusion, a seminal experiment to achieve visuotactile integration of a fake limb into the body perception (Botvinick & Cohen, 1998). Ever since then, plentiful research deriving from the original experiment investigated representations and actions of body parts, yet the global and unitary character of the self was often neglected. The holistic framework which founds our immediate experiences is an elementary condition that enables our bodily self-consciousness. Blanke believes there to be three major aspects related to the phenomenal selfhood: “self-identification with the body (that is, the experience of owning a body), self-location (that is, the experience of where I am in space) and the first-person perspective (that is, the experience from where I perceive the world)” (Blanke, 2012). In his research, Blanke listed extensively the experiment set-ups that target these three facets and revealed how each condition could be altered to mimic the episodes generally occur in out-of-body experiences, heautoscopy, and autoscopic hallucinations. These procedures use video, virtual reality, robotic devices, etc., to induce in healthy individuals “out-of-body illusions or full-body illusions, that arise from visuotactile and visuo-vestibular conflicts”(Blanke, 2012). Blanke’s work is a substantial resource of inspiration for creating the full-body proprioceptive drift during a FoP experience.

Figure 3: Two models of FoP production during sleep paralysis attacks (Nielsen, 2007)

On the other hand, the FoP production models of social imagery and threat activation (see Figure 3) are under much debate. Both are preconditioned through ambiguous sensory cues in the environment, by which influence the brain to “create the perception of a person from partial sensory stimuli during states of increased arousal (fear, paranoia)” (Chan & Rossor, 2002). Simard and Nielson were the first to compare the phenomenon of FoP to social anxiety. Once triggered, the anxiety prompts spontaneous social imagery that replicates past negative social events (Simard & Nielsen, 2005). An alternative explanation of FoP involves the vigilance system being activated by detection of possible threat. According to Cheyne, it would then initiate procedures that “monitor the environment for further cues to corroborate or disconfirm the existence of an emergency” (Cheyne, 2001). In both cases, the ambiguity of sensory stimuli works in favour of the FoP generation. This feature is especially valuable for creating “immersion by suggestion”, a concept first formulated by Frantova in referring to the Holy Theatre. Inside a production of the Holy Theatre, an actor usually becomes “an embodiment of the invisible: a prophet, an oracle or a priest through whom the Other-world talks” (Frantova, Solomonova & Sutton, 2011). It is an art form of invisible-made-visible, but “not automatically – it can only be seen given certain conditions” (Brook, 1996). In a similar way, an environment conducive to FoP should provide ample sensory signals that are open for interpretation, allowing the gap between reality and immersion be filled by the imagination of the visitors.

Results and Discussion

In an effort to induce FoP through immersive artwork, I have designed a series of preliminary prototypes to analyse different aspects of the theoretical foundation. The experiments to date follow a similar formula: The visitor’s location and movements are monitored by a tracking device such as a digital camera or a Kinect sensor. Converting these data into spatial coordinates, the computer would calculate a specific function corresponding to the visitor trajectory. Visual and auditory feedback is then created by the use of a CNC plotter, affecting a layer of physical material with forces like magnetic energy or wind power. The latest trials, nonetheless, diverge away from the X-Y plotting mechanics, turning to computer vision and digital projection for a more immersive design.

The Machine Among Us (Lea, 2018) was my first attempt to achieve the uncanny – a feeling often accompanies the FoP experience. The prototype consists of a basic CNC plotter, a light-box, some playing sand, and a spherical metal scribe. The detection of a visitor approaching the apparatus would trigger the plotter to run a predetermined script. At this point the carriage moves the metal scribe through a magnetic piece underneath the drawing surface, and soon a written message appears in the sand. With the plotter hidden away from sight, this prototype relies on mere illusion to the eye. It is a fanciful stimulant that gives away the bare minimum which becomes an open invitation to a variety of suitable narrative. And yet, this version of the installation does not touch upon a disturbed sensorimotor loop affecting the self-consciousness.

My second prototype, Felt Presence II (Lea, 2018), features an array of wooden stick pendulums that part ways to illustrate the detection of visitor passage. Each pendulum has a tiny piece of neodymium magnet embedded in the lower end. Once again, this model relies on the X-Y plotting mechanism, and the CNC carriage affects the magnetic field of the pendulums with another magnetic piece underneath the platform. When the pendulum movement corresponds timely with the visitor transit, it creates a bizarre illusion that an invisible agent is dwelling among the pendulums. Occasionally it even inspires a vague sensation of proprioceptive drift, one of the common conditions observed in FoP, during which the self-localisation moves outwards from the body.

Intriguingly, scholars at MIT have tried to investigate exactly this condition with a unique variation of the rubber hand illusion: the “invisible hand illusion”, as they named it, demonstrates that “healthy (nonamputated) individuals can refer somatic sensations to a discrete volume of empty space and experience having an invisible hand” (Guterstam, Gentile & Ehrsson, 2013). It is discovered that the volume of empty space could be assimilated into bodily self-representation as long as visuotactile-proprioceptive integration is achieved through “temporal synchrony and spatial congruence” in the appropriate near-person space. This conclusion challenges the assumption held previously that “only objects that have the same visual form with body parts can be experienced as part of one’s body” (Tsakiris, 2010), as many advocated that the process of intermodal matching is only possible when the visual input agrees to one’s internal body image. This prior belief poses crucial technical difficulties for developing the FoP experience using physical materials: the computer-mediated environment must showcase effects that communicate properties of the human body, and even then, there remains the issue that most people tend to have different body images of themselves. On the contrary, if we were to adopt findings from the “invisible hand illusion”, there is much to be said for maximising the illusion of the empty space. A clever arrangement just might provide sufficient ground to elevate the invisible into the visible.

To explore other imaginative dimensions of the empty space, Felt Presence III (Lea, 2018) introduces the whimsical effect of sheer fabric blown in the wind. The organza voile became an obvious candidate considering its lightness. A sizeable piece of the fabric is hung lifted in the air, and underneath, a large-scale CNC plotter manipulates an upward facing fan, with its position compellingly reflecting a visitor’s bodily location in space. A valuable detail of the fan mechanism is the consistent suspension before the flowing air reaches the fabric surface, which produces a visual feedback delay between the visitor movement and the air bump formation. This delaying procedure is designed to reduce the sense of agency in the body, as was indicated in Blanke’s master-servant robot experiment, so that it triggers the sensorimotor conflicts required for inspiring the FoP (Blanke, 2014). While the above is generally true, it was later discovered that an action-effect interval of up to 250 ms would in fact, not undermine, but strengthen the sense of agency (Haering & Kiesel, 2015). In the case of the fan mechanism, since there was no evaluation in place to accurately measure the delay, we had no way of knowing the state of one’s sense of agency. On the other hand, an expansion in the sense of ownership – the other basic requirement for generating the FoP experience – calls for synchronised tactile feedback in order to manifest a volume of empty space through the visitor’s body reference frame. Yet, there were no tactile stimuli incorporated in this model. The only upside here is the mystical visual language. Many visitors at the installation have reported perceiving an otherworldly presence, as they commented on the cultural allusion to a ghostly figure the fabric was made to look like. Unfortunately, this was not the type of FoP experience we intended to induce from an altered bodily self-consciousness, but rather, a higher-order cognitive narrative.

Hitherto we have understood that the sensation of FoP appears under two distinctive circumstances: when the sense of agency is disturbed and when the sense of ownership drifts towards the extrapersonal space. As illustrated in earlier sections of the paper, sensorimotor loss resulted from a mismatch between these two senses would subtly disorient the brain mechanisms that differentiate the self and an alien presence. Both senses are worth thorough experimentations in the prototype I will propose below.

Cast-body shadow is a visual phenomenon that accompanies us for much of our day-to-day living; however, the pervasiveness of its presence sometimes leads us to overlook its significance. Studies have shown that our shadows bear a unique relationship with our bodily perceptions as well as a higher status in our cognitive processing. As an experiment by Kuylen and colleagues has indicated, the cast-body shadow is capable of provoking a sense of ownership through mere visual stimuli, unlike the rubber hand illusion that requires tactile feedback to complete the integration (Kuylen, Balas & Thomas, 2014). This is because the shadow moves in synchronisation with the physical body and “elicit automatic orienting of attention toward the body-part casting the shadow” (Pavani & Galfano, 2015). These features combined seem to evoke a similar outcome as does the synchronous visuotactile feedback, which is needed for the proprioceptive drift to take place. The cast-body shadow is seen to “promote binding between personal and extrapersonal space” (Pavani & Galfano, 2015). Manipulating the cast-body shadow through a mediated environment is no doubt going to yield to some impressive bodily illusions.

Figure 7: Your uncertain shadow (color) (Eliasson, 2010; photo by Jens Ziehe)

Within the domain of installation art, there has been a number of successful examples that utilise the cast-body shadow effect to achieve impressive full-body experiences. These works focus on the particular bond a body has with its cast shadow and examine the dynamics created through changing specific properties that come with the shadow. Your uncertain shadow (color) is an excellent case study that breaks down our common perception of the dark shadow by introducing a colour split of the natural darkness (Eliasson, 2010). As a visitor moves through the exhibition space, coloured light sources continuously generate new shifts in the colour and scale of the body shadow. An uncertainty incurs as a result of this active change in the relationship one has with the everyday phenomenon. Here, Eliasson’s work does not alter the sense of agency or the sense of ownership that makes up one’s bodily self-consciousness, yet one might be surprised by the multitude of its representation. It is my intuition that if some copies of the shadow were to move slightly different from the body, a sensation of FoP should arise in the process. To achieve that, it would require digitally producing the moving shadow as a video footage, and then playing it back onto a projection surface. Thanks to the evolving research in generative adversarial networks (GAN), as illustrated in Figure 8, we have seen success in the motion transfer technology for imposing synthesised body gestures onto a target subject (Chan et al., 2018). The down fall of the GAN method, however, is the amount of time necessary to train a new model for each individual target (Chan, 31 August 2018). This will hinder the realtimeness of the interaction facing a large number of visitors.

Figure 8: Examples of motion transfer technology (Chan et al., 2018)

Another aspect of the shadow that has often been experimented with is the time frame of its occurrence. One of the earliest projects that touched upon this topic was a student one from the Bartlett many years ago. In his endeavour to research material intelligence, Holland came across photoluminescent paint and observed its ability to remember light. He then proceeded to cover an entire screen with the paint (see Figure 9), as he discovered that, through shining a stroboscope, shadow-like traces of the human body emerged on the screen. Howland marvelled at the physical quality of the dark shadow, calling it “an image of presence” (Howland, 1994). This image defines the relationship the human body has with the environment in that particular moment, yet it lingers on long after the moment has passed. It is within reason to suspect that the prolonged appearance of the cast-body shadow would extend the timescale one maintains the sense of ownership, although it makes no influence on the sense of agency because of the static nature of the shadow.

Figure 9: Analogue realisation of the delayed shadow (Howland, 30 August 2018)

Figure 10: Digital realisation of the delayed shadow (Howland, 30 August 2018)

For his graduation thesis, Howland conducted a digital version of the delayed shadow using the most advanced computer vision and video projection technologies at the time (see Figure 10). In this installation, the silhouette of a visitor was recorded and displayed on a translucent screen in the negative colour scheme (Howland, 1994). Because of the lighting arrangement, the illumination would occasionally leave an actual cast-body shadow overlaying onto the projected silhouette. The delay between the physical shadow and the digital projection was intriguing for the people participating, in addition to setting off fascinating visual effects through the ongoing feedback loop from the camera to the projector. While this particular project does not manage to induce sensorimotor conflicts in the fundamental perceptions of our existence – a requirement to trigger the FoP experience – it has invited us to think intuitively about the relationship our body has with the environment. This has reminded us that, beyond the three-dimensional Euclidean space, time is a valuable fourth dimension which demands our attention in the design process. In the future prototype, I shall emphasise time as an essential element to experiment with.

As mentioned in a prior portion of the paper, people feel a unique sense of ownership towards their cast-body shadows. This is largely because the shadow is a faithful visual representation of the body, forever moving with us in unison. However, to achieve a diminished sense of agency, we will have to somehow contradict this synchroneity. Even though this lengthy condition seems like quite a dilemma, inventing new space-time correlations should offer an interesting approach to the problem at hand.

For my next prototype, I propose adopting the slit scan technique to refine cast-body shadows, in an effort to provide the sensorimotor conflicts needed to trigger the FoP experience. Once we enable the infrared camera on a Kinect sensor for skeleton tracking, getting a projected silhouette to look like a dark shadow will not be the biggest challenge. I shall then create a customised computer algorithm basing on two simple guidelines: first, the horizontal or vertical lines selected for the delay effect would correspond to the position and movement of body joints retrieved from skeleton tracking; second, the interval of the delay would fall randomly between 0 ms and 500 ms to achieve the widest range of variation in the sense of agency. The uncertainty in both of these arrangements should open up possibilities to fascinating visual effects connected closely to visitor movements. This, combined with the natural ownership one feels towards one’s shadow, should have a good chance of inducing the sensation of FoP in an installation. To measure the performance of future trials, we should apply an adapted version of the Sensed Presence Questionnaire (SenPQ) focusing specifically on the installation (Barnby & Bell, 2017). In this way, we can be sure to make improvements in the right direction.

On her prior attempt to create an immersive FoP phenomenon, Frantova commented, “Being in doubt about the experience is a reliable way of making someone self-conscious” (Frantova, Solomonova & Sutton, 2011). This is especially valuable considering that producing FoP heavily relies on the participating subject’s attention to each sensation change in their body. The ambiguity of sensory stimuli in our computer-mediated environment is crucial in this regard. A visitor would be more likely to notice the sensorimotor loss in their bodily self-consciousness, and perhaps be more open to experiencing FoP.

Conclusions

In this paper, we argue that the complex sensation of FoP is, in reality, reproducible once we abide by a series of predetermined conditions. This phenomenon is predominantly linked with the conflicted disposition between the sense of agency and the sense of ownership. A computational model used to explain the sensorimotor integration in the body is equally beneficial to illustrate how phenomenal selfhood is formed. Any disturbance to the integration is prone to confusing the brain from telling apart images of the self and an alien presence.

While there are an array of computer technologies we could employ to mediate the installation environment, the prime focus in the next three months should be utilising the slit scanning technique for video production, along with motion tracking technologies, to design the most intuitive variation of the FoP experience based on cast-body shadows. There should also be careful research on the visual features of the shadow, which improves the physical presence of a digital projection. These aspects all help to promote an impactful and evolving experience of FoP.