Today’s school’s systems still adopt antiquated methods of teaching, which are heavily relying on verbal and text-based content. With the current boom and the constantly decreasing costs of new technology such as AR and VR, we have the ability to create virtual educational content gear towards a more interactive and immersive type of learning, that heavily relies on visual content to convey conceptual teachings. In my paper I will illustrate the importance of visual abstraction in learning environments and the methodology needed for their creation. The paper analyses the different tools and aspects needed to be considered for the creation of visual abstraction, focusing on both the physical aspects that comprise it, such as geometry and

texture, as well as more philosophical ones like culture, emotions and messages. I will reference examples that show how different aspects of visual abstraction have changed the visual style of Nero, a VR educational environment currently in development. Ultimately, I analyse how emerging technologies help and influence visual abstraction through new ways of representing digital media, and analyse the effectiveness and validity of AI as a mass content creator for visual abstraction over human design.

This research paper demonstrates the intricate correlation between physical and psychological factors for the framework needed for the creation of visual abstraction and hypothesises on the use of AI as a practical helping tool for content creators rather than an independent ecosystem.

 

 

Background

World Zero is the first iteration of a Masters project that Ahrian Taylor and I have been developing, which focuses on an embodied learning environment, where the user learns theoretical facts through movements and motion. World Zero is an educational VR experience aimed at children age 13 to 15, equivalent to KS3. The environment is based on the subject of water. Unlike how subjects are usually taught at school, in a linear, enclosed step by step fashion, the game heavily relies on the principle of encapsulating different subjects within a common environment, with the focus of helping children create links across these subjects as they go along exploring different areas of the virtual world and completing an increasing number of tasks.

After the first set of World Zero demo trials we received really positive reviews about the style of the experience, in particular, children found the experience of World Zero visually appealing and overall really enjoyable. Nonetheless, very few people understood the core of the game and were able to proceed further than the first exploratory stages. People seemed to be stuck in the initial level, exploring a land depicting a Greek Island, drawn in a low poly Japanese manga style. After 10 to 15 minutes there was nothing else to explore, and with no clue on how to continue, people got bored and stopped playing. In one sentence: The style was visually appealing but unable to convey any message to the user.

 

Nero

Following this, we began developing Nero, a stripped back more precise version of what World Zero didn’t achieve. Nero takes out the unnecessary detail and visual clutter that was a distraction in World Zero, minimising the environment to focus on the tasks of learning. It refines the learning objectives down to just one task, learning evaporation. While retaining the essential core of World Zero by creating links within subjects using embodied movement, the rest has been restructured from the ground up. Reviewing the need to refine the visuals, I am approaching the design of the experience from a much more visual point of view, rather than a technical one.

 

Firstly, decluttering and denoising the scene from every object that does not play a role or is necessary for the overall VR experience; while creating a visually appealing style which children could find pleasing and entertaining even in the less crucial stages of the experience. And secondly, distilling a visual language that could guide in the creation of the experience, letting the visuals help build the mechanics for the experience and not the other way around.

 

 

Focus on visuals

While developing World Zero we had the chance to interview a few teachers and tutors that educate and support children this age in their studies. We asked them where children struggled the most and if they find some specific subjects harder than others. Although all of the people interviewed taught different topics, there was the clear underlying pattern that most children struggle to grasp abstract concept and objects. This is due to the development of ‘abstract thinking’, the ability to mentally visualise concepts and objects that are not physically present (Dumontheil, I. 2014). Things that exist outside the human scale, like atoms and black holes, objects that break the fundamental laws of physics, but also things that couldn’t be seen merely because hidden from the human eye, such as underground rivers and lakes for example. The learning environment we are currently developing is aimed at children in adolescence, all developing their abstract thinking skills at different stages, therefore, a good visual abstraction of educational subject matter is vital. A clearer more defined level of abstraction with contextually relevant geometry, texture, and meaning, will make the abstract concepts being learned in Nero seem more understandable, aiding in the learning of the material.

 

 

 

 

The idea of adopting a stronger visual style that carried messages – other than just being there to complement the experience – also came to mind after viewing the conference about “Art of Monument Valley” (GDC, 2016. 1). There, the designers created a visual style that was not only really engaging and well structured from a designer point of view, but it was also the carrier of the message of what needed to be accomplished to advance in the game. It was both the background and foreground of what was interactive and what was decorative, it was the enabler of the mechanics and most of all it was visually and aesthetically pleasing to the eye. Leaning on aesthetics rather than mechanics has the indisputable advantage of making the game more accessible, especially to children that are not familiar with the gaming industry (GDC, 2016).

 

 

 

 

 

 

 

 

 

Educational environments

Spanning from paintings, illustrations to digital media the world has plenty of astounding and innovative examples of great visuals. Painters such as Paul Klee and Vincent Van Gogh, cartoonist and illustrators such as Christoph Niemann and Barry Blitt, and again companies such as Studio Ghibli and Pixar are just a few examples. If we look at school books and educational content in general, the only place where we could find images of artists of this calliper is within history books. More often than not, educational content is lacking both in the quantity and quality of visuals, which results in being boring and at times even confusing. An example of this could be a digital educational game called Crystal Island: uncharted discovery, “a game-based learning environment for upper elementary science education with a focus on landforms, navigation, and modelling” (Projects. intellimedia.ncsu.edu, 2018).

Visuals explicitly created for educational purposes seems to appertain to an entirely different category if compared to content created for commercial, entertainment and artistic reasons. With more content being digitised every day and schools promptly embracing the use of new technology there is a definite need for a new type of visuals, as well as the benefit for abstraction in education to aid ‘abstract thinking’ skills. These visuals should not only be ornamental but carry the core elements of what a student should learn, while being at the same time precise, engaging and stimulating, which can be achieved via visual abstraction.

 

 

Abstraction and VR technology

The path followed until now leads to a definite conclusion. There is a need to create a clear visual style that not only carries an aesthetic vision, but is an enabler for deeper meaning and action. We need a way to distil and abstract information. A newer, more focused type of visual, would not only allow for messages and meanings to be summoned and conveyed through a unique and coherent visual style but can aid in child’s learning of educational material.

Clean visuals would also get rid of unnecessary content and help reduce the overall size and rendering power needed to run the application. Smaller file size would ultimately open new routes with the introduction of VR and AR apps on more accessible devices such as smartphones rather than specific and more expensive VR headsets. With the availability of cheap new technologies to the mass user, there has been an increase in demand for new content, which is why research centers and Silicon Valley companies started looking into AI, not only as a helper to the to designers, artist and technologist but also a proper media content creator (TechCrunch, 2018).

AI can be an incredibly useful tool for mass production of content, however, I hypothesise that no single AI can undertake the process of creating visual abstraction as a whole, including the capabilities of conveying meaning and messages, which are largely social and human based aspects. Although an AI can quickly analyze and find correct geometries and textures for a particular use, it comes from a more technical than social point of view and is unable to consider the broader context of the use, including combinations of emotions, messages, meanings, and culture. These aspects can be more thoroughly considered by human design, and the following chapters will show a breakdown of how a human design of visual abstraction could create an effective framework.

It should be noted that animated movement is a major carrier of information in this context. It is, however , excluded from this thesis which concentrates on symbolism, geometry, texture and associated technologies.

 

Definition

 

Definition of Abstraction

The Merriam Webster dictionary defines abstractions as “something pulled or drawn away”. In the broader term an abstraction, or abstracting, is the mental process that allows us to visualise a complicated situation from a different perspective and by zooming in or out, choosing to focus on some aspects while entirely discarding others (Merriam-Webster, 2018). Abstraction, despite what we might think, is a process that sits at the core of our mental abilities. It is the means with which we can isolate a piece of information from others and its relations, to then consider it as a specific object of the investigation (Treccani, 2018)

Similarly, visual abstraction allows us to select and discard information from the world that surrounds us. This mental process not only allows us to describe an object with a minimal amount of information but also to quickly comprehend and assimilate its relation within its environment and its dynamics. (Cai, 2011. P7)

 

Language is said to be the ultimate definition of visual abstraction. An image can be described through thousand of words, but the contrary is also true. A single word, even by not carrying any visual nor physical characteristic related to the object that it describes, is still able to trigger just as many images in our brain (Lowe, 1987). There seems to be a direct and proportional correlation between images and words. Variation in the names, adjective, verbs and the overall language used to describe any single image directly alters the resulting image. The more we abstract, the more we pull out, the more likely we are to obtain a similar image across different subjects. (Cai, 2011. P49) A clear example of this would be to describe a picture portraying a white cube sitting on a pedestal in a museum. Everybody with enough knowledge to comprehend the word cube, pedestal and museum will be able to create a mental image of this. Nonetheless, we will not be able to obtain one exact image, but instead many similar variations of it. If we repeat the experiment, abstracting just the white cube, omitting any unnecessary detail, we will be more likely to obtain a single mental image across different people and consequently better convey our message. Deb k. Roy’s work on “Learning words from sights and sounds: a computational model” beautifully describes how “words are the abstraction of images and images are extensions of words.” ref. (Roy, 2002)

De-noising an image from unnecessary information and abstracting a single detail from it might help us create a sharper mental image. The more we know about the purpose and means of an object, as well as its surroundings, the least amount of information we need to know in order to recognise the actual object itself. This practice is also known as the ‘hollow effect’ (Cai, 2011. P48-50). An excellent example of how the hollow effect works and how the meaning and purpose of an object is essential in visual abstraction in order to correctly recognise it, is given to us by the movie ‘The gods must be crazy’ (IMDb, 2018). The film shows the story of an African tribe being overturned when a mysterious object, an empty Coke bottle, falls from the sky after a pilot flying over it throws it out the window. The tribe does not know the purpose nor the context of the object which is why they interpret it as an ambiguous gift from the Gods.

 

 

Importance of visual abstraction

Living in such a chaotic world overwhelmed with information it is easy to comprehend the importance of visual abstraction. It is an embedded instinct that most living creature adapt to survive and navigate in the environments. It is, in fact, thanks to visual abstraction that we very quickly and efficiently augment and focus solely on critical features and comprehend the message conveyed – “simplicity is nature ultimate sophistication – Yang Cai” (Cai, 2011. P6).

This abstracting mechanism that we have embedded in our brain is not easily replicable. Self-driving robots, AI, and camera systems that analyse the visual environment try to replicate with the use of the most advanced technologies what our brain constantly does in fractions of seconds (Cai, 2011. P7). A picture of a real-life environment can contain up to 90% of redundant information (PETERSIK, 1996), unnecessary, and at times confusing, that our brain has to filter in order to decode the embedded message.

In order to construct a functional framework that would allow us to create visual abstraction, is not enough to remove the image of its background and noise, but we need, first of all, to know what message we want to convey, and what are the essential elements to keep in focus. If done right the message should support our cause and extend the core principles conveyed (GDC, 2016. 2).

The more the visuals are abstracted, the more the image is filled with content that is relevant to the interpretation, and therefore the stronger their relationship. Studio Etter was able to create several short animated clips and games that made great use of visual abstraction. Dreii, a mobile game, made distinct use of this concept. Inspired by Swiss typographers, Etter got rid of everything that was unnecessary to the story, while simultaneously creating links and symbolism with the few objects left in the scene, which depicts a simple set of geometry. By accomplishing an increasingly complex set of task featured in the game, the user can advance to the next level. Nonetheless, the game does not contain any textbased instruction nor speech. Everything necessary in order to understand and complete the task is merged within the visual abstractions of the game itself (Etter, 2018).

 

Following Etter, It is possible to say that in order to build a framework that would allow us to achieve an excellent visual abstraction we need to do specific steps: simplify the visual content, abstracting essential information, but without altering their purpose and meaning of the objects. Denoising the image, focusing on the primary elements and their relations will bring several benefits. It will decrease the overall file size, and amount of information to be analysed resulting in an increase of speed at which images are processed both by a human brain and a piece of hardware like a PC or smartphone. However, more than this, visual abstractions will coney a much clearer message with little space for interpretation across different cultures, ages and languages (Winnemöller, Olsen and Gooch, 2006).

 

 

Consideration

In the following chapters, I will analyse how we can use strategies to create visual abstractions through meaning, geometry and texture. It should be noted that this paper deals with the initial three dimensional graphic abstraction that is seen as the first still image to convey information, rather than the use of animation and movement which become a latter part of the technique of meaning, hence why I have not included movement in this discussion.

It needs to be said that done by human design, there is not one specific technique to achieving visual abstractions, as the process and the result are both tied to the final aim and the style, as well as the cultural background, experience and skills of the person creating it. Nonetheless, when creating visual abstraction, it is essential to know that some mediums when compared with others, allow a broader aspect of modifications. Photographs are a mere representation of reality where both colours, geometry and meanings are fixed within the frame. In collages, we can introduce malleable figures and textures that mix with the one already present in the picture. This new introduction allows to slightly control and steer the meaning of the image producing and conveying a new message. Drawings and illustration, both 2D and 3D give the most amount of freedom as the author is in control of the geometries, the colours and the message that the composition creates and conveys.

 

 

 

Meaning as symbolism of visual abstraction

When creating a visual abstraction, regardless of the medium, the author does not start by drawing shapes or making geometries, nor by choosing the colours and textures. The first and most important thing is the meaning. While being the least concrete step defining the visual outcome of a visual abstraction, it is at the same time the most central and influential. Like a script of a theater piece, it will mold everything created on top of it, with the sole purpose of conveying a message in the most clear and direct way.

What message do we want to convey through our abstraction? What do we leave out and what do we accentuate? Answering a series of straightforward questions helps us very quickly and efficiently to define the core messages that we want to convey, and therefore what techniques we might want to adopt in order to achieve our result. In designing Nero, these questions very carefully weighed and considered. The more we could define a group with specific needs and aims, the better we could design the experience in order to leverage on our strength and reinforce the weaknesses. We were able to define an age group of children in KS3, 13 to 15 years old teenagers, immersed in an educational setting. With an educational setting the meaning can become quite literal, as ease of learning becomes a priority.

Several experiments, carried out using an eye tracking system, have demonstrated that a context or purpose guides human visual searching (Yarbus, 1967). Hence, it is evident that having a clear structure obtained by distilling what is essential and leveraging on these points, is vital to obtain powerful visual abstractions. Let us think for a moment about interactive experiences and games where we might be asked to collect or use an object in order to do something and proceed in the experience. If such experiences and games make good use of visual abstraction, we know immediately what object to look for, without any need of explanation; texture and the geometry only emphasise it more. This process allows us to take advantage of our brains embedded mechanism which, unlike computers, allows us to examine and process an image quickly and extract semantically meaningful information (Zitnick and Parikh, 2013). In the case of Nero, I have decided to refine a minimal visual language that would help the students discover and explore one notion at the time instead of overflowing the environment with information. Small environmental cues like paths embedded into the floors and mellow lights in the distance guide the students to each piece of information.

 

Experiments carried out on several drawings and paintings done by children and abstract painters demonstrated that our brain could quickly tell apart intentionality from an involuntary act of drawing. More than that, once people were able to detect the intentionality, they immediately observed an insight of presence and communication. People can see beyond more than what is physically represented, by detecting and interpreting elements embedded in the scene (Zitnick and Parikh, 2013. P3009). While creating a visual abstraction, it is crucial to remember what the end user is expecting, making the message blend with the visuals and the other way around (GDC, 2016. 3). An excellent visual abstraction captures the realism that the user wished for; it is easy to understand and interpret, and further produces a memorable experience (GDC, 2017.).

 

Geometry

The shape or geometry is the single element that most influences the physical appearance of visual abstractions. Geometry does not only carry information regarding the size and dimensionality of an object but uses its geometrical features to emphasise embedded meanings and emotions.

 

Due to a specific human cerebral disposition, independent of race, gender and ethnicity, our brain can recognise and abstract simple geometric shapes consistently, each associated with its specific meaning (Larson, Aronoff and Stearns, 2007. P526). Simple shapes such as triangles and circles can refer to anger and happiness and are very easily recognised. Squares are a slightly harder shape to notice but associated with firmness and rigidity and strength (YouTube, 2016). This basic lexicon does not mean that visual abstraction needs to be represented under a spherical, cubical or triangular shape only in order to convey meaning, on the contrary, it is enough to merely embed such features in more complex figures for the same effect to happen (Larson, Aronoff and Stearns, 2007. P532). It is precisely by melding and planting details into geometries that one can strengthen the meaning one wants to convey. Animated movies make constant use of our innate capability of abstracting simple shapes from more complex ones. By adding and shaping specific features, we are immediately able to classify characters and objects. Villains might have triangular pointy noses or sharp teeth while benevolent characters might have round ears and bellies. Cruella De Vil is a perfect example of an evil character, with pointy cheekbones and a chin that forms a triangle pointing down, as well as thin and sharp fingers, almost like blades. Other triangles included in her hairstyle, dress and shoes only stress this more. Winnie the Pooh, on the other hand, is a classic child character that perfectly matches the benevolent description, with circles and soft curves delineating its body and facial characteristics. It is therefore easy to understand how efficiently geometry can affect visual storytelling, not only by representing characters and objects but by giving them abstract connotations (Larson, Aronoff and Stearns, 2007 P.352).

 

 

In developing the visuals abstraction for Nero it was essential to induce a sense of curiosity and inviting the children in exploring and interacting with relevant objects, not only by looking at them but by touching and analysing them. At one point in the story, the user needs to learn about weight by moving several rocks from its path in order to continue his adventure. These rocks unlike the ones scattered in the environment present some grasping points very similar to handles in order to be grabbed and lifted, on top of that the heavier the rock the rougher the surface.

Studies have demonstrated that visual abstractions that make good use of this basic set of shapes can convey as much meaning and information as pictures do. Non-representational geometric aspects are enough to convey meanings distinctly. (Larson, Aronoff and Stearns, 2007. P526). Interestingly the Bouba Kiki effect also uses somewhat distinguishable geometries such as curves and angles as a medium to associate phonetic features and meaning (D’Onofrio, 2013. P369). The Bouba Kiki effect specifically allows us to mentally link phonetics and visuals containing specific features and vice-versa (D’Onofrio, 2013. P369), consequently associating specific meanings and characteristic to each obtained shape. Bouba is usually associated with a rounded shape, while Kiki tents to be linked with a star or a jagged figure. Because of the phonetic visual association different subjects and feelings relate to each geometry; again Bouba is seen as a more soft, warm and comforting shape while Kiki is interpreted to be more repulsive and to inspire anger. Studies have also revealed that different types of phonetics and geometries, other than just being classified to be either Bouba or Kiki, are also carriers of specific and more physical characteristics. Large, dark and heavy objects are associated with low pitch sounds and vowels pronounced at the back of the mouth while bright light and small objects are usually associated with high pitched sounds and frontness vowels (D’Onofrio, 2013. PP377-378). These very same characteristics can reinforce the meaning, slightly altering our mental images and perception of it; making a knife look even scarier and pointier and a teddy-bear ever more soft and gentle.

 

Ultimately many artists draw inspiration from nature, trying to find symbolism between preexisting natural shapes and their meaning (Etter, 2018). Studio Etter, for example, took inspiration from sharks teeth while creating evil creatures in the first unreleased demo of their app game Drei. Flying creatures had pointed triangular wings with a jagged edge and would hover back and forth frightening the user and eventually cutting the rope of the object one was trying to lift (Etterstudio.com, 2015). witn Matt Nava took a similar approach in its game Abzu, where instead of merely copying marine creatures and fish he decided to slightly abstract their shapes resulting in a cleaner, more iconic and more memorable geometry (GDC, 2017.). The creatures obtained, stripped of every unnecessary detail, make for a more straightforward and more identifiable shape. It is important to notice that overdoing this process of removing geometric features also removes emotions and meanings attached to them; this is why simpler the geometry, the more important the texture and the other way around (GDC, 2016. 2).

Other than merely representing the subject matter of a particular topic, geometry can be used as a tool to shape the style of a visual abstraction, altering and embedding the characteristics of an object to become a carrier of meaning. In the case of Nero, our embodied educational VR experience, molecules of water for examples are shaped with a sort of bulge rather than entirely spheric; this invites the user to grasp and examine them. Since the experience is also geared towards young teenagers, most of the geometries are soft and round – spiky and triangular shapes that might scare the user are reduced to a minimum – in order to create a calming and inviting atmosphere ideal for learning.

It is now easy to understand how the same topic, developed to convey another message, or simply geared towards a different type of audience would have significantly influenced the overall visual style and physicality of the game.

 

Texture

Half of our brain cells are devoted to visual cognition (Cai, 2011. P7). It is precisely due to our brain cells and our eyes that we are not only able to merely see what surrounds us but also quickly analyse the environment by abstracting critical elements, their shape and their texture. The texture gives essential visual cues about the typology of the object, information such as its physicality and composition, its position relative to us and its borders. A shiny or glossy material, for example, could be regarded as hard, whereas a smooth or matte material could be regarded to be soft, and therefore conveying different emotions as well. These types of information can be obtained merely with visual cues and without the need for physical contact.s we can now use this ability to create virtual elements where we need to convey physicality without being able actually touch or hold the object in our hands, like in VR or AR environments for example. We achieve this by using different tools and visual characteristics at our disposal such as textures and patterns, colours and light reflections or brightness (Cai, 2011. P7).

Light reflections and textures, like aforementioned, can communicate a significant amount of information regarding the physicality of the object and its position in space. Colours on the other hand carry an entirely different type of information, which is less tied to the objectivity of the body and more to its metaphysics, conveying meaning and feeling first and foremost. Colours can make a direct impact on feelings (Kaya and Epps, 2004. P396). as they are deeply associated with emotions and emotions with colours (Kaya and Epps, 2004. PP396-397). We would be driven to think that specific colours are bound to specific emotions, but studies on young children and university students have revealed that those bonds only fix themselves in our adolescents and are deeply influenced from childhood experiences but also culture and marketing (Boyatzis and Varghese, 1994. P83). The constant exposure and homogeneity of information in western society is the primary reason for the creations of large-scale similarities on colour-emotion associations (Boyatzis and Varghese, 1994. P83). Animations studios, games designer, illustrators and many others heavily rely on these broadly accepted preconceptions for their content. Dark colours, for example, are often used to represent sad feelings and evil characters, but also unknown and scaring places. Bright and colourful scenes tend to transmit joy and happiness, whereas grey and dull colours tend to transmit security and strength. At the same time, cool colours can make us perceive a space to be bigger and quieter if compared to a warm colour space which in parity of size might appear smaller (Kaya and Epps, 2004. P397). Alien and unfamiliar objects and places tend instead to use mix colour in unexpected ways, breaking our preconceptions, creating a feeling of unknown and insecurity; trees, for example, might be depicted using shades of purple and skies and waters in different variations of greens and yellows (Kythreotis, 2018). Moebius as an example was a master in using this technique of mixing colours for his surreal illustrations.

 

Journey is a visual storytelling game released in 2012 that makes astounding use of colours in order to convey different emotions and feelings throughout the levels (GDC, 2016. 3). Chapters of the story, dangers and aids were all divided and classified under specific shades in order to create a robust repetitive system for the narrative, which was both clear to comprehend and capable of communicating different moods and emotions (GDC, 2015). The design team also heavily relied on colours in order to communicate and enhance material characteristics of the environment given the minimal amount of information embedded into geometries; due to a specific low poly style chosen for the game (GDC, 2015).

In a highly abstract virtual world like Nero, where children can simultaneously interact with atoms, water, and various elements present in the scene, it is obvious how colours can become a guide other than just a filler. There is not a single colour associated with education, but rather a series of emotions and moods triggered by colours that can enhance learning. Different shades of blue and white are used in abundance in Nero, creating a tranquil and positive environment ideal for maintaining attention and learning. Objects that can be grasped and suggest interaction are usually represented in tones of yellow; this helps them to emerge from the background and distinguish themselves from other objects. Yellow is also a colour that encourages creativity and positive feelings.

Once more we can see how the meaning and the message of our educational experience changed the visual output of our game, but also how the technology employed, and the idea of exploration and embodied learning suggested the used of a different colour to communicate a different message. Visually separating interactable from non-interactable objects was essential to create a smooth experience for the user.

 

Future visions and Tech

 

Specific observations for virtual educational learning

With the advancement of technology, we have seen a graphical revolution since how images were first rendered and displayed. In the early 2000’s, mass produced PCs that could render 3D images were available to most (Moltenbrey, 2001). The same year also coincided with the release of the PlayStation 2; it was, in fact, the game industry that pushed this graphic revolution forward to new standards unthinkable just a few years before (Moltenbrey, 2001). As soon as 2001 complex 3D games were shipping globally, movie and animation industries were also starting to produce 3D content such as high quality animated movie like Monster Inc. from Pixar, or pre-rendered VFX cuts used for the creation of The Lord of the Rings saga.

Although the content was 3D looking and characters were moving in a 3D fashion exploring open worlds and environments, everything was still displayed through 2D monitors (Juul, 2007 PP. 512-513), and therefore unable to create a real sense of depth. Initial attempts to represent 3D content were achieved in the mid-2000s using polarised glasses. Due to the inferior resolution and the field of view limited to the size of the monitor this technology was quickly abandoned.

Virtual Reality headsets existed since the 60’s in military and research fields, Jaron Lanier is among the pioneers of this technology, having coined the term Virtual Reality (UCL, 2017). Headsets first made their public appearance in the mid-90s but never really caught on due to the high costs, low graphics quality and lack of visual content. In 2007 VR made its second public reappearance. Better hardware, specifically developed content, lower costs, motion tracking and a standardised format allowed for VR to grow and develop; quickly followed by Augmented Reality (UCL, 2017).

These two technologies have been developing and growing since. AR was thought to be the successor of VR, but as more and more content is being developed it seems likely that both VR and AR will coexist in the near future, specialising in different sectors and fields (Akçayır and Akçayır, 2017) (Chavan, 2016. P1951).

 

 

The reason I am discussing this is because humans have evolved for millions of years in a 3D environment; we have always learned to interact with things and think spatially. VR and AR now allow us to do just that by interacting with digital media in a very natural and spatial way, leveraging on our embedded reasoning abilities rather than forcing us to translate bizarre movements and gestures between 3D tools and 2D mediums (Juul, 2007 PP. 512-513).

The main reason I am focused on virtual instead of augmented is due to AR being a type of technology that creates an overlay of information on top of our physical world, the amount of content displayed is directly dependent on what is present in our surroundings. Depending on the type of environment and the number of people this can result at times disturbing and or confusing for both the technology and the user. Virtual reality, on the other hand, allows the creation of a virtual environment with no boundaries and wholly detached from our realm. This absence of realism allows for the creation of more abstract content that can detach itself from both visual and physical restraints resulting in a more immersive but solitary experience (Chavan, 2016. PP1948-1950). VR is better suited for immersions and abstract scenarios, therefore being far more applicable to a learning environment that requires abstract thinking (Chavan, 2016. P1950).

Only a small part of the new educational content being created takes full advantage of current hardware and or our spatial abilities. VR Videos and 360 images, are a limited medium, in the sense that they do not enable for any visual abstraction that might help children overcome learning difficulties, nor do they embrace our need for spatial experiences by allowing children to grasp, touch and experience things. The creation of VR educational experiences that utilise visual abstraction through the use of geometry and texture in order to convey meaningful content can be a useful tool set, geared towards an educational purpose that uses our spatial abilities and helps us to visualise things that otherwise would be impossible to represent.

Tests in learning environments with the aid of VR have proven to be successful, showing better results and comprehension from students who have used these technologies compared to classic methods of learning, and VR use has proved particularly effective in subjects that do require abstract thinking (Behmke et al., 2018). Notably, the use of VR enables the creation of a complete abstract environment with no relation to the current context, solely focused on the learning subjects. In the case of Nero, exploring evaporation and seeing the atoms that compose a molecule of water – both highly abstract concepts – virtual reality would be the most efficient way of presenting this subject with abstract visuals for children to interact with in an abstract world.

Our visual abilities allow us to analyse an environment and extrapolate an analogue semantic meaning even if the arrangement of the objects composing the scene varies significantly (Zitnick and Parikh, 2013. P3009). VR has opened the route to a new type of graphics and visual abstraction able to aid us in analysing an environment and extract meaning. Unlike ever before we can visualise and interact with 3D virtual objects in our physical space, bringing both new aspirations and challenges. Graphics and visual abstraction will have to be rethought to fit a new format, changing from a 2D to a 3D perspective, where users are actively immersed in the environment. A strong graphic language, with a view stripped of non-essential objects, will be essential in order to enable the user to comprehend the scene regardless of positioning.

 

 

Human VS AI & tech limitations

I previously wrote that AI can be extremely beneficial for the mass production of content however I do not believe it can fully master the task of visual abstraction over that of human design. Every time humans create a piece of art, regardless of the medium, we do so under the influence of others’ styles of arts that have preceded our work. We are always under the influence of art, voluntarily or not, from places we inhabit, museums, schools, public areas and social media. This factor seems to influence our works but is still unknown how this past knowledge fuses in the process of creations of new pieces (Elgammal et al., 2017. P2). In order to overcome habituations artists continuously try to evolve and push their boundaries to create new styles that still take roots from existing ones but try to detach themselves from the current standards (Elgammal et al., 2017. P2).

In recent years AI has been tested with the aim to create art that was indistinguishable from human-made works (Ward, 2017). Other tests wanted to prove if it was possible to develop an AI with an aesthetic sense – capable of creating, rather than generating – new types of visual languages that do not relate to any existing style and yet that don’t distance themselves too much from what we are used to seeing already (Elgammal et al., 2017. PP 4-5).

When creating a new visual, humans heavily rely on their knowledge, instincts and emotions, the result is a piece of art that has been influenced by many factors simultaneously. AI cannot rely on personality, instinct and emotion, therefore it is almost impossible for it to aim towards this same goal, making the resulting art piece a more random choice or selection from predefined means rather than a thought expression. Furthermore, AI compositions are a random modified collage of pre-existing human art, with no reason nor intention behind it (Elgammal et al., 2017. PP 5-6).

Ahmed Elgammal, from Rutgers University, was able to create an AI -trained on a human database – that unlike Deep Dream from Google, could create images that were indistinguishable from human works, without being a simple alteration and collage of preexisting works; although lacking an intrinsic meaning, the images were truly unique both in style and composition (Ward, 2017). As previously stated, textures and shapes are not the only elements that compose visual abstraction. Culture, the consideration of a meaning and the communication of a given message through a particular composition are active components of what visuals abstraction are. These are elements that even Elgammal’s AI still struggles to accomplish, and goes far beyond the realm of purely computer science (Cai, 2011. P48). The ideal system, human or AI, capable of creating visual abstractions able to convey a message and intention, should have the ability to create unique art itself apart from any current style, it should have the ability to execute such ideas, and it should also have the ability to auto asses itself (Elgammal et al., 2017. P 20).

 

 

Artists and computer scientist have tried to overcome such problems with different solutions. The game ‘No Man’s Sky’, for example, makes use of deduplication for the creation of the visual style in their game. This system allows the creation of an almost infinite series of original outcomes by mixing and matching in different ways a series of pre-made assets. All the assets need to be pre-made by an artist and therefore the abstraction that emerges is limited to the final assembly of the pieces. This method allows for the creation of a considerable amount of data with very minimal input, but at the same time patterns and repetitions start to emerge quickly. Since this system is procedural rather than evolutionary, every output is not a consequent alteration of the previous, but a different mix composed by the same starting assets (Kwaste, 2018). This task completed by the AI would be simply impossible if done by human design, so in this case an AI is an effective response to design. For Nero, an educational game, an AI may be able to choose the correct shape to be grasped in the correct way for a specific interaction, done with more detailed precision and speed than a human, however, it will not take into consideration the context of the learning task or meaning of the interaction. The meaning of the interaction would be important to an educational task, therefore in this case, human design prevails.

Ian Cheng, a US artist, created a series of four evolutionary virtual environments. The first project regarded the creation of assets, shapes and geometries. A custom-made AI, specially developed for the task created a series of unique geometries and shapes resulting from a three month long evolutionary metamorphosis. In subsequent experiments, the same characters obtained from the first experiment were divided into groups and implanted in three different environments. Here two different types of AI were taking care of relations and aesthetics respectively. Depending on the mood, the status and the messages that the characters wanted to convey, both to the viewer and other characters present in the scene, an AI was altering both textures, colours and illuminations of each object consequently. The result was a unique style of geometries and colours that were able to convey a meaning based on their actions and mood. Although successful in the outcome, the art piece required 120 different people, including engineers, artist and computer scientists, constantly working over the span of two years for the realisation of the piece (Cheng, 2018). With this man power, it could be possible to re-create a similar framework for an educational piece to be visually designed by an AI, but the AI would have to be so meticulously designed that the efforts may not be worthwhile.

 

 

Conclusion

 

From my given examples we can see how with the right careful combination of geometry, texture and meaning it is possible to create a clear visual style that is pleasant to the eye, conveys a defined visual style and enables the transfer of message and meaning. The way to create such visuals is through carefully analysing the broader scope of context of the material you want to visualise, and address factors such as the technology employed, the topic of the material, the age group that the experience is aimed for, and emotional factors that might affect both experience and user. In Nero, an educational experience, the meaning is more literal for ease of education, with the use of soft off-perfect geometries that invite interaction and contrasting colours that show where attention and focus should be located. The more information gathered, the more precisely we can direct our abstraction to address a specific target group by using and creating geometries and colours that specifically leverage on their social, emotional and cultural background.

In the case of educational learning, I have discussed visual abstraction to be of in-commensurable aid for the comprehension of difficult topics, especially ones that deal with abstract concepts and objects such as in physics and chemistry. Visual abstraction allows us to visualise those concepts and objects in a direct form rather than merely relying on vocal and text-based description. Visual abstraction also allows us to pull out particular elements of a concept based on relevance or importance within a context, and represent those elements singularly in a given scene. This decreases the amount of external noise and information, proportionally decreasing the number of distracting factors that have negative impact on learning. VR further helps the aim of educational learning by placing visual abstraction in a virtual spatial setting, where objects and scene can not only be seen but experienced physically aiding the teaching by using our existing spatial comprehension abilities.

A de-cluttered scene not only benefits the user concerning focus and attention but also dramatically impacts the overall file size. The use of abstracted shapes with a low poly count as well as simple colours that don’t rely on multi-layered textures, reduces both the amount of objects present in the scene at any given moment, the complexity of the geometry used and the size of the texture compared to hyper-realistic virtual environments. A smaller files size allows for higher speed and performances which provides a smoother and faster experience in high spec hardware; but the more significant advantage given by smaller files sizes that run on low power software is the ability to tackle less powerful, less expensive devices like smartphones, therefore the educational material becomes more readily available to the public.

Finally, this paper demonstrates that although different experiments and test have achieved impressive results with the use of AI for the creation of various graphics, the current state of technology, the heterogeneity and social aspect embedded in the creation of visual abstraction, do not currently allow for an AI to independently create effective visual abstraction capable of conveying a message geared towards a specific audience. Nonetheless, AI should not be disregarded entirely as it can become a helpful tool in functioning as a helper, an extension to design, rather than a standalone entity.

Future tests and research will focus on comparing learning results obtained through the use of the particular style of visual abstraction in Nero run against a regular classroom learning environment. This type of test will help us assess the effectiveness of using modified shapes to encourage interaction and the use of paths and mellow lights to guide users, against a diagram in an exercise book for example.

The next technical step will be to focus on the creation of a machine learning tool such as Wekinator, trained on the existing Nero database of colours and geometries. This will create a tool that could suggest colour palettes based on vertex count such as a geometry with 3000 vertices suggesting a bright palette to reflect the smooth and pleasant shape. This will build a tool for the creation of similar content within predetermined boundaries and styles.

 

 

Bibliography

Book & Journal

Akçayır, M. and Akçayır, G. (2017). Advantages and challenges associated with augmented reality for education: A systematic review of the literature. Educational Research Review, 20, pp.1-11.

Boyatzis, C. and Varghese, R. (1994). Children’s Emotional Associations with Colors. The Journal of Genetic Psychology, 155(1), pp.77-85.

Behmke, D., Kerven, D., Lutz, R., Paredes, J., Pennington, R., Brannock, E., Deiters, M., Rose, J. and Stevens, K. (2018). Augmented Reality Chemistry: Transforming 2-D Molecular Representations into Interactive 3-D Structures. Proceedings of the Interdisciplinary STEM Teaching and Learning Conference, 2(1), pp.5-11.

Cai, Y. (2011). Computing with Instinct. Berlin, Heidelberg: Springer Berlin Heidelberg, pp.1-173

Chavan, S. (2016). Augmented Reality vs. Virtual Reality: Differences and Similarities. International Journal of Advanced Research in Computer Engineering & Technology (IJARCET), [online] Volume 5(Issue 6), pp.1947-1952. Available at: https://pdfs.semanticscholar. org/7dda/32ae482e926941c872990840d654f9e761ba. pdf [Accessed 22 Sep. 2018].

D’Onofrio, A. (2013). Phonetic Detail and Dimensionality in Sound-shape Correspondences: Refining the Bouba-Kiki Paradigm. Language and Speech, 57(3), pp.367-393

Dumontheil, I. (2014). Development of abstract thinking during childhood and adolescence: The role of rostrolateral prefrontal cortex. Developmental Cognitive Neuroscience, 10, pp.57-76.

Elgammal, A., Liu, B., Elhoseiny, M. and Mazzone, M. (2017). CAN: Creative Adversarial Networks Generating “Art” by Learning About Styles and Deviating from Style Norms. [online] pp.1-22. Available at: https://arxiv.org/ pdf/1706.07068.pdf [Accessed 22 Sep. 2018].

Juul, J. (2007). Situated Play – A Certain Level of Abstraction. In: DiGRA. Copenhagen, pp.510-515.

Kaya, N. and Epps, H. (2004). Relationship between Color and Emotion: A Study of College Students. College Student Journal, University of Georgia, pp.396 – 405.

Larson, C., Aronoff, J. and Stearns, J. (2007). The shape of threat: Simple geometric forms evoke rapid and sustained capture of attention. Emotion, 7(3), pp.526-534.

Lowe, D. (1987). The viewpoint consistency constraint. International Journal of Computer Vision, 1(1), pp.57-72.

Moltenbrey, K. (2001). The art of games. Developers reach new level in computer graphics and intelligence. COMPUTER GRAPHICS WORLD, pp.27-30.

PETERSIK, J. (1996). The Detection of Stimuli Rotating in Depth Amid Linear Motion and Rotating Distractors. Vision Research, 36(15), pp.2271-2281.

Projects.intellimedia.ncsu.edu. (2018). About | Crystal Island: Uncharted Discovery. [online] Available at: http:// projects.intellimedia.ncsu.edu/uncharteddiscovery/ about/ [Accessed 21 Sep. 2018].

Roy, D. (2002). Learning words from sights and sounds: a computational model. Cognitive Science, [online] 26(1), pp.113-146. Available at: http://citeseerx.ist.psu.edu/viewdoc/ download?doi=10.1.1.69.8985&rep=rep1&type=pdf.

Snapper, L., Oranç, C., Hawley-Dolan, A., Nissel, J. and Winner, E. (2015). Your kid could not have done that: Even untutored observers can discern intentionality and structure in abstract expressionist art. Cognition, 137, pp.154-165.

Winnemöller, H., Olsen, S. and Gooch, B. (2006). Realtime video abstraction. ACM Transactions on Graphics, 25(3), p.1221.

Yarbus, A. (1967). Eye Movement and Vision. 1st ed. New York: Plenum, pp.171-211.

Conference

UCL:, (2017). In: Jaron Lanier, A Journey Through Virtual Reality. [online] Available at: http://events.ucl.ac.uk/ event/event:j1hg-j9pm85j6-82svmv/jaron-lanier-ajourney-through-virtual-reality [Accessed 22 Sep. 2018].

Interview

Cheng, I. (2018). Visual/graphics Emissaries. [email]. Etter, C. (2018). Etter Studio. Kythreotis, G. (2018). Shedworks.

Website

Etterstudio.com. (2015). Etter Studio | Drei. [online] Available at: https://etterstudio.com/en/drei.php [Accessed 21 Sep. 2018].

GDC. (2015). The Art of Journey. [online] Available at: https://www.youtube.com/watch?v=RoHrwAacTwo [Accessed 22 Sep. 2018].

GDC (2016). 1 The Art of Monument Valley. Available at: https://www.youtube.com/watch?v=i0X85PpYVg&t=917s [Accessed 21 Sep. 2018].

GDC (2016). 2 The Art of The Witness. [online] Available at: https://www.youtube.com/watch?v=A_Gni_2ecd4 [Accessed 21 Sep. 2018].

GDC (2016). 3 Designing Journey. [online] Available at: https://www.youtube.com/watch?v=UGCkVHSvjzM [Accessed 21 Sep. 2018].

GDC. (2017). Creating the Art of ABZU. [online] Available at: https://www.youtube.com/watch?v=l9NX06mvp2E [Accessed 21 Sep. 2018].

IMDb. (2018). The Gods Must Be Crazy (1980). [online] Available at: http://www.imdb.com/title/tt0080801 [Accessed 21 Sep. 2018].

Kwaste, G. (2018). No Man’s Sky — Procedural Content | 3dgamedevblog. [online] 3dgamedevblog.com. Available at: http://3dgamedevblog.com/wordpress/?p=836 [Accessed 22 Sep. 2018].

Merriam-Webster, (2018). [online] Merriam-Webster, Incorporated. Available at: https://www.merriamwebster.com/dictionary/abstraction [Accessed 21 Sep. 2018].

TechCrunch. (2018). Microsoft’s new drawing bot is an AI artist. [online] Available at: https://techcrunch. com/2018/01/18/microsofts-new-drawing-bot-is-an-ai-artist/ [Accessed 21 Sep. 2018].

Treccani, (2018). [online] Treccani. Available at: http:// www.treccani.it/enciclopedia/astrazione [Accessed 21 Sep. 2018].

YouTube. (2016). Movie Geometry – Shaping the Way You Think. [online] Available at: https://www.youtube.com/ watch?v=lLQJiEpCLQE [Accessed 21 Sep. 2018].

Ward, T. (2017). AI Can Now Produce Better Art Than Humans. Here’s How.. [online] Futurism. Available at: https://futurism.com/ai-now-produce-better-arthumans-heres-how/ [Accessed 22 Sep. 2018].

Zitnick, L. and Parikh, D. (2013). Bringing Semantics Into Focus Using Visual Abstraction. [online] Available at: https://www.cv-foundation.org/openaccess/ content_cvpr_2013/papers/Zitnick_Bringing_Semantics_ into_2013_CVPR_paper.pdf [Accessed 21 Sep. 2018].

Image

All rights reserved. Images. ©Eugenio Moggio 2018. Moggio, E. (2018). Nero Collection. [Digital] London.