ILLUSION SPACE Yu Wang Chalmers School of Architecture ACEX35 Master's thesis in Architecture 2022 Examiner: Jonas Lundbery Supervisor: Kengo Skorick, Jonas Runberger Exploration and application of projection mapping technology in spatial design Examiner: Jonas Lundbery Supervisor: Kengo Skorick Jonas Runberger ACEX35 Master's thesis in Architecture Spring 2022 | Material Turn Chalmers School of Architecture Department of Architecture and Civil Engineering YU WANG Email: wanyu@student.chalmers.se Phone: +46765580328 EDUCATION EMPLOYMENT Sichuan University, Bachelor of Architecture Sichuan, China Internship in Urban Design China VTR International Engineering Design Co. Ltd. 2014.9 -2019.6 2017.9 -2019.11 2019.9 - 2022.6 2018.7 - 2018.10 2021.8 - 2021.10 2021.10 - 2021.12 Chalmers University of Technology, MPDSD Göteborg, Sweden - Design and Planning for Social Inclusion - Sustainable Architectual Design - Matter Space Structure Internship in Architecture CISDI Municipal & Architectural Design Co. Ltd. Internship in Architecture L&M Design Lab Internship in Sustainable Community Development Lab of Shanghai Urban Renewal and Space Optimization Technology of Tongji University THANKS TO First of all, a big thanks to the Material Turn team and especially my examiner Jonas Lundbery and supervisors Kengo Skorick and Jonas Runberger, for giving me encouragement and suggestions to help me push my project forward. To Naima and other staff at Chalmers for always answering questions patiently and providing updated information. To all my friends and colleagues at Chalmers, especially Jieming Yan and Zonghao Wu for helping me with my exhibition while I’m working remotely, Yige Qin and Peilin Yin for helping me with the printing. To all the other friends for always giving me emotional support. To my best friend, Liuyang Wang for having academic discussions with me and helping me with some of my renderings. To my family for your endless support. And last but not least to my dearest boyfriend Han Wang for always being there, following every step, and accompanying me during the whole graduate years. CONTENT ABSTRACT INTRODUCTION DESIGN EXPERIMENTS DESIGN - IMMERSIVE THEATER REFERENCES DISCUSSION BIBLIOGRAPHY 9 11 27 51 85 97 101 In this era of rapid development of various technologies, architects are seeking more diverse approaches as other professions. In The Second Digital Turn, Mario Carpo pointed out that variability is a deep-rooted ambition of architects and designers, and the design professions are now coming to terms with a new kind of digital tools they have adopted— no longer tools for making but tools for thinking (Carpo, 2017). In this context, the thesis explores transformations that new technologies can bring to architecture design and try to expand the boundaries of the architectural discipline. This thesis discusses how projection mapping technology may affect the architectural design process. The general principles of projection mapping are summarized through case studies, and these principles are used as the basics of the design experiments. With the method of research by design, the design experiments are used as the main tool to explore the interaction between physical objects and virtual projection, as well as to simulate the spatial effects that projections can achieve speculatively. These effects were eventually applied to an Immersive theater design where the audience can follow the actors in a sequence of space to experience the narratives of the classic Kurosawa film Rashomon. In this theater, different scenes can be represented in the same physical space through projection mapping, the limited space can be extended, non-existent objects can be created, and the shape of the space can be changed according to the demand. There are two l ines of design method in this thesis: one is the design of the physical environment that will be projected, and the other is the design of the virtual projection content. The combination of projected images and the physical environment is the main focus of the design, and the architect controls the boundary and relationship between real and virtual. In this thesis, the design experiments and immersive theater design parts have an equally important place in the outcome, because the exploration and application of the possibility of projection mapping technology is the main goal of this thesis. ABSTRACT Key words: The Second Digital Turn, Spatial Augmented Reality, Projection Mapping, Immersive Theater INTRODUCTION - Background - Discourse - Method - Workflow - Main Questions & Objections - Delimitations 01 ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 1312 / Illusion Space // Introduction // INTRODUCTION / Background Figure 1: Borderless World. teamLab, 2018 Figure 2: The virtual city by Space Popular is designed to host exhibitions, awards, and conferences for Spanish architecture festival Arquia Proxima. Space Popular, 2020 // INTRODUCTION / Background I n the 1980s, des igners began to use computers for design and documentation, triggering a digital transformation in the field of design. Today, with the rapid improvement of computing power and highly developed data storage, the so-called Second Digital Turn is coming. This transformation has the potential to provide a more immersive experience based on virtual reality (VR), augmented reality (AR), and mixed reality (MR) technologies. By applying immersive technologies in real design practice, designers are able to experience and evaluate unbuilt projects in a more efficient way. The definition of immersive technologies “Immersive technology is an integration of virtual content with the physical environment in a way that allows the user to engage naturally with the blended reality. In an immersive experience, the user accepts virtual elements of their environment as part of the whole, potentially becoming less conscious that those elements are not part of physical reality.” (Wigmore, 2018) Suppor t ing techno log ies fo r immers ive experiences include AR, MR and VR headsets, 3D displays, 3D audio, gesture recognition, spatial sensing, speech recognition, haptics, drones, cameras and omnidirectional treadmills. Application of immersive technologies in spatial design Immers ive technology is most ly used in exhibition spaces and recreational spaces. TeamLab is an architectural group that explores new architecture and space through digital technology and art that crosses architectural boundaries. Using digital projection technology, they have created a ser ies of immersive interactive art exhibitions that showcase various experiences of digital nature (Fig. 1). Space Popular is also a research-driven art, design, and media studio that explores the future of spatial experience through virtual reality. They have conceptualized galleries for live architecture talks in an immersive virtual format for the Spanish festival, Arquia Proxima (Fig. 2). The future of immersive technologies Although the application of immersive technology in real practice is still in the beginning stage, the experts and relevant practitioners assume that the 5G era will accelerate new innovations of immersive technology, which could be as widespread as smartphones. Immersive technology has been widely adopted and commercial ly successful in the video game industry. Immersive technology creates experiences that approximate physical displays and are therefore well suited as a means of telling a story rather than just demonstrating p roduc ts . By mak ing the mos t o f th i s increasingly pervasive technology, architects can bring people into a new reality through immersive technology. For designers, immersive technology has huge potential to change the way of design and better immerse users in the design process. By clearly seeing what can be changed and linking the design results to the users, the design process becomes easier. Immersive technology and new spatial experiences ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 1514 / Illusion Space // Introduction Figure 1: 10.000 Moving Cities SAR. Marc Lee, 2013 Figure 2: 10.000 Moving Cities AR. Marc Lee, 2018 Figure 3: 10.000 Moving Cities VR. Marc Lee, 2016 // INTRODUCTION / Background To some extent, virtual space design can be seen as a threat to one of the fundamental dimensions of architecture, its material ity. Materiality is put at risk by the contrast between the richness of real physical experience and the abstractions created by technology-determined env i ronments. The oppos i t ion between materiality and immateriality brings challenges to architects’ exploration of virtual space design. In this case, how to make architects embrace new technologies while taking into account both sides becomes a worthwhile question to explore. Boundaries and limitations Virtual space design can ignore the material and scale of architecture, and its close relationship to mechanical properties such as weight, thrust, and resistance. In virtual space, forms seem to change freely, with no limits other than those imposed by the program and the designer's imagination. This freedom without boundaries is always deeply disturbing, and it seems to threaten the very nature of architecture discipline (Picon, 2011). Virtual space design can almost have infinite possibilities due to its lack of physical limitations, but designers cannot explore in a huge maze. When forms are infinitely variable, decisions can only be made by establishing design principles for virtual space. At this stage, the total disconnect from the physical world makes architects uneasy, since architect is a profession whose roots go back 3,000 years and still explores the materiality of our living environment. AR: exploring the intermediate between the virtual and reality In virtual reality, the users' perception of reality is completely based on virtual information. In augmented reality, in addition to data collected from the real world, addit ional computer- generated information is provided to the user to enhance the user's perception of reality. The difference between AR and VR is that in AR, part of the surrounding environment is “real”, just layers of virtual objects are added to the real environment. The notion of augmented reality conveys the idea of different materiality made possible by the hybridization of the physical and the digital. For architects, this intermediate between the virtual and the real makes it easier to manipulate and explore the new spatial experience that virtual spaces can produce. Ways to achieve immersive experience using AR or VR technology There are a lot of approaches to achieve immersive experience. Take Marc Lee’s work for example: The first approach is to project the video onto real objects, so observers can get an immersive experience without any devices (Fig. 1). The second approach is to experience the AR space through the screen of the mobile phone (Fig. 2). And the third approach is a fully virtual environment provided by a head-mounted display (Fig. 3). The last two approaches are device-assisted while the projective display in the first approach can be experienced without a device. The opposition of materiality and immateriality // INTRODUCTION / Background ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 1716 / Illusion Space // Introduction // INTRODUCTION / Background Figure 1: Video See-through Augmented Reality. Pokemon Go Figure 2: Optical See-through Augmented Reality. Holographic Projection Using in Concert, 2019 Figure 3: Projective Display Augmented Reality. Interactive performance, 2017 // INTRODUCTION / Background Augmented reality technology achieves the experience beyond reality by superimposing information from the real world and the virtual world. It features the integration of various components of the digital world into people's perception of the real world, not simply by displaying data, but by incorporating immersive sensat ions that a re cons idered natura l components of the real world. The definition of augmented reality Augmented reality complements the real world with virtual objects that appear to coexist in the same space as the real world. AR can be defined as a system that incorporates three basic features (Van Krevelen, 2010): • combines real and virtual objects in a real environment; • registers (aligns) real and virtual objects with each other; and • runs interactively, in three dimensions, and in real-time. The three aspects indicate that the definition of augmented reality is not limited to specific display technologies, such as head-mounted displays or hand-held displays. And this definition applies not only to human vision, but to all senses, including hearing, touch, and smell. The technology of visual display There are basically three ways to visually present an augmented reality (Van Krevelen, D, 2010): • The first one is video see-through, where the virtual environment is replaced by a video feed of reality and the AR is overlayed upon the digitized images (Fig. 1). • Another way is optical see-through, it leaves the real-world perception alone but displays only the AR overlay by means of transparent mirrors and lenses (Fig. 2). • The third approach is to project the AR overlay onto real objects themselves resulting in projective displays (Fig. 3). According to the display position between the viewer and the real environment, AR displays may be classified into three categories: head- worn, hand-held, and spatial. The first two types of AR displays are device-assisted, with restrictions on mobility and experiencers can easily sense the boundary between virtuality and reality. The last type is statically placed in the environment, including screen-based video see- through displays, spatial optical see-through displays, and projective displays. See-through displays need to overlay the content on the screen with the real world to achieve AR effect, which limits the visual field and makes it easy for viewers to perceive the barrier between the virtual and real world. By contrast, the projective display can be experienced without a device and has an unlimited field of vision. Augmented reality technology ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 1918 / Illusion Space // Introduction // INTRODUCTION / Background Figure 1: Visual display techniques and positioning. Bimber and Raskar, 2005 Figure 2: Spatial projective setup. Bimber and Raskar, 2005 real object virtual object video projector real light source re al e nv iro nm en t observer image retinal display projector projector projector head- mounted display head-attached spatialhand-held hand-held display spatial see-through display real object // INTRODUCTION / Background Projection Mapping “Projection mapping, also known as video mapping and spatial augmented reality, is a projection technology used to turn objects, often irregularly shaped, into a display surface for video projection. These objects may be complex industrial landscapes, such as buildings, small indoor objects or theatrical stages.” (Wise, 2017) Spatial Augmented Reality (SAR) increases the experience of real-world objects and scenes without the use of attached displays such as monitors, head-mounted displays or hand-held devices. SAR makes use of digital projectors to display images or videos onto physical objects. The key difference in SAR is that the display is separated from the users of the system. The projection is attached directly to the physical object, and the virtual image is the furthest from the observer among different display techniques (Fig. 1). Spectators can experience the effects of SAR without using any equipment. Since this kind of display does not have a direct relationship with the spectators, it allows multiple audiences to participate in the same experience and allow for interaction between them. The working principle of projection mapping Specialized software is used to spatially map 2D or 3D objects on the virtual program which simulates the real environment it is to be projected on (Fig. 2). Interacting with a projector, any desired image can fit onto the surface of a particular object. Based on this technique, designers can add additional dimensions, optical illusions, or impulse of movement onto static objects. An audio-visual narrative is usually achieved by combining video with audio, or triggered by audio. The application of projection mapping Projection mapping was first known by the public through the form of advertising. Many commercial companies use video projection to advertise their products in public spaces around the world and use this technology to project scenes onto the facades of buildings. Projection mapping can also be interactive to interact with human behavior. Besides, images can be projected onto flat surfaces or onto irregular objects, such as cars or furniture. The use of projection mapping in fi lm and theater is becoming more and more popular. The director uses projection mapping to create an immersive environment, transforms the content of the projection according to the needs of different scenes, or makes the projection interact with the actors to form a more dynamic experience. Artists can use projection mapping as a form of presentation because it is a technology that has the potential to transform 2D artworks into 3D illusions, connecting with the audience in a way that merges the real and the virtual. In this way, artists can show their work anywhere, because any place can be the stage. People can also use it as a tool of public participation to create in public spaces or to participate in a part of creation together. ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 2120 / Illusion Space // Introduction // INTRODUCTION / Discourse Figure 1: Allegory of the Jesuits’ Missionary Work. Andrea Pozzo, 1691-1694 Figure 2: Zebra. Victor Vasarely, 1938 // INTRODUCTION / Discourse Optical illusion art Optical i l lusion art is usually through some means to make a flat painting to be three- dimensional from a specific angle. It uses human perception to enable their eyes to trick their brains. By manipulating patterns, shapes, colors, materials and forms, optical artists strive to create dazzling illusions that confuse spectators and make them see more than is actually there (Barcio,2016). The history of optical illusion art Optional illusion Art has its roots in a technique called trompe-l'œil, which means “deceive the eye” in French. As far as the ancient Greek period, the artists had already experimented to paint in a realistic way, aiming to fool the audience into feeling the painted things were actually existing. (Barcio,2016). Architect Filippo Brunelleschi pioneered linear perspective by researching perspective and proportions. This gave impetus to the optical illusion art, allowing artists to manipulate space and depth not only in painting, but also in architecture space. This style was often seen in ceiling paintings and wall paintings during the Renaissance period. The technique is called quadratura, which means to open up an architectural space such as the ceiling. One famous example is the painting Sant’Ignazio (1685) on the dome of the Jesuit church (Fig. 1). The Optical Art movement of the 1960s was started by artist Victor Vasarely, who used abstract black and white lines or patterns to create the illusion of three-dimensional space. One of his famous painting, Zebras (1938), depicts two zebras intertwining with one another, and their black and white stripes seem to merge with each other. Vasarely painted zebras because the patterns on their bodies are easy to adapt to optical illusions (Fig. 2). Illusions and space manipulation To create optical illusions, the artists need to control many things. They need to utilize the spectators’ cognitive illusions which are arisen from interaction with assumptions about the world to create and manipulate spaces to appear as if they move, change shapes, or perspectives. They also need to carefully design the viewpoint and the angle of observation of the audience to provide them with the perfect location to experience the i l lusion. At the same time, the artists need to combine two- dimensional graphics and three-dimensional space to create a new spatial experience. For example, to make the ceiling of the church appear higher, or to extend the space to the distance, it is necessary for the two-dimensional graphics to continue the perspective lines of the original space. Projection mapping technology brings more possibilities to spatial optical illusions. Artists can design different projections depending on the space to be projected, changing the experience of the space at any time, or they can project videos onto physical objects to create a more dynamic illusion. By manipulating the physical space and overlaying virtual projections on it, the spatial experience can be greatly enriched. ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 2322 / Illusion Space // Introduction // INTRODUCTION / Method Method This thesis starts with the case studies of projection mapping technology, understanding the potential of spatial effects that can be achieved by such technique. Based on this technique, series of experiments has been designed to explore the changes of physical spatial experience, as well as spatial manipulations that adapt virtual projections. These principles are aiming to simulate the spatial effects that projections can achieve in a speculative way. There are two lines of design in this thesis: one is the design of the physical environment that will be projected, and the other is the design of the virtual projection content. The combination of projected images and the physical environment is the main focus of the design, and the architect controls the boundary and relationship between real and virtual. In this way, architect can create an unpredictable augmented reality space by manipulating the form and materiality of the physical space and the brightness, direction, and scope of the projection. Physical Model + Projection Images // INTRODUCTION / Workflow Workflow After the object which will be projected and the image or video which to project are chosen or created, software is used to map the corners of the video to the surfaces. First, one needs to set up the projector to connect with After Effect. Then adjustments are commonly made by manually adjusting either the physical or virtual scene for best results. To edit the projection, one can cut out part of the image or video in After Effect and move it around until it fits the corresponding part on the physical object. Projected Object After EffectProjector Projector Projected Space ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 2524 / Illusion Space // Introduction - What kind of elements can be actively designed in a design process combining projection mapping technology? - What effects can be created in a space augmented by projection mapping? - How can an "illusion space" be made more immersive? - How is the process of spatial design reshaped by projection mapping technology? // INTRODUCTION / Main Questions & Objectives Main Questions Objectives This thesis aims to explore a new design method based on projection mapping (also known as spatial augmented reality technology), which treats virtual layers as a new type of building material attached to the body of physical space. Through a series of experiments to explore the spatial effects that projection mapping can bring, and the operation methods to achieve these effects. These methods are used to design an immersive theater where the audience can follow the actors in a sequence of space to experience the narratives. The projection layer allows the scenes to change with the movements of the actors and the audience, and conveys a spatial experience that traditional architectural design cannot achieve. // INTRODUCTION / Delimitations Delimitations Based on the idea in The Second Digital Turn, digital technology is used as a tool for thinking in this thesis. It focuses on exploring the impact projection mapping techniques can bring to the spatial design process, so the exploration of the possibilities of the technology is the main goal of the thesis. The process of research is also an important part of the final product, and the design of the theatre is also a part of the research. Projection Mapping Technology: all the methods in this paper are carried out under the premise of using projection mapping technology. Form Finding: for the design of the physical space that serves as the body of the projected image, the focus is on the process of form-finding. Narrative Space: projection mapping technology can enhance the experience of spatial storytelling, which is reflected in the immersive theater design. This project is aiming to use the theater as a medium for displaying the results of experiments, rather than simply constructing a functional space. It can have different shapes according to different narratives, which can be regarded as one of the possibilities to apply projection mapping technology. Therefore, the project is not located on a certain site, nor does it consider the materials and construction. Instead, the project focuses more on the creation of the form combining spatial design and projection mapping technology. Local Context: there is no certain site for this project, it can be built in any suitable indoor space. Material: the material of the theater should be suitable for being projected as screens. Tectonics: the structure of the theater should be lightweight and easy for assembly and disassembly. DESIGN EXPERIMENTS 02 - What effects can be created in space by projection mapping? - How to make the illusion space more immersive? - Challenges & Possibilities ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 2928 / Illusion Space // Design Experiments // DESIGN EXPERIMENTS / What effects can be created in space by projection mapping? What effects can be created in a space augmented by project ion mapping? According to the previous case studies, projection mapping technology can achieve a variety of spatial effects, such as changing the shape of objects, changing the shape of space, expanding or shrinking a space, or creating some non-existent objects or textures. In this section, the research focuses on the experience that the space can convey to the audience. As the projected image changes, so does the viewer's experience. They may experience changes in the atmosphere of the space, such as the transition from indoors to outdoors, from narrow rooms to wide halls. By combining the changes of shape, scale, material and other characteristics of the space, audience can experience complex spatial atmospheres. ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 3130 / Illusion Space // Design Experiments // DESIGN EXPERIMENTS / Changing Surface The Original Space Projected Space - Outdoor Effect Projected Space - Indoor Effect Changing Surface Project ion mapping can a c h i e v e t h e e f f e c t o f changing surfaces. Surface changing can affect human p e r c e p t i o n i n s p a c e . Projecting different materials or textures on the surface, s imu l a t i ng t he l i g h t i ng env i ronment , or add ing characters or objects that can imply the nature of the space are basic methods to change the atmosphere of the space and switch between different scenes i n t h e s a m e p h y s i c a l environment. // DESIGN EXPERIMENTS / Changing Surface ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 3332 / Illusion Space // Design Experiments // DESIGN EXPERIMENTS / Extending Space The Original Space The Extended Space // DESIGN EXPERIMENTS / Extending Space Extending Space Projection mapping can create the effect of extending space. Architects can use the perspective of shadows and lines projected into the physical environment to provide the observers with the illusion that the space is being extended. Due to the perspective of the projection content, the viewpoint of the observers is better to be restricted, otherwise, they will not be able to get a convincing optical illusion experience. ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 3534 / Illusion Space // Design Experiments // DESIGN EXPERIMENTS / Changing Shape Changing Shape Changing the shape of space is also one of the effects that projection mapping can create. Architects can cover the original physical environment with shadows, grids or patterns with perspectives of space in other shapes, making the observers feel that the space has been transformed. This effect can be better achieved by controlling the brightness of the projected light, hiding the shape information such as the edges and corners of the physical space, thereby highlighting the virtual projection overlaid on it. // DESIGN EXPERIMENTS / Changing Shape The Original Space The Original Space ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 3736 / Illusion Space // Design Experiments // DESIGN EXPERIMENTS / Creating Objects // DESIGN EXPERIMENTS / Creating Objects Creating Objects Projection mapping can create objects in space that do not exist. By projecting shadows of objects on the surface, the observers can perceive the existence of virtual objects. By carefully designing virtual and real objects, architects interweave them so that it is more difficult for the observers to distinguish between the real and the virtual. The Original Space The Created Objects ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 3938 / Illusion Space // Design Experiments // DESIGN EXPERIMENTS / How to make the illusion space more immersive? How can an "illusion space" be made more immersive? In an immersive space, how to make the audience more believe in what they see also needs to be considered in the design. The projected image will always have a certain d i f ference f rom the phys ica l space due to technical and environmental l imitations. Therefore, how to minimize these differences, blur the boundary between the real and the virtual, and thereby "deceive" the audience's senses is the main task of the designer. In this section, the main research objects are space and objects inside. This issue is explored from the details of the objects and the arrangement of the space. For example, objects with rounded corners are more likely to be changed by the projected image than those with square corners due to their blurry boundaries. In the spatial layout, projecting a space with more layers can make it more difficult to distinguish the projected virtual object from the real object. ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 4140 / Illusion Space // Design Experiments // DESIGN EXPERIMENTS / Rounded corners & Square corners // DESIGN EXPERIMENTS / Rounded corners & Square corners Rounded corners & Square corners This is an experiment about rounded corners and square corners. Project the image of a square corner on the model of a rounded corner and project the image of a rounded corner on the model of a square corner. Through this experiment, it is found that rounded corners are more likely to change shape when it is projected with images of other shapes. ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 4342 / Illusion Space // Design Experiments // DESIGN EXPERIMENTS / Rounded corners & Square corners // DESIGN EXPERIMENTS / Rounded corners & Square corners Rounded corners & Square corners Another experiment is about cubes with rounded corners and square corners. The images are projected in the same way. The image of the square-corner cube cannot fit into the outline of the rounded-corner cube, so it doesn’t perform well. On the contrary, the square-corner cube can contain the image of the rounded-corner cube, but the sharp edge cannot be erased by the projection. ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 4544 / Illusion Space // Design Experiments // DESIGN EXPERIMENTS / Overlap of real and virtual layers // DESIGN EXPERIMENTS / Overlap of real and virtual layers Overlap of real and virtual layers This experiment is about the overlap of real and virtual layers. If there is only one layer of an image projected on a single surface, it is easy to be distinguished, even if it has perspective. However, if there are multiple surfaces that can be projected, and increase the overlaps of virtual and real layers, this can increase the difficulty for the observers to distinguish the illusion. ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 4746 / Illusion Space // Design Experiments // DESIGN EXPERIMENTS / Overlap of real and virtual layers // DESIGN EXPERIMENTS / Overlap of real and virtual layers Overlap of real and virtual layers Similarly, in this experiment, there are three windows that look similar. The leftmost window, with only one layer of projection and surface overlap, is the most unreal. The central window is harder to distinguish because the projection is sandwiched between two real surfaces. On the far right is an actual window with no projection covered. By comparing the three, we can know that the alternating layers of virtual and real can increase the sense of immersion of space. ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 4948 / Illusion Space // Design Experiments // DESIGN EXPERIMENTS / Challenges & Possibilities The Viewpoint of The Spectators As the spectator moves, the viewpoint changes. But if the perspective of the projection does not change with people's movement, the immersion of the space will disappear. And if there are multiple observers in the space, it is more difficult to make the projection fit everyone's perspective. The Position of The Projector If the observer moves between the projector and the projected object, the light will not reach the projected object and the projection mapping will be disturbed. In order to get a complete projection, it is necessary to avoid the crowd blocking the light from the projector. Environmental Conditions Projection is very much influenced by the environment. Ordinary projectors are not bright enough to get a clear projection during the daytime, so it’s better to use the projection mapping in areas with less natural light or during the nighttime. Possibilities There are many challenges in creating immersive spaces with projection mapping technology, and these restrictions also provide clues for spatial design Immersive theater is a spatial prototype that is suitable to design with projection mapping. The architect can design a touring path so that the observers are always in the designed perspective. In addition, the actors in the theater can also attract the attention of the audience and guide the audience to move in the theater in sequence. In addition, the projector can be installed on the ceiling, and then through the design to limit the movement of people, so that they will not be in the projection range of the projector. Moreover, theater space often utilizes artificial light, so they are not disturbed by daylight. Also, the theater needs some immersive scenes for the audience to engage in the play, and projection can create multiple spatial experiences in limited space. Projection mapping can meet the performance needs of immersive theater, and its physical environment and spatial sequence also create suitable conditions for projection mapping. // DESIGN EXPERIMENTS / Challenges & Possibilities DESIGN - IMMERSIVE THEATER 03 - Narratives - Manhattan Transcripts - Movements & Events - Collection of Different Scenes - The Physical Body of The Theater - Location of The Projectors - Tour Route in The Theater - Rendering of Different Scenes ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 5352 / Illusion Space // Design - Immersive Theater // DESIGN - IMMERSIVE THEATER / Narratives The Film Rashomon Rashomon tel ls the story of a mysterious murder. A samurai is found dead in a quiet bamboo grove. One by one, the cr ime’s only known witnesses recount their version of the events that transpired. But as they each tell the tale, it becomes clear that every testimony is plausible, yet different. And each witness impl icates themselves. Samurai’s Wife Rashomon The Forest The Court Samurai Robber Monk Woodcutter Main Charaters Main Scenes Witnesses // DESIGN - IMMERSIVE THEATER / Manhattan Transcripts Manhattan Transcripts “Architecture is not simply about space and form, but also about event, action, and what happens in space.” (Tschumi, 1981) Manhattan Transcripts is the theoretical work of architect Bernard Tschumi. He developed the concept that programs can become generators of form, and an attempt at this concept is embodied in a series of photographs and drawings. The movements of the characters are the functional program, and the architecture created from it is the form that follows the function. Manhattan Transcripts can provide a theoretical reference for the form creation of theater design. But as a set, the theater design is not just about the relationship between space and use, it also needs to have a connection with the script. So this project tries to find a more suitable method while combining Manhattan Transcripts. "The Transcripts aimed to offer a different reading of architecture in which space, movement, and events are independent, yet stand in a new relation to one another, so that the conventional components of architecture are broken down and rebuilt along different axes." Bernard Tschumi Figure 1: Screenplay By Bernard Tschumi taken from, Architecture Concept: Red Is Not A Color, 2014 ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 5554 / Illusion Space // Design - Immersive Theater // DESIGN - IMMERSIVE THEATER / Movements & Events Encounter Tracking Tempting Fight Fight samurai’s wife and robber samurai and robber Events // DESIGN - IMMERSIVE THEATER / Movements & Events Events & Movements In order to create the space of the theater, events and movements must first be extracted from the movie. Events with more movements in the movie Rashomon mostly take place in the forest, and they can be divided into five events. In the first event, the encounter of the robber with the samurai and his wife. In the second event, the robber tracks the samurai and his wife. In the third event, the robber tempts the samurai into his trap. In the fourth event, the robber fights with the samurai's wife. In the fifth event, the robber fights with the samurai. The drawings of the movement are created according to these five scenarios and serve as the basis for the plan layout or sections. Then, these five events are summarized into four scenes: encounter, tracking, tempting, and fight. Abstracted geometries are extracted from the screenshots of the film selected from four scenes, so as to build the basis for the form of the set in the theater. In addition, there are also two static scenes: Rashomon and the court. Since these two scenes are already architectural spaces, there is no need to re-create them, but to abstract and simplify them. Overall, the above operation is to extract the spatial layouts and forms of the sets in different scenes, which can be used as a basis to build the sets. Samurai Samurai's Wife Robber Movements ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 5756 / Illusion Space // Design - Immersive Theater // DESIGN - IMMERSIVE THEATER / Movements & Events Encounter Tracking // DESIGN - IMMERSIVE THEATER / Movements & Events Tempting Fight ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 5958 / Illusion Space // Design - Immersive Theater // DESIGN - IMMERSIVE THEATER / Collection of Different Scenes // DESIGN - IMMERSIVE THEATER / Collection of Different Scenes The events and movements provide a sufficient basis for the creation of architectural spaces. And the 3D modeling of different scenes is necessary to prepare the materials for the design of the physical space of the theater and the projected images. Since those projected content with perspectives, not only the patterns or material, need to use the 3D model as an aid, it is also part of the design to build all the scenes that the projection can achieve. These models are then integrated into the physical space of the immersive theater. In this process, the projected images on different parts of the theater are designed, and different effects are achieved according to different narratives. How is the process of spatial design reshaped by projection mapping technology? ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 6160 / Illusion Space // Design - Immersive Theater Scene 01 Encounter The samurai and his wife were walking in the forest, passing the robber who was resting under a big tree. The samurai and the robber looked at each other from a distance, then the samurai and his wife bypassed the robber and continued on their way. At this time, the wind blew off the veil of the samurai's wife, and the robber saw the beautiful face of the samurai's wife. In this scene, the trees exist as a dividing factor, while the roots of the trees can be abstracted into some curved surfaces for the actors to lean on. // DESIGN - IMMERSIVE THEATER / Scene 01 Encounter // DESIGN - IMMERSIVE THEATER / Scene 01 Encounter ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 6362 / Illusion Space // Design - Immersive Theater // DESIGN - IMMERSIVE THEATER / Scene 02 Tracking Scene 02 Tracking Attracted by the beauty of the samurai‘s wife, the robber decides to track down the samurai and his wife. He followed the path through the forest, climbed down the hill, and caught up with the samurai and his wife. In this scene, the path is linear. The elements of the façade come from the abstraction of the forest form, while the interior space also simulates the spatial form of the path in the forest. // DESIGN - IMMERSIVE THEATER / Scene 02 Tracking ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 6564 / Illusion Space // Design - Immersive Theater Scene 03 Tempting Using valuable swords as bait, the robber led the samurai up the mountain along the path to find the tomb. On a meadow in the mountain, when the samurai was unprepared, the robbers attacked him and tied him up. In this scene, the road up the mountain becomes the main component of the space, and the stairs are formed according to the drawing of the movement. In addition, the shape of the façade comes from the abstraction of the woods, and the wavy shape of the inside of the walls are also a simulation of the growth of plants on both sides of the road. // DESIGN - IMMERSIVE THEATER / Scene 03 Tempting // DESIGN - IMMERSIVE THEATER / Scene 03 Tempting ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 6766 / Illusion Space // Design - Immersive Theater // DESIGN - IMMERSIVE THEATER / Scene 04 Fight Scene 04 Fight The robber lured the samurai's wife to the same locat ion and had a f ight with the samurai's wife, fol lowed by another f ight with the samurai. This scene is a meadow in the mountains. As the movements of the characters are more complicated, the plane layout is mainly formed by the drawings of the movements. Then, some holes are opened in the wall according to different angles of views, simulating various sight lines and paths through the woods. // DESIGN - IMMERSIVE THEATER / Scene 04 Fight ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 6968 / Illusion Space // Design - Immersive Theater // DESIGN - IMMERSIVE THEATER / The Physical Body of The Theater The Physical Body of The Theater The main body of the theater is an integration of different scenes, in other words, it is a hybrid of the collected models in the last step. This hybrid incorporates formal elements from different models of scenes and thus provides a complete set for the narratives in Rashomon. The parts framed by the red dotted line in the diagram are respectively linked with the independent models, indicating the relationship between them. At the same time, the overlaps of the dotted boxes also represent the hybrid between the different elements. ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 7170 / Illusion Space // Design - Immersive Theater // DESIGN - IMMERSIVE THEATER / Location of The Projectors 70 / Illusion Space // Design - Immersive Theater Location of The Projectors In addition to the physical body, the projectors are also an important part of the immersive theater. On both sides of the theater are scaffoldings, and the platforms on them serve as the standing point for the audience to participate in the performance. At the same time, the projectors are also suspended on the scaffoldings, which can be moved along the tracks as required to provide projections for the facades of the theater. And the projectors are also installed on the inner walls of the theater to provide projections to the interior space. ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 71 ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 7372 / Illusion Space // Design - Immersive Theater Tour Route in The Theater The actors move to different positions in the theater according to the narratives. At the same time, the audience moves in the order of designed standing points in the main body of the theater and on the platforms of the scaffoldings following the different positions of the actors. Every time the audience moves to a standing point, the projectors are set to project the required images onto the physical body of the theater according to the audience's perspective. The purpose is to ensure the accuracy of perspective and bring an immersive experience to the audience. Audiences can have a wider view when watching the performance on the platforms of the scaffoldings, which provides the audience with bystander perspectives as needed in the script. When the audience is inside the main body of the theater, they are more involved and can interact with the actors. // DESIGN - IMMERSIVE THEATER / Tour Route in The Theater // DESIGN - IMMERSIVE THEATER / Tour Route in The Theater ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 7574 / Illusion Space // Design - Immersive Theater // DESIGN - IMMERSIVE THEATER / Rendering of Different Scenes The Immersive Experience in The Theater In order to simulate the spatial experience that the audience can get in the theater, the model of the theater was rendered by computer and then processed by Photoshop. Due to technical limitations, there are difficulties to set up the theater space on a real scale. According to the experience obtained from the experiments, the augmented reality experience that projection can bring to the audience is simulated in a speculative way. In the rendering process, light is projected onto the wall from the locations of projectors, simulating the indoor light environment that projections can create. On this basis, the projected images are overlaid on the corresponding part on the wall to obtain a simulated spatial rendering. // DESIGN - IMMERSIVE THEATER / Rendering of Different Scenes ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 7776 / Illusion Space // Design - Immersive Theater // DESIGN - IMMERSIVE THEATER / Rendering of Different Scenes ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 7978 / Illusion Space // Design - Immersive Theater // DESIGN - IMMERSIVE THEATER / Rendering of Different Scenes ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 8180 / Illusion Space // Design - Immersive Theater // DESIGN - IMMERSIVE THEATER / Rendering of Different Scenes ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 8382 / Illusion Space // Design - Immersive Theater // DESIGN - IMMERSIVE THEATER / Physical Model of Rashonmon Scene REFERENCES 04 ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 8786 / Illusion Space // References // REFERENCES / James Turrell's Work with "Light and Space" Ganzfeld, 2009 Wedgework, 1974 Corner Shallow Space, 1968 Tunnel Pieces, 1999 // REFERENCES / James Turrell's Work with "Light and Space" Ganzfeld “Ganzfeld” is a German word to describe the phenomenon of the total loss of depth perception as in the experience of a white- out. When everything in the visual field is the same color and brightness, the visual system shuts down. Corner Shallow Space Corner Shallow Space uses light to make the illusion of a three-dimensional object in convex corners. The projected light is overlapped on the original physical object, using its own characteristics, such as edges and corners, shadows, to create new objects or change the shape of the object. Wedgework Wedgework uses projected light to create an illusion of virtual walls. Through the projection on the wall, the space is further expanded, which af fects people’s percept ion and changes the shape of the space from the perspective of the viewers. Tunnel Pieces Tunnel Pieces uses projected light to create an illusion of expanded tunnels. It creates a sense of space by projecting l ight of different brightness on both sides of the tunnel, providing the viewer with the illusion of widened tunnel. ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 8988 / Illusion Space // References // REFERENCES / Box - Bot & Dolly // REFERENCES / Box - Bot & Dolly Transformation Shadows are projected on the screen to create the illusion of a box. This is a simple way of creating three-dimensional space on a two-dimensional screen. Box Box explores the combination of real and virtual through projection-mapping on moving surfaces. This work is served as both an artistic statement and technical demonstration. This methodology has tremendous potential to radically transform theatrical presentations, and define new genres of expression. Levitation Two different patterns are projected on two screens, and the illusion of a hole is created in the screen through the matching of two patterns. At the same time, the interaction between the two screens makes people feel that the spheres on the bottom screen are sucked to the other screen through the virtual hole. Intersection Shadows and t e x t u r es a r e p r o j e c t ed on t o t he sc r een simultaneously, creating concave or convex corners to create the illusion of space. The direction of the texture can emphasize the illusion of spatial perception. ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 9190 / Illusion Space // References // REFERENCES / 3D Projection Mapping Installation "FORMATS" - Anne Marie Himmelmann // REFERENCES / 3D Projection Mapping Installation "FORMATS" - Anne Marie Himmelmann 3D Projection Mapping Installation "FORMATS" Formats is a 3D projection mapping (videomapping) on pillar-like objects and the floor. It uses only simple animation, a cheap projector, dimmable lights and a laptop. The animation is projected on the pillars, giving the pillars different morphing and material changes, such as twisting the pillars, turning them into a paper-like material and causing wrinkles, and creating shadows of holes in the pillars. Various new objects created by the projection may not fill the entire pillar, but can only be restricted to the outline of the pillar. At the same time, by controlling the brightness of the lights, the illusion is less perceivable to the viewers, thus creating an immersive perception. ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 9392 / Illusion Space // References // REFERENCES / Kipling Indoor Projection Mapping Show - Pixel Artworks Indoor Projection Mapping Show Kipl ing indoor projection mapping show recreates the volume of objects by projecting shadows on them and giving them a variety of colors. The shadow projected can also create the illusion of depression on the flat surface, matching the outline of the original object to create different letters which form a meaningful pattern. // REFERENCES / The Cave - GASTA The Cave The cave creates the illusion of a flowing corridor. The room must be as dark as possible and there must be enough space for the projector to “throw” its light onto the surface. It controls the l ight intensity and textures at the same time to create a dynamic cave. The black and white worked perfectly to enhance the contrast of the projection itself. ACEX35 - Master's thesis in Architecture 2022 // Spring Term / 9594 / Illusion Space // References // REFERENCES / Sony Realtime Projection Mapping - Marshmallow Laser Feast // REFERENCES / Sony Realtime Projection Mapping - Marshmallow Laser Feast Expanded Space By keeping the space dark and projecting only high-brightness grid textures, all shadows are eliminated, which can reshape the viewer’s perception of the space. If the perspective of the grid is adjusted so that the grid projected on the wall seems to extend further, the illusion of an expanded space can be created. Reshaped Space When the space is dark enough, it can avoid the visual interference of shadows to the viewers. And in this case, the edges of the room can also be erased. The shape of the space can be reshaped by the high-brightness grid textures projected on the surface. If a grid of a sphere is overlaid on the space, the viewers will have the illusion that the space has been reshaped. DISCUSSION 05 DISCUSSION Based on the research of projection mapping technology and its current application, this paper investigates its possible value in architectural design. It aims to contribute to the discourse of the second digital turn from the perspective of one of the immersive technology and use this technology as a design tool. In this context, the thesis explores transformations that new technologies can bring to architecture design and try to expand the boundaries of the architectural discipline. The thesis consists of two parts, one is the experiments, and the other is the design of an immersive theater. It uses conclusions drawn from the experiments to speculatively imagine possible application scenarios with projection mapping technology. With the method of research by design, the design experiments are used as the main tool to explore the interaction between physical objects and virtual projection, as well as to simulate the spatial effects that projections can achieve speculatively. And it uses the theater as a medium for displaying the results of experiments, rather than simply constructing a functional space. It can have different shapes according to different narratives, which can be regarded as one of the possibilities to apply projection mapping technology. The thesis raises four questions. The first question is researched throughout the whole process. The answers to the other three questions are explored in the part of experiments and the design of an immersive theatre. The main question “what kind of elements can be actively designed in a design process combining projection mapping technology” is also answered in the research method in the thesis. The first is the form of the physical space, and the second is the content of the projection. Designers need to make them fit together and interact with each other, and manipulate the shape, size, and material of the space in the process. The thesis also has potential for improvement, especially in the representation of the project. Because there are difficulties to get technical support, such as VR and AR, and it is also hard to set the devices in a real-scale space, it is impossible to present an immersive experience to the audience. BIBLIOGRAPHY // BIBLIOGRAPHY / Literature & Websites Carpo, M. (2017). The second digital turn. Cambridge: MIT. Handa, M., Aul, G., & Bajaj, S. (2012). Immersive Technology – Uses, Challenges and Opportunities. International Journal Of Computing & Business Research. Krautsack, D. (2011). 3D Projection Mapping and its Impact on Media & Architecture in Contemporary and Future Urban Spaces. Journal Of The New Media Caucus, 07(01). Picon, A. (2011). Architecture and the Virtual Towards a new Materiality?. Praxis: Journal Of Writing + Building. Tschumi, B. (1981). The Manhattan transcripts. London: Academy Editions. Van Krevelen, D., & Poelman, R. (2010). A Survey of Augmented Reality Technologies, Applications and Limitations. International Journal Of Virtual Reality, 9(2), 1-20. Literature Barcio, P. (2016). Optical Illusion Art That Marked the 20th Century. Retrieved 10 March 2022, from https://www.ideelart.com/magazine/optical-illusion-art Illusion Art – Delving into the Deceptive World of Optical Illusion Art. (2021). Retrieved 10 March 2022, from https://artincontext.org/illusion-art/#Famous_Illusion_Paintings_Artists Schipper, D. (2022). Immersive Technologies: How Architects Can Position Themselves For Future Success. Retrieved 21 February 2022, from https://cuningham.com/2020/01/15/immersive- technologies-architects-position-themselves-success What is Projection or Spatial Augmented Reality? - AREA. (2017). Retrieved 10 March 2022, from https://thearea.org/projection-augmented-reality/ Wigmore, I. (2018). What is immersive technology? - Definition from WhatIs.com. Retrieved 17 February 2022, from https://whatis.techtarget.com/definition/immersive-technology Wise, Z. (2017). Projection Mapping: Spatial Augmented Reality. Retrieved 10 March 2022, from https://studio.knightlab.com/projects/projection-mapping/ Websites // BIBLIOGRAPHY / Images & Videos Bialick, Z. (2022). [Image]. Retrieved from https://www.sadplaw.com/pokemon-go-harm-child/ Bot & Dolly. (2013). Box [video]. Retrieved from https://www.youtube.com/watch?v=lX6JcybgDFo GASTA. (2014). The Cave [video]. Retrieved from http://www.gasta.org/portfolio/the-cave/ HIMMELMANN, A. (2021). 3D Projection Mapping Installation "FORMATS" [video]. Retrieved from https://www.youtube.com/watch?v=OrQUdTT6BHQ Lee, M. (2013). 10.000 Moving Cities [video]. Retrieved from https://marclee.io/en/10-000- moving-cities-same-but-different/ Lee, M. (2018). 10.000 Moving Cities AR [image]. Retrieved from https://marclee.io/en/10-000- moving-cities-same-but-different-ar/ Lee, M. (2016). 10.000 Moving Cities VR [video]. Retrieved from https://marclee.io/en/10-000- moving-cities-same-but-different-vr/ Marshmallow Laser Feast. (2012). Sony Realtime Projection Mapping [video]. Retrieved from https:// vimeo.com/34021153 Pixel Artworks. (2014). Kipling Indoor Projection Mapping Show [video]. Retrieved from https://www. youtube.com/watch?v=jl9-ZLj9hEA Space Popular. (2020). The virtual city [Image]. Retrieved from https://www.stirworld.com/see- features-an-architectural-conference-in-vr-by-space-popular-reflects-virtual-togetherness teamLab. (2018). Borderless World [image]. Retrieved from https://borderless.teamlab.art/ The 'holograms' revolutionising digital communication. (2019). [Image]. Retrieved from https://www. inavateonthenet.net/features/article/the-holograms-revolutionising-videoconferencing Turrell, J. (2009). Ganzfelds [image]. Retrieved from https://jamesturrell.com/work/dhatu/ Turrell, J. (1968). Corner Shallow Space [image]. Retrieved from https://jamesturrell.com/work/ afrum-pale-pink/ Turrell, J. (1974). Wedgework [image]. Retrieved from https://jamesturrell.com/work/wedgework3/ Turrell, J. (1999). Tunnel Pieces [image]. Retrieved from https://jamesturrell.com/work/thelightinside/ Images & Videos