During the late afternoon of November 9th 2016 — while the while the US presidential election results were trickling into various devices and the reality of the next four years was setting in for the people of the US and around the world — a group of researchers, academics and students gathered in Osaka to participate in a workshop focusing on storytelling in virtual reality (VR).
VR Story Telling
The workshop, “Interactive Storytelling for 360°VR”, was part of the ACE 2016 academic conference. It was organized by Mirjam Vosmeer from the Amsterdam University of Applied Sciences. The focus of the workshop, as stated in the title, was storytelling. One of the most interesting features of the workshop was that interaction which occurred did not involve the use of a head mounted display (HMD), a high-power pc or any other technological devices, the interaction was amongst the participants. The participants, from various backgrounds, was made up of men and women representing 10 countries and 5 continents. This information is important because, in her presentation Dr. Vosmeer mentions that, through her research in VR she and her colleagues have found that age and gender are not a factor in people’s enjoyment/entertainment of VR narratives. Taking into account the diverse pool of participants at the workshop, one can say that nationality is not a factor either.
VR has been making news for the past few years. Recently the term as become ubiquitous. Much of the reason for the pervasive spread of VR is the ease with one can enjoy VR, for example with a simple cardboard HMD like Google . Also, creating VR content has become easier and more hassle free with software like Unity . Now a household term, VR as a field faces a seemingly simple obstacle — how to tell a story in VR — with a variety of new challenges brought on by the technology involved along with the depth and scope of what is visually and immersively possible.
There were a significant number of academic contributions discussing VR at ACE 2016. In “You’re the Camera! Physical Movements For Transitioning Between Environments in VR”, the authors discuss the challenges involved with transitioning in a VR environment. In their study, they set out to find if scene transitions in virtual environments should be executed by users in sync with virtual camera movements or if a virtual camera should more directly influence the transition. After conducting a user study, the researchers found that users actually preferred to have less control and to be guided by a virtual camera in order for transitions to occur.
This idea of control is central to constructing VR narratives. Workshop participants also discussed and considered from which perspective a VR narrative should be told. Should it be from a first person point-of-view (POV), second person POV or third person POV? The different perspectives and voices have their own advantages and disadvantages for both those creating and those experiencing the content. Similar to the challenges posed by transitioning scenes and locations in VR, the challenge of deciding from which perspective a VR story is told affects both the storytellers and the interactive viewers. Giving more power to those experiencing VR narrative entails providing them with more tools/devices with which they can navigate inside the framework of an immersive story. In many cases, requiring the viewer to use, hold and interact with more devices is not only bothersome but takes away from the interactive/immersive narrative experience. The viewers may become distracted not be able to properly, or fully, experience the VR story. From a design and user experience, requiring the user interact with a controller or haptic device, typically held in the user’s’ hand(s), is difficult as the user is wearing an HMD. Even if all that is required of the viewer is pushing a button or buttons, the viewer is required to do something outside of the VR environment that affects the viewer’s virtual experience. In order to overcome this problem, the researchers behind “DUKE: Enhancing Virtual Reality based FPS Game with Full-body Interactions” developed a VR system integrating three existing technologies: Oculus Rift, Leap Motion and Microsoft Kinect. In the Duke environment, the leap motion sensor is attached to to the outside of the Oculus Rift HMD and the user stands in front of a Microsoft Kinect. Through the HMD, the user is an immersive virtual first-person shooter (FPS) game experience. The Leap Motion attached to the front of the HMD tracks the user’s arm and hand movements and gestures. The Kinect, used for navigation inside the virtual game world, tracks the user’s body movements, especially the legs and feet. Combining these three technologies, user’s can not only move around in the game, but they can also select which weapons to use from the combat menu and fire weapons. Through the integration of these existing technologies, the researcher have made it possible have a fully immersive and interactive VR game experience.
A seamless immersive experience is not only necessary in VR gaming, but also for VR narratives. This was one of the key topics at the VR storytelling workshop. Different from gaming however, VR narratives have to engage the viewers while balancing the narrative direction with the level of interactivity given to the user. At the workshop, video games were discussed as a “lean forward” medium. Television and movies on the other hand fell into the category of a “lean back” medium. “Lean forward” refers to the user/viewers body position and accompanying gestures when they are holding a controller while playing video games. “Lean back” refers to the viewer’s body position while on a sofa or in a movie theater seat. VR narratives are do not fall perfectly into either category; rather a new category, “Lean in”, is needed to construct and experience VR narratives. “Lean in” refers to the fact that the viewer may at times engage and interact with the story. It is an inexact middle between the two aforementioned traditional styles. One of the workshop participants, Akihiko Takashima (高島 瑛彦), of Keio University has developed VR narratives in the course of his research that are representative of “lean in” engagement. In “The Viewpoint of a Zombie” and “Police & Mafia” the viewers are able to see outside (into), as well as in and around the frame, of what is shown on the monitor by shifting their body position and the direction of their gaze. The narrative is traditional and linear in the sense that the audience is guided through the story, i.e they cannot control the scenes, timing or transitions; however, by shifting their physical body position in relation to the screen at which they are looking, the audience can see more than what fits into a typical frame. The video that appears was captured via a fish-eye lens. And shown on a commercially available monitor (tv). A Microsoft Kinect with depth-sensing was incorporated to track the user’s movement. Based on the user’s position and viewing angle, the viewers get a different perspective of what is going on in the scene. As a result, the viewers, without the use of an HMD, can enjoy an immersive narrative experience that is simultaneously similar to that of other viewers in terms of content and story with a unique and individual perspective.
Whether the “Lean In” approach is the right approach or the correct approach for making VR narratives is up for discussion. However, what is important and what was discussed at the workshop was the idea that traditional narrative or entertainment methodologies and techniques do not fit perfectly into creating VR stories. Traditional roles such as director, screenwriter, cameraman, etc. need to be re-thought and repurposed for VR narratives. In addition, new roles and story specific functions and tools need to be introduced and implemented. While VR has become a part of daily conversation in many circles, the industry is still in its nascent phase. Telling a good VR story does not primarily depend on the latest technology both in terms of hardware and software. Just like any good story, from books to movies, the audience must be engaged and entertained in order to have a enjoyable experience.