I-Sheng Lin

Tsai-Yen Li
email: li@nccu.edu.tw

Quentin Galvane
email: quentin.galvane@inria.fr

Marc Christie
email: marc.christie@irisa.fr

Design and Evaluation of Multiple Role-Playing in a Virtual Film Set

Keywords: Human-centered computing → Virtual reality, User interface programming, • Applied computing→Media arts, Virtual Reality Application, Virtual Film Set, Virtual Cinematography, Film Previsualization

Cinematography affects how the audience perceives a movie. A same story plot can be interpreted differently through the presentation of different camera movements, which show the importance of cinematography in filmmaking. Typically, filmmaking is costly, and beginners and amateurs rarely have the opportunity to play and do an experiment on a film set. In this work, we aim to design and construct a virtual environment for film shooting, allowing a user to play multiple roles in a virtual film set and emulating the process of the filmmaking. Our system provides camera shooting assistants, tools for field directing and real-time editing, aiming to help novices learn cinematographic concepts, track the progress of filmmaking, and create a personalized movie. In order to verify that our system is a user-friendly and effective tool for experiencing filmmaking, we have conducted an experiment to observe the behaviors and obtain feedback from participants with various cinematographic backgrounds.

INTRODUCTION

Several factors affect how the audience perceive and understand the story presented in a film such as the storyboarding style of the director, the shooting skills of the photographer, and the editing preference of the editor. How a film is perceived is greatly influenced by the way that the cameras are positioned and the shots are arranged. However, most novices do not have many chances to practice and learn the skills of various roles in the real filmmaking process due to the enormous costs such as the construction of the set, the personnel expenses for actors, photographers, lighting operators, and directors, just to name a few.

In recent years, the development of computer graphics technologies has revolutionized the process of filmmaking. In addition, the advances of real-time graphics and Virtual Reality (VR) technologies also make possible many novel applications. Through the Head-Mounted Display (HMD) device, motion trackers, and hand controllers, a user can immerse into a virtual environment to interact with virtual object and experience a story in the applications of education, training, entertainment and many others.

VR technologies can also be used to build a virtual film set for the training and practice of various roles in filmmaking such as director, photographer, and editor, and switch among them according to the needs of composing the shots through cinematography. In this work, we use a popular VR system (HTC Vive in our case) to design a virtual film set to emulate a filmmaking studio, in which a user can play these roles at any time of the film production to learn the important concepts through realistic operations. In addition to emulating the functions of the equipment in a common film set, we have also implemented some assisted tools such as recommendations for static or dynamic shots, camera motion stabilizer, etc. to help novice users in producing a good shot more easily.

RELATED WORK

Virtual camera control and planning has been a research topic for more than two decades. The objectives are usually to assist the users in generating camera motions to satisfy certain cinematography goals such as avoiding occlusion, keeping frame cohesion, and achieving stylistic or emotional effects. [START_REF] Christie | Camera control in computer graphics[END_REF] has provided a good survey of the common techniques and applications for camera control and planning. [START_REF] Chen | Context-aware Camera Planning for Interactive Storytelling[END_REF] have proposed a camera planning system for interactive storytelling, which allows the motion of the camera to be generated on the fly according to the storyline chosen. In order to achieve more intuitive camera control Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party coponents of this work must be honored. For all other uses, contact the owners.author(s). VRCAI '18, Dec. 2018, Tokyo, Japan @2018 Copyright held by the owner/author(s). in a virtual environment, some research has adopted interactive devices to assist the film director or cinematographer to manipulate a virtual camera. For example, [START_REF] Lino | The director's lens: an intelligent assistant for virtual cinematography[END_REF] have proposed a Director's Lens system, which adopts a motion-tracked hand-held device with a monitor screen to control the camera motion in a virtual environment for film previsualization. [START_REF] Gouchet | Scp camera[END_REF] have also proposed a similar system except for that a physical camera was used to provide a more realistic experience, which is important for training.

In modern filmmaking, many shots are taken in a studio instead of a real scene, thanks to Computer Graphics technologies, to save money or to create something infeasible to construct in real life. However, the actors often need to face the air or a blue screen to perform the action without a good idea of relative positions with other actors or objects. In order for the actors to immerse into the scene better, [START_REF] Bouville | Virtual reality rehearsals for acting with visual effects[END_REF] have proposed a virtual reality system allow the actors to rehearse a play through realistic visual effects. In addition, it is also a hassle to gather all actors for rehearsals. Therefore, [START_REF] Slater | Acting in virtual reality[END_REF] have proposed to use the concept of Shared Virtual Environment (SVE) to allow actors to rehearse remotely together with other actors or animated virtual objects.

VR has also been used as a tool to teach cinematography with the visualization of virtual studio and camera paths [START_REF] Chiu | Cinematography tutorials in virtual reality[END_REF] . Despite the efforts in the previous work, we have not found systems that can allow a user to play through and switch between different roles in a film set to experience the whole filmmaking process. Therefore, we have proposed to design a system that can take advantage of the up-to-date VR devices to allow a novice user to experience and learn the operations through the interactions with the film set and visualize the result.

SYSTEM DESIGN

System Workflow

In our system, a user can play multiple roles in the process of filmmaking in a virtual film set. These roles include the director who can decide where to put the cameras in order to get good shots to tell the story. We assume that the animations of the actors are available and can be played back and forward in the virtual studio according to the needs of the production team. The director needs to decide the number of cameras needed and where to place these cameras (static or dynamic) at rough locations. Once the cameras are created, the user can play the role of photographer or cameraman and determine a precise location for each camera by manipulating the cameras to obtain the desirable shots. Then the user can play the role of the editor and determine how the shots are selected and arranged to present the whole story. The relations between the system modules and their workflow are shown in Fig. 1. Note that the system allows a user to switch back and forward between these roles. Therefore, the user can add a new camera, change its position, reframe a shot, or re-select or adjust a specific clip at any time, even at the editing stage. This workflow is different from the traditional production flow but provides an integrated view of the production process and will allow the user to make adjustments in any stages with the least efforts in switching between the tools.

The Director Functions

Usually, the director has a good control over how the cameras are arranged through an array of monitor screens showing the current list of shots taken by the given number of cameras. In our virtual film set, we provide the director with a monitor screen as shown in Fig. 2(a) including a chosen main screen and a list of six smaller monitor screens. In order to help novice users place the cameras at their desired positions according to the storyboard, we have also provided a function to import storyboard images as a guide for the novice user to create shots similar to the target images. The storyboard display function is shown in Fig. 2

The Cinematographer Functions

According to the rough location specified by the director, the cinematographer needs to determine the configuration of each camera to produce the desired shots. Each camera is represented by a 3D camera model that can be grabbed through the controller. We have also provided several viewports to show the framing results. A special viewport design is through a detachable display panel (see Fig-

ure 3) showing the shooting result. This detachable function is especially useful when the camera is placed at a position not close to the eye level or when a camera is used to perform tracking.

The Editor Functions

How the final clips are determined and how they are arranged to deliver the story are usually finalized in the hand of the editor. Our system provides an editing board (as shown in Fig. 4) allowing the user to specify the selected shot and desirable length through the laser pointer and control handle in the virtual studio. Our editing system provides a timeline to arrange the film clips from different cameras. A user can add, delete, and adjust a clip by using the laser pointer of the controller. The editing screen also consists of a preview screen (top), a selected clips part (middle). Another main feature of this editing system is that we allow the user to go back to the director or the cinematographer modes to make an adjustment to the shots if necessary and switch back to the editor mode, which is only possible in an integrated system like ours.

Cinematography Features

Our system also provides various assisting features to make tasks of camera operations and shooting easier, especially useful for novice users. First, in the selection of static shots, we provide recommendations with an array of shots according to user-specified constraints inputted by specifying two points or two body parts of the actors on the screen. Then the system will compute the shots in six categories including Establish, Long Shot, Medium Shot, Medium Close-Up, Close-Up, and Extra Close-Up shots that appear commonly in films. A sample image taken from the camera location that can satisfy the given constraints will be generated and listed in the array of recommended shots for the user to select.

For dynamic shots, we have designed several control modes to manipulate the virtual camera or to track objects to capture the recording of the acts. The first group of camera functions allows the user to grab the virtual camera and record the trajectory of the camera by pressing the record button. Since the hand movement usually cannot be very smooth and stable for novice users, we have created a camera stabilization function utilizing a low pass filter to smooth the camera motion on the fly.

The second group of camera functions includes three camera tracking modes: Look-At, Follow, and Dolly. The Look-At and Follow modes are similar to what most animation software has where the camera will look at a specific target statically or moving along by keeping a fixed distance from the target. In the third Dolly mode, the user will lay out a dolly track as a trajectory for the camera to follow as shown in Figure 10. It is interesting to design such an interface in a VR environment since it could be more intuitive to layout the track by direct manipulation in the 3D space. The user can specify the initial and final points of the track by dragging the control balls with the hand controller as shown in Fig. 5(a). In addition, the user can drag any part of the track to change the curve (Fig. 5(b)), and the system will compute the tangential vector of the curve that is used as a constraint to generate the curve for the dolly track.

EXPERIMENT AND EVALUATION

We have designed and implemented our system with the current popular VR platform: HTC VIVE and its associated devices such as a head-mounted display, hand controller, and trackers. The software was developed using the Unity3D platform while the 3D contents were designed on a 3D modeling and animation package. The example play that we have used for our experiment is a short clip reproduced for the movie "Back to the Future."

Experimental Settings

In this experiment, we invited eight subjects to participate in experiencing and evaluating the system and see if the design goals for the various functions and as a whole system have been achieved. A survey of their experiences in using VR systems and video editing software was conducted. The subjects can be roughly divided into two groups according to their cinematography experiences: more experienced (A-D) and less experienced (E-H). All of them have used editing software. The subjects were asked to follow the same experimental procedure consisting of several steps as follows: 1) watching instruction video; 2) practicing basic operations; 3) accomplishing guided tasks; 4) accomplishing assigned task; 5) doing free practice; 6) filling survey. In accomplishing the guided tasks and the assigned task, the subjects were shown a virtual storyboard that was used as the production objective. The main difference between the two tasks is that for each guided task, instructions about how to accomplish the task were given to the subjects aside from the storyboard.

Experimental Results

At the end of the experiment, the subjects were asked to fill in a survey consisting of 22 scale questions. The scale questions use a 5-point Likert scale (1-5) with 1-5 meaning from the least to the most agree. Questions 1-5 are about the evaluation of the director functions, and the results are shown in Table 2. Most subjects considered the functions provided for the director mode are useful (scores above 4.0) except for God's perspective (Q1). From the interviews with the subjects, we learned that some people do not like this function due to some degree of Acrophobia. We also found that the more experienced subjects, the more functions he/she would expect for the director. For example, they expected the director to guide the movement and performing of the actors, which is out of the scope of this work. 5 I can experience the role of director in the VR film set.

4.00 0.87

From Table 3, we can find the feedback of the subjects on the cinematographer functions. All agreed on the function of detaching the view panel from the camera. The score for the camera recommendation is below 4.0. From the interview, we found that the user feedback is mixed. More experienced people in cinematography preferred this function better, the less experienced people preferred to position the cameras through direct interaction and manipulation. The score for the question of setting up the dolly track (Q14) is also below 4.0 while most subjects consider the dolly track function is better than the traditional tracking (Q15). We found that the main concern was on the precise speed control of the camera on the dolly track instead of adjusting the curve. Adjusting the curve of the track does not always return the desirable shot unless the user can control the progress of the camera motion, which is not available in the current implementation. Nevertheless, the overall score for the cinematographer functions (Q16) is rather high, which mean that the subjects all like the functions that we have provided for camera control. For the functions of the editor, we can find from Table 4 that all subjects considered the interface and functions clear or intuitive. However, the overall score (Q22) is not the highest. From the interview, we found that more experienced subjects expected more editing functions that can be found in many video editing software such as changing the scale of the timeline.

CONCLUSION AND FUTURE WORK

In this paper, we have described a novel VR system aiming at providing cinematography experience for novice users through a virtual film set. Some of the novel features and designs provided in the current system such as shot recommendation, automatic tracking, switching between different roles, etc. are unique in this kind of VR systems. We have also conducted experiments to evaluate the system. The preliminary results are encouraging since no matter the background of the subjects, all can find the value of such a system in practicing cinematography for filmmaking. In any case, we believe that our system has shed some lights on the future development of VR systems for filmmaking on training or previsualization.

From the feedback provided by the subjects in our experiment, more features that currently do not exist in our system are expected in the future such as zoom-in/out, the field of depth, etc. and more precise control of camera on a dolly track. A VR system like ours can have several potential applications. For example, it can be used for a training tool for novice users to learn or experience the filmmaking process. In addition, such a system can be used as a previsualization tool for real film production to confirm how a movie will be filmed.

Figure

  Figure 1: System workflow

  (b).

Figure 2 :

 2 Figure 2: (a) Camera monitor panel and (b) Storyboard display function

Figure 3 :

 3 Figure 3: Emulating virtual camera through a 3D camera model and detachable panel screen. (a) attached (b) detached

Figure 4 :

 4 Figure 4: Film editing board for the editor

Figure 5 :

 5 Figure 5: Adjusting the track by (a) dragging the ball at the endpoint or (b) any segment along the curve

Table 2 : Evaluation of the director functions

 2

	Q	Questions	AVG	STD
	1	Using God's perspective to get an overview of the setup is useful.	3.50	0.71
	2 Being able to play the animation in real-time is useful.	4.75	0.43
	3	Using the monitor screen to monitor all cameras is use-ful.	4.63	0.48
	4	Storyboard is useful for practicing how to position a camera	4.50	0.50

Table 3 : Evaluation of the cinematographer functions

 3

	Q	Questions	AVG	STD
	6	Grabbing a camera and shot is simple and natural	4.63	0.48
	7	The Cinematographer functions are simple and natural to use.	4.13	0.78
	8	The system provided camera recommendation is useful.	3.88	0.60
	9	The classification of camera recommendation is useful.	4.50	0.50
	10	Positioning the camera through camera recommenda-tion is useful.	4.25	0.43
	11 Detaching the view panel from the camera is useful.	5.00	0.00
	12	Recording the trajectory of the camera manipulation is intuitive.	4.13	0.60
	13	Specifying the body part of an actor for camera tracking is intuitive.	4.50	0.50
	14 Setting up a dolly track is intuitive.	3.63	0.86
	15	Using the dolly mode to track an actor is smoother than using the Look-At and Follow modes.	4.50	0.50
	16	I can experience the role of the cinematographer in the VR film set.	4.88	0.33

Table 4 : Evaluation of the editor functions

 4

	Q	Questions	AVG	STD
	17	The interface of the editing board is clear and easy to understand.	4.50	0.71
	18	The timeline on the editing board is clear and easy to understand.	4.75	0.43
	19 The timeline on the editing board is intuitive.	4.25	0.83
	20 Adding, editing, and removing a clip is intuitive.	4.63	0.48
	21	Switching between the cinematographer mode and the editing mode is useful.	4.63	0.48
	22	I can experience the role of editor in the VR film set.	4.38	0.86

ACKNOWLEDGMENTS

This work was supported by the Ministry of Science and Technology of Taiwan under contracts MOST107-2911-I-004-501 and MOST and MOST 106-2221-E-004-014.