Anjul Patney Nvidia

Marina Zannoli

Oculus Vr

George-Alex Koulieris

Kim Nvidia

Gordon Wetzstein

Frank Steinicke

Applications of Visual Perception to Virtual Reality Rendering

Density (per mm2) Eccentricity (degrees) Rods Cones Ganglion Cells Cell Distribution on Retina (b) A Brief Dive into Human Visual Perception High-fidelity foveal pixels Low-fidelity peripheral pixels (c) Foveated Rendering (d) Focus Cues (e) Redirected Walking

Figure 1: Our course provides (a) an introduction to the role of perception in modern VR, (b) an overview of human visual perception and modern psychophysical methods, accompanied by several case studies of using perceptual insights to improve VR performance using (c) foveated rendering, addressing the vergence-accommodation conflict by providing (d) focus cues, and improving VR immersion by enabling large virtual spaces using (e) redirected walking.

Introduction

Over the past few years, virtual reality (VR) has transitioned from the realm of expensive research prototypes and military installations into widely available consumer devices. These devices enable experiences that are highly immersive and entertaining, and have the potential to redefine the future of computer graphics. Yet, several challenges limit the practicality and accessibility of modern virtual reality Head-Mounted Displays (HMDs), including:

• Performance: The high pixel counts and frame rates of increase rendering costs by up to 7 times compared to 1920×1080 30 Hz gaming, and next-generation HMDs could easily double or triple costs again.

• Visual Quality / Immersion: Visual immersion using contemporary HMDs is limited due to several factors including image resolution and field-of-view. It is also subject to discomfort and even sickness because of various sparsely-explored factors such as incorrect visual cues and system latency.

• Physical Design and Ergonomics: Modern HMDs tend to be unwieldy and unsuitable for hours of continuous use. Further, while room-scale VR experiences successfully permit intuitive locomotion, they are limited by the size of the physical room.

This course explores the role of ongoing and future research in visual perception to solve the above challenges. Human visual perception has repeatedly been shown to be an important consideration in improving the quality of computer graphics while keeping up with its performance requirements. Thus, an understanding of visual perception and its applications in real-time VR graphics is vital for HMD designers, application developers, and content creators.

We begin with an overview of the role of perception in modern Virtual Reality. We follow this overview with a dive into the key characteristics of the human visual system and the psychophysical methods used to study its properties. After laying the perceptual groundwork, we present three case studies outlining the applications of human perception to improving the performance, quality, and applicability of VR graphics. Finally, we conclude with a forward looking discussion. 1. Establish a framework to identify key questions and aspects of perception-action systems that impact VR/AR design while building the scientific foundation required for future advances. • FoV varies by facial anatomy

Index

• Typically 60°superior, 60°nasal, 70°inferior, 100°temporal

• Binocularly combined FoV, typically 130°v ertical and 200°horizontal

Rods and Cones

• Light is detected and converted to an electrical signal by the photoreceptors on the retina (~5cm 2 )

• ~5 million Cones are clustered around the center of the retina (fovea)

• Cones are responsible for our fine detailed and color vision

• ~125 million Rods are scattered around the retina

Sensitivity

• Static contrast ratio of the retina about 100:1 (6.5 f-stops)

• Exposure re-adjusted during saccades, both mechanically by the iris and biochemically

• The equivalent of changing the aperture and the film "speed" respectively • Non-linear response: twice as many photons/sec do not appear twice as bright

• Total range about 46,5 f-stops (10 -6 -10 8 cd/m 2 )

• The function of the iris is not only to control the intensity of light coming into the eye • From 10°towards the center, rod density declines rapidly

• From 18°away from the center, rod density declines more gradually, in a curve with two distinct inflection points

• The outer edge of the second inflection point is at about 30°, and corresponds to the outer edge of good night vision

Peripheral Vision (3)

• The density of receptor and ganglion cells in the retina is greater at the center and lowest at the edges • The representation of the periphery in the visual cortex is much smaller than that of the fovea • Periphery has a relative advantage at noticing flicker • Peripheral vision is also relatively good at detecting motion Eye Movements

• Saccades • Micro-saccades • Smooth Pursuit • Vestibulo-ocular Reflex • …
Six extra-ocular muscles, allowing for elevation, depression, convergence, divergence and rolling.

Saccades (1)

• Quick simultaneous movements of both eyes to locate interesting parts of the scene

• Necessary to bring the fovea in alignment with the fixated target

• Increase the effective visual resolution of a scene

• One of the fastest movements produced by the human body • Once started cannot be stopped • Used to build a mental map of the scene

• Volitional / Involuntary
Saccades (2)

• Speed of movement cannot be controlled • Peak angular speed 900°/s, plateaus at around 60°amplitude (angular distance travelled)

• 200ms to initiate, last 20-200 ms depending on amplitude • For small amplitudes velocity linearly depends on the amplitude, for higher is an inverse power law What about large gaze shifts?

• During large gaze shifts ( > 20°) the eye 1. first produces a saccade to get gazed point on target 2. the head follows slower and 3. the vestibulo-ocular reflex (VOR) causes the eyes to roll back in the head to keep gaze on the target

• Saccades can be visually guided (reflexive or scanning)

• Anti-saccades to correct errors • Memory guided saccades

• Predictive saccades

Micro-saccades

• Micro-saccades (max 0.2°) when looking on a single spot, necessary to ensure individual photosensitive cells are continually stimulated in different degrees

• Otherwise cells stop generating output

Vestibulo-ocular Reflex

• Reflex eye movement that stabilizes images on the retina during head motion

• Produces an eye movement in the direction opposite to head movement in response to neural input from the vestibular system of the inner ear

• Maintains the image in the center of the visual field

Smooth pursuits

• A way to shift gaze by following a moving object around

• Real-time correction of pursuit velocity to compensate for retinal slip • Less accurate tracking than the vestibulo-ocular reflex which only occurs during head motion

• Require the brain to process incoming visual information and supply feedback (closed loop)

• Most people are unable to initiate pursuit without a moving visual signal

• Unless in total darkness and involving proprioception!

• Speed up to 100°/s in adult humans, however in such high speeds catch up saccades may still be needed

Accommodation

• The eye has two lenses, the cornea and the crystalline lens

• maximum total optical power ~60 diopters in children

• The cornea does most of the focusing on the retina, ~40 diopters

• about the length of the eye!

• The crystalline lens is of variable power, up to ~20 diopters • The changing optical power of the eye lens helps maintain a clear image (focus) of an object as its distance varies • Side-by-side or back-to-back comparison against reference • A conservative method for measuring threshold • Do we want to be this conservative?

• Yes/No • Mean opinion score (MOS)

• Supra-threshold comparison • A lot of source of noise (comparing questions can help)

Discussion 2: Testing Environment Ivan Sutherland's HMD

• optical see-through AR, including:

• displays (2x 1" CRTs)

• rendering

• head tracking

• interaction

• model generation

• computer graphics

• human-computer interaction I. Sutherland "A head-mounted three-dimensional display", Fall Joint Computer Conference 1968

Where we are now

IFIXIT teardown

Case Study 2: Computational Near-eye Displays with Focus Cues

Tutorial Overview

• conventional, fixed-focus near-eye displays

• focus cues & the vergence-accommodation conflict

• advanced optics for VR with focus cues:

• adaptive and gaze-contingent focus displays

• monovision

• volumetric and multi-plane displays

• near-eye light field displays

• Maxwellian-type displays

• AR displays
Magnified Display

1 d + 1 d ' = 1 f d d' f
• big challenge: virtual image appears at fixed focal plane!

• no focus cues "W "W "W "W "W "W "W "W "W "W "W "W "W "W "W "W "W "W "W "W "W "W "W "W "W "Wit it it it it it it it it it it it it "With appropriate programming such a display could literally be the Wonderland into which Alice walked."

it it ith h h h h h h h h h h

d d d d dis is is is is is is is is is is is is is is is ispl pl pl pl
"W "W "W "W "W "W "W "W "W "W "W "W "W "W "W "W "W "W "W "W "W "W "W "W "W

"W "Wit it it it it it it it it it it it it it it it it it ith h h h h h h h h h h h h h h h ap ap ap ap ap ap ap ap ap ap ap ap ap ap ap ap ap ap ap ap ap ap ap appr pr pr pr pr pr pr pr pr pr pr pr prop op op op op op op op op op op op op op opri ri ri ri ri ri ri ri ri ri ri ri ri ri ri riat at at at at at at at at at at at at at at at at ate e e e e e e e e e

e e e e pr pr pr pr pr pr pr pr pr pr pr pr prog og og og og og ogra ra ra ra ra ra ramm mm mm mm mm mm mmin in in in in in in ing g g g g g g g su su su su su su su su su su such ch ch ch ch ch ch ch ch a a a a a a a d

d d d d d d d d d d d d d d dis is is is is is is is is is is is is is is is is is is ispl pl pl pl

nd i i i i i i i i i i i i i i int nt nt nt nt nt nt nt nt nt nt nt nt nt nt nt nt nt nt nt nt nt nt nt nto o o o o o o o o o o o o o o o o o o o o o wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh whic ic ic ic ic ic ic ic ic ic ic ic ic ic ic ic ich h h h h h h h Al Al Al Al Al Al Al Al Al Al Al Al

Conclusion

Modern VR is an exciting platform for creating novel and immersive visual experiences that promise to revolutionize all aspects of our lives, including gaming, education, communication, business, and healthcare. Performance, visual quality and comfort are some of the challenges that limit the practical use of contemporary VR devices.

Research in human visual perception will play a key role in unlocking this protential. The case studies included in this course demonstrate that perceptual approaches have provided solutions to several of these challenges. Foveated rendering, improvement in focus cues, and redirected walking offer significant improvements to the quality and capabilities of VR experience, and are thus likely to be included in a large proportion of future pipelines.

However, despite the success of these efforts, VR graphics still has a long way to go for the ultimate viewing experience: retinal resolution, full field of view, and comfortable viewing without noticeable latency. Thus it is important to continue to search for insights in human visual perception, and how they apply to VR graphics. As more immersive and unique VR devices get developed, they will undoubtedly uncover more challenges, further enhancing the need for more perceptual research in the area. We hope to address both classes of challenges in future iterations of this course.

  what we need to understand to advance VR/AR Goal directed actor To inform VR/AR technical development & advance our science, we must focus on humanenvironment dynamic interaction

•

  Two chambers (anterior and posterior) fused together

•

  Iris only reduces light by a factor of ~20 • Constriction increases Depth-of-Field • Reduces spherical aberration by occluding the outer parts of the lens • Dark/Light Adaptation Peripheral Vision (1) • Fovea vision is the central 1.5-2°of the highest visual acuity • Visual acuity declines by about 50% every 2.5°from the center up to 30°, at which point visual acuity declines even steeper • Peripheral Vision is outside the range of stereoscopic vision • Mid-peripheral outside of a 60°-diameter area around a fixation point • Far-peripheral outside of a 120°-diameter area around a fixation point Peripheral Vision (2)

••

  Takes around 350 ms • Accommodation is both reflexively and consciously controlled • Eye has limited Depth-of-Field Near response -Accommodation Reflex When attending to a near object 1. Eyes converge 2. Pupils constrict (miosis) • Possibly to increase the Depth-of-Field by reducing the aperture of the eye, thus reducing the amount of accommodation needed to bring the image in focus on the retina. • This also reduces spherical aberration (light coming from the edges of the lens) 3. Eyes accommodate • Nearest point of Convergence ~10cm in children • Responses are linked: Vergence-Accommodation, Accommodative-Vergence How does accommodation work? When viewing a far object, 1. the circularly arranged ciliary muscle relaxes 2. allowing the lens zonules and suspensory ligaments to pull on the lens, flattening it The opposite happens when viewing near objects Amplitude of accommodation declines with age (Presbyopia) for Foveated Rendering • 'Sandbox' experiments: Understand visibility of artifacts • Temporal stability • Contrast preservation • Implementation: Realize the lesson learned • Coarse pixel shading • Temporal anti-aliasing • Verification: Confirm it works! • Discussion Items • Selection of psychophysical method • Hardware for testing environment Sensitivity in Periphery Varies with Tasks Strasburger et al. 2011 Tyler 1987Experimental Procedure• Flow• Forced choice (removes the criterion effect)• Subjects had to fixate at the center of the screen (control of variables)• Desktop setup and HMD setup Two alternative forced choice (2AFC)

•VRAccommodation

  Heilig "Sensorama", 1962 (US Patent #3,050,870) • P. Mills, H. Fuchs, S. Pizer "High-Speed Interaction On A Vibrating-Mirror 3D Display", SPIE 0507 1984 • S. Shiwa, K. Omura, F. Kishino "Proposal for a 3-D display with accommodative compensation: 3DDAC", JSID 1996 • S. McQuaide, E. Seibel, J. Kelly, B. Schowengerdt, T. Furness "A retinal scanning display system that produces multiple focal planes with a deformable membrane mirror", Displays 2003 • S. Liu, D. Cheng, H. Hua "An optical see-through head mounted display with addressable focal planes", Proc. non-presbyopes: adaptive focus is like real world, but needs eye tracking! et al., SIGCHI 2016; Johnson et al., Optics Express 2016; Padmanaban et al., PNAS 2017 Monovision VR Konrad et al., SIGCHI 2016; Johnson et al., Optics Express 2016; Padmanaban et al., PNAS 2017 • monovision did not drive accommodation more than conventional • visually comfortable for most; particularly uncomfortable for some users VR Displays with Focus Cues 3. Multiplane Displays Multiplane VR Displays • Rolland J, Krueger M, Goon A (2000) Multifocal planes head-mounted displays. Applied Optics 39 • Akeley K, Watt S, Girshick A, Banks M (2004) A stereo display prototype with multiple focal distances. ACM Trans. Graph. (SIGGRAPH) • Waldkirch M, Lukowicz P, Tröster G (2004) Multiple imaging technique for extending depth of focus in retinal displays. Optics Express • Schowengerdt B, Seibel E (2006) True 3-d scanned voxel displays using single or multiple light sources. JSID • Liu S, Cheng D, Hua H (2008) An optical see-through head mounted display with addressable focal planes in Proc. ISMAR • Love GD et al. (2009) High-speed switchable lens enables the development of a volumetric stereoscopic display. Optics Express • … many more ... idea introduced Rolland et al. 2000 benchtop prototype Akeley 2004 near-eye display prototype Liu 2008, Love 2009 Multiplane VR Displays • Rolland J, Krueger M, Goon A (2000) Multifocal planes head-mounted displays. Applied Optics 39 • Akeley K, Watt S, Girshick A, Banks M (2004) A stereo display prototype with multiple focal distances. ACM Trans. Graph. (SIGGRAPH) • Waldkirch M, Lukowicz P, Tröster G (2004) Multiple imaging technique for extending depth of focus in retinal displays. Optics Express • Schowengerdt B, Seibel E (2006) True 3-d scanned voxel displays using single or multiple light sources. JSID • Liu S, Cheng D, Hua H (2008) An optical see-through head mounted display with addressable focal planes in Proc. ISMAR • Love GD et al. (2009) switchable lens enables the development of a volumetric stereoscopic display. Optics Express • … many more ... idea introduced Rolland et al. 2000 benchtop prototype Akeley 2004 near-eye display prototype Liu 2008, Love 2009 • Rolland J, Krueger M, Goon A (2000) Multifocal planes head-mounted displays. Applied Optics 39 • Akeley K, Watt S, A, Banks M (2004) A stereo display prototype with multiple focal distances. ACM Trans. Graph. (SIGGRAPH) • Waldkirch G (2004) Multiple imaging technique for extending depth of focus in retinal displays. Optics Express • Schowengerdt • Liu S, Cheng D, • Love GD et al. (2009) High-speed switchable lens enables the development of a volumetric stereoscopic display. Optics Express • … many more ... near-eye display prototype Liu 2008, Love 2009 Huang et al., SIGGRAPH 2015 Near-eye Light Field Displays Idea: project multiple different perspectives into different parts of the pupil! Near-eye Displays • eyebox of display is a pinhole very large depth of field (no retinal blur cue) • exit pupil size of 0.5 mm accommodation in open loop • pinholes are dim and reduce eyebox severely! (not practical) Maxwellian-type Near-eye Displays Accommodation-invariant Near-eye Displays Konrad et al., SIGGRAPH 2017 display would, of course, be a would, of course, be a room within which the room within which the computer can control the computer can control the existence of matter." Ivan E. Sutherland "A chair displayed in such a room would be good enough to sit in... ... handcuffs displayed in such a room would be confining,... ff ffs s di di disp sp sp sp spla la la laye ye yed d in in in s suc uc uc uc uch h h h a a ro ro ro room om om om w w w wou ou ould ld b be e e co co conf nfining,... ... and a bullet displayed in such a room would be fatal." . and a bullet displayed in such a room would be fatal." "With appropriate programming such a display could literally be the Wonderland into which Alice walked."

  h h ap ap ap ap ap ap ap ap ap ap ap ap ap ap ap ap ap ap ap ap appr pr pr pr pr pr pr pr pr pr pr pr prop op op op op op op op op op op op op op op op opri ri ri ri ri ri ri ri ri ri ri ri ri ri riat at at at at at at at at at at at at at at ate e e e e e e e e pr pr pr pr pr pr pr pr pr pr prog og og og og og og ogra ra ra ra ra ra ra ra ramm mm mm mm mm mm mmin in in in in ing g g g g g g g g su su su su su su su su su su su such ch ch ch ch ch ch ch ch ch a a a a a a a ad d d d d d d d d

  pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl pl play ay ay ay ay ay ay ay ay ay ay ay ay ay ay ay ay ay ay ay ay ay ay ay ay c c c c c c cou ou ou ou ou ou ou ou ou ou ou ould ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld l l l l l l l l l l l l l l lit it it it it it it it it it iter er er er er er er er er er er er er er er er eral al al al al al al al al al al al al al ally ly ly ly ly ly ly ly ly ly ly ly ly ly ly ly ly ly ly ly ly b b b b b b b b b b b b b be e e e e e e e e e e th th th th th th th the e e e e e e e e Wo Wo Wo Wo Wo Wo Wo Wo Wo Wo Wo Wo Wond nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nder er er er er er er er er er erla la la la la la la la la la la la la land nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd i i i i i i i i i i i i int nt nt nt nt nt nt nt nt nt nt nt nt nt nt nto o o o o o o o o o o o o o o wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh wh whic ic ic ic ic ic ic ic ic ic ic ic ic ic ic ic ic ic ich h h h h h Al Al Al Al Al Al Al Al Al Al Al Al Alic ic ic ic ic ic ic ic ic ice e e e e e e e wa wa wa wa wa wa wa wa wa wa wa wa wa wa wa wa wa wa wa wa wa walk lk lk lk lk lk lk lk lk lk lk lked ed ed ed ed ed ed ed ed ed ed ed ed." ." ." ." ." ." ." ." Case Study 3: Perception & Cognition during Redirected Walking L. & A. Wachowski: The Matrix, 1999 R. W. Fassbinder: World on a Wire, 1973 I.E. Sutherland: Head-mounted 3D display, 1968

  pl pl pl pl pl pl pl pl pl pl pl pl pl pl play ay ay ay ay ay ay ay ay ay ay ay ay ay ay ay ay ay ay ay ay ay ay ay c c c c c c cou ou ou ou ou ou ou ou ou ou ou ou ou ou ould ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld ld l l l l l l l l l l l lit it it it it it it it it it it iter er er er er er er er er er er er er er er er eral al al al al al al al al al al al al al al al ally ly ly ly ly ly ly ly ly ly ly ly ly ly ly ly ly ly ly ly ly b b b b b b b b b b be e e e e e e e e th th th th th th th th th th th the e e e e e e Wo Wo Wo Wo Wo Wo Wo Wo Wo Wo Wo Wond nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nder er er er er er er er er er er er er er er er erla la la la la la la la la la land nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd nd

  Alic ic ic ic ic ic ic ic ic ic ic ice e e e e e e e e e e e e e e wa wa wa wa wa wa wa walk lk lk lk lk lk lk lk lk lklked ed ed ed ed ed ed ed ed. . ed ed ed ed " " " AixCAVE, RWTH Aachen, 2012 Oculus VR, Oculus Rift, 2013 Case Study 3: Perception & Cognition during Redirected Walking

• gaze-contingent focus gives natural focus cues for non-presbyopes, but require eyes tracking

• presbyopes require fixed focal plane with correction

• multiplane displays require very high speed microdisplays

• monovision has not demonstrated significant improvements

• Maxwellian-type displays can be interesting, but provide small eyebox

• light field displays may be the "ultimate" display need to solve "diffraction problem"

(Some) Technology Challenges

• Vergence-accommodation conflict (VAC)

• Vestibular-visual conflict (motion sickness)

• AR