Foveated Rendering in Virtual Reality

Samuel A. MillerDivision of Science and Mathematics

University of Minnesota, MorrisMorris, Minnesota, USA 56267mill5978@morris.umn.edu

ABSTRACTFoveated rendering is a technique in which images are dis-played at differing levels of detail dependent on where theuser is looking. The question this paper addresses is howbest to implement foveated rendering to reduce computa-tional load when rendering three-dimensional environmentsin virtual reality. The main goal for these renderers is toprovide an experience in which the user is unable to distin-guish whether they are seeing environments that are usingthese optimizations or not. Ideally you should be able toimplement this technology to render the same scene withthe same impact on the user, while only needing a fractionof the computing power. At the current level of develop-ment, renderers have been designed that reduce the numberof pixels shaded by up to 70 percent, without the user de-tecting the change. In this paper, we describe two studiesthat use these renderers to succesfully reduce the amount ofcomputation done with negligible change in perception bytheir users.

Keywordsfoveated rendering, gaze tracking, multiresolutional displays,virtual reality

1. INTRODUCTIONThere is an abundance of modern computer programs and

media that require the generation of complex visual scenes.From entertainment like video games, to more practical ap-plications such as any computer program that uses 3D mod-eling, effectively rendering three dimensional scenes has be-come necessary for modern computers. Due to this main-stream need for rendering power, any technique that canmake the process less costly has become more important asmedia increases in complexity. Foveated rendering is onesuch method.

The phrase foveated rendering refers to a process that ex-ploits the way the human eye works in order to render onlywhat is necessary in a three-dimensional scene. Normally,the entire display is rendered at the same resolution, regard-less of where the user is looking. Because the human eyeonly perceives great detail at the center of vision [10], thisuniformity in resolution regardless of user focus is a waste of

This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. To view a copy ofthis license, visit http://creativecommons.org/licenses/by-nc-sa/4.0/.UMM CSci Senior Seminar Conference, December 2017 Morris, MN.

valuable resources such as computing power. Foveated ren-dering uses a practice known as gaze tracking to determinethe center of the user’s focus, then chooses regions aroundthat centerpoint to render at progressively lower quality. Be-cause of the aforementioned inability of the human eye todetect detail toward the outer boundaries of their field ofview, these regions of lower quality in the scene are per-ceived the same way they would be if there was no foveatedrendering involved.

This process has been discussed for decades [4], however itwas limited by the primitive nature of essential componentssuch as gaze tracking. As it currently stands, the hardwarerequired has caught up to make this process possible, justas higher pixel densities and refresh rates are becoming in-creasingly more in demand. With demanding technologiessuch as virtual reality becoming more mainstream, foveatedrendering will be an essential tool for developers.

In this paper, we start with some background necessaryto understanding foveated rendering, followed by a look atthe efficacy of existing approaches. Section 2 begins with anexplanation of the properties of the eye that are exploitedby this process. It then introduces the hardware required forapplying these renderers to virtual reality, as well as somecomponents involved in renderer design. Section 3 delvesinto more specific facets of designing a foveated rendererby examining two implementations. Section 4 takes a lookat user studies of these implementations, with a summaryof their results. Section 5 then provides some concludingremarks.

2. BACKGROUNDIn order to understand how foveated renderers are de-

veloped, some background concepts must be explained. Wefirst look at the properties of human vision that are involved,as well as how to track the motion of the eye in an existingvirtual reality setup. We then examine some components ofthe rendering process that play a key role in how effectivefoveated rendering can be.

2.1 Human Vision OverviewDetermining the strengths and weaknesses of the human

eye is essential when developing technology that is primarilyperceived visually. Foveated rendering focuses on the differ-ences between how humans perceive the fovea and peripheryregions. The fovea is an area in the central region of visionapproximately 5◦ in diameter that contains the highest spa-tial density of photoreceptors in the eye, while the peripheryis the surrounding area [3]. This is depicted in Figure 1 with

Figure 1: Representational map of the human retinaand visual cortex. [5]

the fovea represented by a circle in the center of the diagramon the left with all other regions representing the periphery.

A key difference in how these two regions are perceivedis that the optic nerve undersamples the retinal image fromthe periphery, while all data gathered in the foveal region isnot undersampled. This means that the brain is fed more in-formation by the photoreceptors at this center point, whichleads the subject to build a mental image that has far moredetail near the object in focus.

The difference in detail perception between the fovea andperiphery is due to a concept known as Cortical Magnifica-tion Factor (CMF), where the section of the brain that isdedicated to visual processing (the visual cortex) is dividedinto subregions that each correspond to varying proportionsof fovea and periphery. The further from the center point ofvision you go, there is a progressively smaller portion of thevisual cortex dedicated to processing information from thatsection [9]. This is shown in the diagram on the right in Fig-ure 1, with almost half of the cortex dedicated to the fovea(the region in light blue). It is this facet of the human brainthat leads to the visual acuity falloff that foveated renderingis based on.

Motion is different than detail in that it is perceived roughlythe same throughout the field of view. Because of this, un-intended motion anywhere in the scene can distract the userfrom the experience and break their immersion. If distortioncauses flickering motion towards the edge of a user’s field ofview, they will be drawn to this new focal point and divergefrom the experience designed by the developers. Because ofthe potential for immersion to be compromised, eliminatingunintended motion in both fovea and periphery is a prioritywhen designing renderers. [3, 10]

Additional important concepts for this subject matter arethat of retinal eccentricity and foveal layers. Retinal eccen-tricity is the angular distance as you move away from thecentral focus of your vision [3]. This measure is representedin Figure 1; the units of the diagram on the left are degreesof retinal eccentricity. Foveal layers (also referred to as ec-centricity layers) are multiple renders of a scene at differentresolutions and sizes, that are then overlapped to create acomposite image of varying quality. Figure 2 is a depictionof three foveal layers that are separated to show the size dif-ferences and what the overlap looks like. These layers arebased on retinal eccentricity, as it is used to dictate how largeeach region should be according to photoreceptor density onthe retina. The size of these layers varies depending on im-plementation, however the typical approach is to quantifyCMF in some way and use that as a heuristic to determine

Figure 2: Visualization of nested foveal layers. [3]

which how large each subdivision should be based on theamount of visual cortex that is dedicated to processing thatsection.[9]

Saccadic eye movements (see Figure 3), are the simulta-neous shift in focus of both eyes to a point that is a consid-erable distance from where they started. These movementsare a critical challenge that foveated renderers must handle,as these movements are what make tracking the eye unpre-dictable. It is difficult to smoothly render a space based onthe focal point of the eyeline if the user can shift his gazesomewhere else in an instant. Handling this weakness is onefactor that held the advancement of this field back for solong, as latency proved to be an insurmountable restriction.Because the motion of the eyes is unpredictable, the usercould make a saccadic movement and the system would ex-hibit a delay as the scene was re-rendered to suit the newfocal point. This is demonstrated in Figure 3 by the con-centric yellow circles representing the foveal layer positionsof the current foveated render. In this instance, the systemhas not caught up to the new focal point of the user yet(as shown by the discrepancy between the render focus andvisual focus in the second frame pictured).

Figure 3: Visualization of a saccadic eye movementfrom one foveated rendered frame to another. [1]

This delay, or latency, has become less severe as hardwarehas advanced and computing speed has increased. Once animmersion breaking hurdle, latency has now become negli-gible enough that developers are able to implement foveatedsystems that are difficult to distinguish from normal ones(as discussed in the studies in later sections).

2.2 Virtual Reality HeadsetsVirtual Reality (VR) displays have been a science fiction

staple that developers have long tried to implement. Therehave been countless failures to make this technology feasi-ble over the years, however it seems computing hardware hasfinally caught up. There has been a steadily increasing avail-ability of affordable consumer options for VR, leading to acomplementary increase in both public and corporate inter-est. This interest has yielded options for early adopters, suchas the Oculus Rift and the HTC Vive, as well as the morecasual consumer, such as Samsung’s Gear VR and GoogleCarboard.

Figure 4: HTC Vive Virtual Reality Head MountedDisplay. [2]

While all of these options have their niche in the mar-ket, in this paper we focus on the Head Mounted Displays(HMDs) with the most computing power available to them,e.g. the HTC Vive (see Figure 4). The Vive targets a mar-ket that wants applications with complex scenes as well ashigh resolutions. It is devices like these that have an im-mediate demand for any avenue of optimization in order tomeet the needs of their consumer base,and will likely adopttechniques like foveated rendering first.

2.3 Gaze TrackingGaze tracking is the practice of using a combination of

hardware and software to compute the trajectory of theuser’s eyeline to determine what the center of their viewis focused on. This technology has been used for a variety ofpurposes over the years (e.g. accessibility services for thosewho are handicapped), and has now become increasingly inthe public eye due to the commercial popularization of VRdevices.

While gaze tracking hardware has been around for a supris-ingly long time, it hasn’t always been feasible for practicaluse. In 1990, Mark Levoy and Ross Whitaker [4] exploredways of utilizing gaze tracking to develop rendering opti-mization algorithms. Their results showed promise, withtheir software responding to the gaze tracking well, how-ever the subjects were placed in bulky headwear that limitedtheir practicality. At the time, the setup demanded an NACEye Mark eye Tracker (pictured in Figure 5), which cov-ered a large portion of their head and had to be physicallyconnected to rendering engine hardware. This hardware re-striction limited what they could accomplish, as nearly theentire head was covered with technology purely devoted togaze tracking. Integration with virtual reality would havebeen impossible at this time.

Figure 5: Pictured left: NAC Eye Mark eye trackerfrom the 1980s. Pictured right: aGlass eye trackerfrom 2017. [4] [8]

However, gaze tracking has advanced drastically over theyears. 2017 is a landmark year in that there are several im-plementations being released for public purchase. “aGlass”,developed by 7invensun from Beijing, is an affordable mod-ular upgrade to the HTC Vive that will be available forpurchase later in 2017. This product is notable in that itwill be able to fit directly over the lenses of the Vive dueto its compact size. Other emerging gaze tracking products,such as the Tobii Pro VR implementation for the Vive head-set, are so compact that they come embedded in existingVR HMDs. The differences between these products and thepreviously mentioned NAC Eye Mark eye Tracker are signif-icant enough for gaze tracking to be properly capitalized onas a technology. Developers no longer have to consider thehardware restrictions for the use of gaze tracking in foveatedrendering, as soon the majority of the consumer base will al-ready have gaze tracking functionality integrated into theirdevices.

2.4 3D Rendering ConceptsSome critical components involved in rendering a three

dimensional scene are that of aliasing and artifacting.

Figure 6: Depiction of a continuous analog pictureconverted into a discrete digital picture of separatepixels. [6]

Aliasing is a phenomenon that occurs in computer graph-ics due to the fact that the rendered images are discrete rep-resentations of analog constructs. This means that in orderto represent an object, the rendered image is generated bypicking individual sampling points and using them to createa model of the source image. The distances between thesesample points typically depend on resolution/pixel position,

however if the renderer uses a sampling rate that doesn’taccurately represent the complexity of the source, the fre-quency of the signals can become indistinguishable from oneanother and can then be interpreted falsely [6]. This am-biguity of signal interpretation manifests in the completedimage as jagged edges, as seen in the rendered image on theright in Figure 6.

This effect is an example of artifacting, i.e. visual dis-tortion due to complications during image processing. Ar-tifacting and aliasing have proven troubling in the imple-mentation of foveated renderers, as one of the methods usedto lower quality in the periphery is to increase the distancebetween samples (otherwise known as subsampling). Thisprocess puts less computational strain on the system by hav-ing fewer points to shade, however it magnifies the negativeeffects associated with subsampling: spatial and temporalaliasing/artifacting.

Temporal aliasing is the effect of having consecutive framesof a rendered scene contain aliasing in the same general re-gion, leading to consistant visual distortion over time. Areascontaining this consistant distortion are referred to as tem-poral artifacts, and are essentially objects that can attractthe user’s attention and break their immersion by distractingthem from the intended experience of the scene. Because thehuman eye detects motion uniformly throughout the field ofview [7], these artifacts are problems no matter where theyoccur in the scene.

Due to the prevalence of this distortion in the process offoveated rendering, developers must strike a balance betweenlowering the quality in the periphery enough to yield savingson resources used, while avoiding these immersion breakingartifacts.

3. FOVEATED RENDERER DESIGNAs previously discussed, the main goals of foveated ren-

derers are to save on computational resources and to keepthe impact subtle enough that the user cannot detect it.

Perhaps the first notable work that actually claimed sig-nificant performance advantages in graphical rendering fromthis technology was that of Guenter et al. in 2012 [3].They claimed that their renderer could produce results in-distinguishable from full-resolution images, yet with aroundone fifth or sixth of the computational power previously re-quired.

3.1 Desktop Display Foveated Rendering (2012)In 2012 Guenter et al. attempted to validate previous re-

search in the field by using what was cutting edge technologyat the time to bridge the gap between theory and practice.Hardware had finally gotten to the point that they couldtake advantage of the work put forth by the likes of Levoyand Whitaker to make a renderer that capitalized on the op-portunity for low cost optimization via foveated rendering.

3.1.1 Renderer Design GoalsGuenter et al. aimed to obtain considerable speedup in

rendering times while also avoiding any noticeable spatialor temporal artifacts. They conducted numerous user stud-ies in order to verify that users had difficulty distinguishingbetween the foveated and non-foveated renders. The teamalso had to deal with a self-imposed constraint in that theywanted their solution to have the capacity to be retrofitted toimprove existing applications. This meant that their method

of handling aliasing had to be effective while also requiringlimited modification to how the native applications handledpixel shading. They had to strike a fine balance between be-ing specific enough to address the artifacting problem, whilealso being applicable to a range of applications [3].

3.1.2 Techniques UsedGuenter et al. developed their foveated renderer for a

desktop computer with a 1920 x 1080 resolution display,supported by a Tobii TX300 Eye tracker. Their system wastested on a virtual moving camera through a static three-dimesnsional scene.

In this instance, successful optimization was achieved byminimizing system latency, handling aliasing in the periph-ery, and finding the ideal balance of size and resolution fortheir eccentricity layers. They were able to reduce the num-ber of pixels shaded due to their implementation of the fol-lowing:

• temporal jitter of the spatial sampling grid: the shift-ing back and forth of the eccentricity layer by smalldistances between frames to reduce artifacting.

• temporal reverse reprojection: blending of previousand current pixel values to increase the effective sam-pling.

Essentially, these two techniques amount to a smoothingof the jagged, aliased edges described before. The temporaljitter allows samples to cover a larger portion of the sourceimage so that it can be more faithfully represented, Howeveron its own it provides images that are constantly shifting,which triggers unwanted motion that is detectable by theuser. This unintended motion of the image is eliminated bythe second technique: temporal reverse reprojection. Thisprocess is basically just a way to average consecutive frames,so any pixels that were rapidly alternating colors due totemporal jitter would then be a constant intermediate shade.When combined, the resulting image has reduced aliasing,with no unwanted flickering motion.

Guenter et al. also used Multisample Anti-Aliasing (MSAA)to help combat distortion due to aliasing, however for thesake of brevity this will not be covered in this paper, moreinformation on this subject can be found in their work. [3]

Guenter et al. became among the first developers to fullyrealize the potential of foveated rendering; they were able toreduce the number of pixels shaded by a factor of 10-15 whilestill presenting scenes of acceptible quality as determined bytheir user study [3]. Their system is even applicable to mod-ern renderer setups, which is discussed later in the paper.However, it is worth noting that they were still limited bythe technological constraints of their time:

Ideally, the graphics card would render at fulldisplay resolution where the gaze is centered andcontinuously decrease resolution outward fromthere. A more efficient method on current graph-ics hardware is to approximate this ideal by ren-dering several overlapped rectangular regions calledeccentricity layers. [3]

This hardware-imposed weakness to their approach is oneof the areas improved upon by subsequent efforts by otherdevelopers building on their work.

3.2 Virtual Reality Foveated Rendering (2016)Four years after the publication of Guenter et al. [3], Pat-

ney et al. [7] brought the technology of foveated renderingto the burgeoning field of virtual reality. Guenter’s teamproposed that the true benefits of this rendering technologywould be most apparent when the display had a larger fieldof view (FOV), and virtual reality head-mounted displays(HMDs) boast the largest FOV in any display to date: true360◦. Virtual reality is a perfect match for foveated render-ing as many headsets already have gaze-tracking built in,so implementing these algorithms yields significant gain forvery low cost.

Patney et al. set out to bring the speedup of foveated ren-dering to a field where the computational demand is rapidlyrising. Any potential for increased computational powerwould have significant impact on the capabilities of VR.

3.2.1 Renderer Design GoalsPatney et al. [7] specifically call out the aforementioned

work from 2012 of Guenter et al. [3]:

Prior foveated renderers like Guenter et al. . . .focus on practical near-term techniques to reducecosts without explicitly identifying and minimiz-ing perceptible artifacts introduced by foveation.Our studies show they often exhibit significanthead- and gaze-dependent temporal aliasing, dis-tracting users and breaking immersion. [7]

Having specified weaknesses in previous approaches, Pat-ney et al. worked to find new methods that would minimizetemporal aliasing and any sensation of tunnel vision: onlyperceiving visual stimuli from the central field of view dueto reduction in peripheral detail. In order to identify whichfoveation technique best achieved these goals, Patney et al.developed a prototype to test different approaches. Thisprototype was referred to as a Perceptual Visual Target.

3.2.2 Perceptual Visual TargetPatney et al. differ in their design process from some of

their predecessors in their use of a Perceptual Visual Tar-get [7]. Essentially, their design team created a demonstra-tional model that emulated different versions of foveatedrendering using pre-processed scenes. They then conducteduser studies to confirm which of their intended optimizationswould be effective. Once they had a user-approved rendereremulation, they began the design process with the goal ofbeing able to render what was shown in the perceptual visualtarget, albeit in real time.

This process of guiding their development lead to impres-sive overall user satisfaction, with their renderer yieldingsuperior results in a number of facets.

3.2.3 Techniques UsedPatney et al. built directly on the existing work of others

by adding parameter tweaks and supplementary algorithms.For example, rather than render three nested foveal layerslike the approach of the Guenter et al. (pictured in Figure 2),they utilize a piecewise linear variation of shading rate in asingle layer.

In the graph shown in Figure 5, the foveal layers of Guenteret al. are represented by the solid orange lines at differingsampling factors. The dashed orange lines represent thetransitional blend applied to turn multiple foveal layers into

Figure 7: Top: visualization of nested foveal lay-ers. [3]. Bottom: comparison of sampling ap-proaches by Patney et al. and Guenter et al. [7]

one smooth composite image. In contrast, the dashed blueline represents the single layer of Patney et al. with onetransition from high to low detail starting right before 20◦ ofretinal eccentricity.

Rather than having to transition between three layers withdifferent sample rates, Patney et al. apply one filter thatshifts pixel shading rate from their bounds of maximum de-tail (base sampling factor) to minimum detail (four timesthe distance between samples).

This alternate approach enables them to transition to alower level of detail at smaller retinal eccentricities, whichleads to lower computational demand due to fewer pixelsbeing shaded. (See Figure 8).

Additionally, they differ from other approaches in thatthey are sampling at full resolution throughout the entireimage rather than sampling different layers with multipleresolutions such as the approach of Guenter et al. This de-cision ended up requiring them to use a new algorithm, vari-ance sampling, to improve visual fidelity. Variance samplingis a post-process image reconstruction algorithm developedby Patney et al. to reduce/eliminate temporal aliasing intheir end product. For the sake of brevity, this technique isbeyond the scope of this paper. More information can befound in the work of Patney et al. [7]

4. RESULTSWhile Guenter et al. conducted a user study to gauge the

efficacy of their product, they used hardware that is nowoutdated which makes it difficult to directly compare theirresults to more modern offerings such as Patney et al.

Fortunately, Patney et al. conducted a user study directlycomparing their foveated renderer to that of Guenter et al.’srenderer, using the same hardware setup. This study effec-tively highlights the differences between the two approachesand demonstrates how far the field has progressed in sucha short time frame. In this section we discuss the setup ofthe experiment and analyze the reasons for the difference inresults.

4.1 User Study SetupPatney et al. conducted a test of four users. They pre-

sented the user with two versions of a scene (one foveated

Figure 8: Comparison of the Transition Region Sizeused in Patney et al.’s user study [7]. The pointshere represent the degree of retinal eccentricity atwhich each of the four users could detect the pres-ence of foveation. The error bars represent outliersin the trials, in which the user’s ability to detectfoveation was irregular when compared to the restof their results. These outliers could have been dueto identifying foveation by chance instead of whenthey noticed it, or if the user had cheated and lookedaway from the intended gaze point.

and one non-foveated) and asked the subject which scenelooked better. The scenes were presented in a randomizedorder and were varied using different transition region sizes.This variation was accomplished by altering the size of theregion between the foveal and the peripheral regions closerto or farther from the center. In Figure 8 this change wouldpresent as the shift towards larger sampling factors occur-ing at smaller or larger degrees of retinal eccentricity; thedepartures from base sampling factor would occur sooner orlater depending on the intention of the trial. This was doneto see which system could best present images with lowerdetail before the subject noticed the foveation.

The study applied both systems to a head-mounted dis-play setup in order to confirm their hypotheses about theefficacy of foveated rendering on VR, and to keep results con-sistant. All four users were subjected to two hundred trialseach which followed a one-up two-down staircase-method ofvariation, in that they would increase the transition regionsize for one trial, then decrease it for the next two to reducetraining their subjects to expect constant increase. Reversestaircases of the same pattern were also used.

4.1.1 User ResultsAs seen in Figure 8, all of the users tolerated lower detail

closer to the fovea in the scenes using Patney et al.’s setup.This validates their use of a perceptual visual target as wellas their efforts towards minimizing temporal artifacts whenlowering quality in the periphery. Not only did their systemappear nearly indistinguishable from the non-foveated ver-sion when set to the right transition region, they also wereable to shade 70 percent fewer pixels than the non-foveatedscene.

Their study confirmed that they were able to use lowerquality shading up to 30◦ closer to the fovea than Guenter etal.’s multi-resolution approach, implying that their piecewiselinear shading rate is more effective than the three foveallayers of varying resolution used by Guenter et al. [7]

5. CONCLUSIONIn this paper we discussed the efficacy of foveated ren-

dering and gave a sense of how feasible integration withexisting platforms could be. As the hardware required forsuch a system becomes more commonplace, this process willreceive increasing amounts of attention and development.The most recent research implies that a temporally-stablefoveated renderer implementation such as that developed byPatney et al. [7] is a proven method for reducing computa-tional workload while still preserving the user’s perceptionof normality when rendering scenes.

6. ACKNOWLEDGEMENTSI would like to thank Nic McPhee, Kristin Lamberty, and

Kristin Shirley for their excellent feedback and advice duringthe writing of this paper.

7. REFERENCES[1] R. Albert, A. Patney, D. Luebke, and J. Kim. Latency

requirements for foveated rendering in virtual reality.ACM Trans. Appl. Percept., 14(4):25:1–25:13, Sept.2017.

[2] D. Forrest. Tobiis integrated VR HMD eye tracking:Hands-on, 2017. [Online; accessed 8-October-2017].

[3] B. Guenter, M. Finch, S. Drucker, D. Tan, andJ. Snyder. Foveated 3D graphics. ACM Trans. Graph.,31(6):164:1–164:10, Nov. 2012.

[4] M. Levoy and R. Whitaker. Gaze-directed volumerendering. SIGGRAPH Comput. Graph.,24(2):217–223, Feb. 1990.

[5] E. Morson. Your brain is made of maps, 2017. [Online;accessed 15-October-2017].

[6] G. S. Owen. Overview of aliasing in computergraphics, 1999. [Online; accessed 13-October-2017].

[7] A. Patney, M. Salvi, J. Kim, A. Kaplanyan,C. Wyman, N. Benty, D. Luebke, and A. Lefohn.Towards foveated rendering for gaze-tracked virtualreality. ACM Trans. Graph., 35(6):179:1–179:12, Nov.2016.

[8] A. Robertson. The HTC Vive will track eye movementwith a 220 upgrade kit, 2017. [Online; accessed10-October-2017].

[9] N. T. Swafford, J. A. Iglesias-Guitian, C. Koniaris,B. Moon, D. Cosker, and K. Mitchell. User, metric,and computational evaluation of foveated renderingmethods. In Proceedings of the ACM Symposium onApplied Perception, SAP ’16, pages 7–14, New York,NY, USA, 2016. ACM.

[10] L. N. Thibos. Image processing by the human eye. InVisual Communications and Image Processing IV,volume 1199, 1989.