AMD FidelityFX Super Resolution

I. Introduction

Overview

AMD FidelityFX Super Resolution (FSR) encompasses a suite of open-source image upscaling technologies engineered to enhance application framerates. This is achieved by rendering scenes at a lower internal resolution and subsequently employing intelligent algorithms to upscale the image to the desired, typically native, output resolution. The primary benefit is a significant boost in performance, which can be utilized to enable more demanding graphical settings, achieve higher resolutions, or ensure smoother gameplay, particularly on hardware with limited computational resources.

A defining characteristic of FSR is its open-source nature, distributed under the permissive MIT license, and its broad cross-platform compatibility. FSR supports a wide array of graphics processing units (GPUs), including those from competitors, and integrates with multiple graphics APIs such as DirectX 11, DirectX 12, and Vulkan. This open approach fosters wider adoption within the game development community, as a single FSR implementation can benefit a diverse user base across various hardware configurations. The strategic decision to make FSR open and cross-platform, contrasting with some proprietary hardware-locked solutions, allows developers to reach a broader audience without fragmenting their engineering efforts. This approach aims to establish FSR as a widely available upscaling standard, indirectly enhancing the value of AMD’s ecosystem by promoting a technology accessible to all.

The Evolution of FSR

The FSR technology suite has undergone significant evolution since its inception, with each iteration addressing limitations of its predecessor and incorporating more advanced rendering techniques.

• FSR 1.0 (Released 2021)¹: This initial version introduced spatial upscaling. Its core components were an Edge-Adaptive Spatial Upsampling (EASU) algorithm and a Robust Contrast-Adaptive Sharpening (RCAS) pass. FSR 1.0 was designed for ease of integration and targeted broad hardware compatibility, operating on a single input frame without reliance on temporal data.
• FSR 2.x (Released 2022)¹: Representing a fundamental shift, FSR 2.x transitioned to temporal upscaling. This version was developed from the ground up and is not an incremental update to FSR 1.0. It leverages data from previous frames, such as motion vectors and depth information, to reconstruct images with significantly higher quality and more effective anti-aliasing compared to FSR 1.0.1 Subsequent minor versions (e.g., FSR 2.1, 2.2) introduced refinements to improve image quality further, such as reducing ghosting artifacts and enhancing the stability of motion vector processing.
• FSR 3.x (Released 2023², 2024³): This iteration builds upon the temporal upscaling foundation of FSR 2.x and introduces a significant new feature: frame generation. Technologies like Optical Flow and Frame Interpolation are employed to synthesize new frames, aiming to further increase the displayed frames per second (FPS) beyond what upscaling alone can achieve. FSR 3 also introduced a “Native AA” (Anti-Aliasing) mode, which applies the FSR processing pipeline at native resolution for improved anti-aliasing without upscaling.

This evolutionary trajectory highlights a pattern of iterative problem-solving. FSR 1.0, while effective for boosting performance, exhibited limitations in image fidelity, particularly concerning the reconstruction of fine details and maintaining temporal stability, sometimes resulting in blurriness or shimmering. FSR 2.x directly targeted these shortcomings by incorporating temporal data, a well-established method for enhancing reconstruction quality by accumulating more information per output pixel over successive frames. This advancement, however, necessitated more complex input data from the game engine and a more intricate integration process. FSR 3.x addresses a different performance aspect: scenarios where even temporally upscaled rendering may not achieve the very high frame rates desired for modern high-refresh-rate displays. By introducing frame generation, FSR 3.x decouples the displayed FPS from the rendered FPS. This, however, introduces new engineering challenges, primarily related to increased input latency and the potential for artifacts in the synthetically generated frames. Each generation thus reflects a drive to overcome prior limitations by employing more sophisticated graphics techniques, leading to a richer feature set but also potentially increasing integration complexity and imposing stricter demands on the quality of input data.

II. Fundamental Computer Graphics Concepts

Rendering Resolution vs. Display Resolution

In the context of real-time 3D graphics, it is crucial to distinguish between rendering resolution and display resolution:

• Rendering Resolution: This refers to the internal resolution at which the 3D scene geometry is rasterized, and fragments are shaded by the GPU.8 A lower rendering resolution means fewer pixels are processed for complex shading calculations, lighting, and other effects. This reduction in pixel load directly translates to decreased computational work for the GPU, typically resulting in higher frame rates.
• Display Resolution (Output Resolution): This is the resolution at which the final image is presented to the user, often corresponding to the native pixel grid of the display device (e.g., 1920x1080, 2560x1440, 3840x2160).

Upscaling technologies like FSR serve as a bridge between these two resolutions. They take an input frame rendered at a lower resolution and algorithmically reconstruct it to match a higher target display resolution.1 A common characteristic of this process is that while the 3D rendered scene might be processed at a lower fidelity, user interface (UI) elements can be rendered directly at the display resolution and composited over the upscaled scene, ensuring their sharpness and legibility.

Aliasing and Anti-Aliasing (AA)

Aliasing is a pervasive artifact in computer graphics, manifesting as jagged, “stair-step” patterns on the edges of rendered objects, particularly noticeable on diagonal or curved lines. It arises from the process of representing continuous geometric shapes on a discrete grid of pixels, effectively an undersampling of the original signal.

Anti-Aliasing (AA) techniques are employed to mitigate these aliasing artifacts. The general goal is to create smoother, more visually appealing edges by, for example, sampling the scene at a higher frequency than the final display resolution (e.g., Supersampling Anti-Aliasing - SSAA) or by employing analytical methods to blend pixel colors along detected edges (e.g., Morphological Anti-Aliasing - MLAA).

Temporal Anti-Aliasing (TAA) is a widely adopted AA technique that leverages information from previously rendered frames to improve edge smoothing and reduce temporal artifacts like shimmering or crawling of edges on moving objects. TAA achieves this by strategically jittering the sample positions within pixels across successive frames and accumulating these samples over time. This temporal accumulation effectively increases the sampling density per pixel, leading to better anti-aliasing quality, often with a more favorable performance-to-quality ratio than purely spatial methods like SSAA. FSR 2.x and subsequent versions are designed to replace a game engine’s existing TAA solution, as their temporal upscaling process inherently includes a high-quality anti-aliasing component.

Motion Vectors, Depth Buffers, and Other Temporal Data

• Motion Vectors: These are 2D vectors, typically stored in a dedicated buffer, that describe the screen-space displacement of pixels or surfaces from their position in the previous frame to their position in the current frame. They are indispensable for temporal upscalers to correctly reproject data from the history buffer onto the current frame’s geometry. Motion vectors are also fundamental for frame generation techniques, as they inform how pixels are likely to move into the synthesized intermediate frame. The accuracy, precision (e.g., 16-bit float format is often recommended over 8-bit), and coverage (ensuring all dynamic scene elements, including particles and animated textures, contribute to the motion vector buffer) are paramount for high-quality temporal processing. Engine-generated motion vectors often require careful handling of camera jitter to ensure static objects truly have zero motion.
• Depth Buffer: This buffer stores per-pixel depth information, usually representing the distance of the visible surface from the camera. The depth buffer is critical for several aspects of temporal upscaling:
  • Reprojection: Assisting in accurately mapping historical samples to their correct 3D locations in the current frame.
  • Disocclusion Detection: Identifying regions of the scene that have become newly visible (e.g., when an object moves away, revealing what was behind it) or newly occluded. This allows the upscaler to invalidate or appropriately blend historical data that is no longer relevant, thereby mitigating ghosting artifacts.
  • Edge and Feature Analysis: Depth discontinuities can aid in edge detection and help differentiate between distinct geometric surfaces. For optimal precision, especially with distant objects, rendering pipelines often use reverse depth (where the far plane is at z=0 and the near plane at z=1) and floating-point depth formats (e.g., R32_FLOAT).
• Exposure and Color Space: The color data fed into FSR, and how it’s processed internally, is sensitive to color space and exposure levels. FSR 1.0, for example, is documented to work best in a perceptual color space and should be integrated after tone mapping. FSR 2.x and FSR 3.x, when handling High Dynamic Range (HDR) content, typically require input color to be in linear RGB space. The engine’s exposure value might be required by FSR to correctly normalize or tonemap HDR data internally, or to consistently combine frames that may have been rendered with varying pre-exposure factors. If the engine applies a dynamic pre-exposure factor that changes frame-to-frame, this value must be communicated to FSR to ensure temporal consistency.
• Reactivity, Transparency, and Composition Masks: These are optional but often crucial input textures that provide per-pixel guidance to the temporal upscaler on how to treat specific areas of the image.
  • A Reactivity Mask (often derived from alpha values of transparent objects) informs the upscaler to increase the influence of current frame samples over historical samples for pixels corresponding to highly reactive elements like particles, smoke, or other alpha-blended effects. This helps prevent stale historical data from “smearing” or “ghosting” through such effects.
  • A Transparency and Composition Mask can be used to denote areas with other types of special rendering, such as raytraced reflections or animated textures, where standard history accumulation and protection mechanisms might need adjustment. These masks allow developers to fine-tune the upscaler’s behavior, improving quality in challenging rendering scenarios where default heuristics might fall short.

III. Deep Dive: AMD FidelityFX Super Resolution 3.1

Overview: Combining Temporal Upscaling and Frame Generation

FSR 3.1 is architecturally designed as a two-stage process to maximize framerates and visual quality.

1. Temporal Upscaling: The first stage employs a sophisticated temporal upscaling algorithm, which builds directly upon the principles and advancements established in FSR 2.x. This component takes a lower-resolution, jittered input frame along with auxiliary data (depth, motion vectors, etc.) and reconstructs a high-resolution, anti-aliased output frame by leveraging historical information from previous frames.
2. Frame Generation: The second stage, a key differentiator of FSR 3.x, utilizes the upscaled frames produced by the first stage to generate and insert additional frames. This process aims to significantly increase the final displayed frame rate, enhancing perceived motion smoothness.

It is important to note that the frame generation component can operate in conjunction with the upscaling component or, through the “Native AA” mode, it can be used with frames rendered at native resolution if only frame rate multiplication is desired without upscaling.

Core Upscaling Algorithm

The upscaling algorithm within FSR 3.1 inherits its core mechanics from FSR 2.0, which was a ground-up redesign focused on high-quality temporal reconstruction.

• Temporal Data Utilization and History Reconstruction: The algorithm maintains a history buffer that stores color information and other relevant data from previously processed frames. This historical data is reprojected into the coordinate space of the current frame using game engine-provided motion vectors. The process relies on jittered input samples (sub-pixel offsets applied to the projection matrix each frame) to allow the accumulation of sub-pixel detail over time, effectively increasing the information density available for reconstructing each output pixel at the higher resolution. Lanczos resampling is identified as a component in this process, featuring mechanisms to mitigate ringing artifacts and dynamically adjust resampling sharpness based on local temporal luminance stability to enhance detail in stable regions.
• Handling of Disocclusions and Thin Features: Robustly managing dynamic scenes requires specialized handling for areas where historical data becomes invalid or insufficient:
  • Disocclusion Detection: The system identifies regions of the scene that were occluded in previous frames but are now visible in the current frame (or vice-versa). This is primarily achieved by comparing depth values from the current frame with reprojected depth values from the reconstructed previous frame.
  • History Rectification (Depth and Color Based): When a disocclusion is detected for a pixel, its reliance on accumulated historical color is significantly reduced or clamped to a range derived from neighboring pixels in the current frame. This prevents “ghosting” artifacts, where outdated information from occluded surfaces might persist. For smoother transitions in disoccluded areas, FSR 2.0’s design includes pushing a new, potentially more blurred, sample during the upsampling stage for these regions.
  • Thin Feature Locking: To preserve the stability and integrity of very fine geometric details (e.g., wires, railings, distant foliage) that might only be sparsely sampled in any single low-resolution frame, FSR 2.0 introduced a “locking” mechanism. This logic detects pixel-wide ridges and emphasizes their historical color during rectification, preventing them from flickering or breaking up over time. These locks are designed to persist for the length of the jitter sequence or until invalidated by significant scene changes like disocclusion or substantial shading modifications.

The upscaling techniques employed in FSR 2.x and, by extension, FSR 3.1, are distinct from machine learning-based approaches. They rely on meticulously engineered, hand-coded algorithms. The detailed descriptions of mechanisms like biased Lanczos resampling, depth-based disocclusion handling with adaptive sample pushing, color-based history clamping, and the intricate logic for thin feature locking and unlocking underscore a complex array of heuristics and carefully designed solutions. This suggests a substantial investment in empirical tuning and expert graphics engineering to create algorithms capable of addressing a wide spectrum of challenging rendering scenarios, including fast motion, transparencies, and the preservation of fine details. This can offer advantages in terms of predictable performance characteristics and potentially more straightforward debugging, but may also necessitate more explicit guidance from developers (e.g., through reactivity and composition masks) to optimally handle specific edge cases or unique rendering styles.

Frame Generation

The frame generation component of FSR 3.1 is designed to further elevate the perceived frame rate by synthesizing new frames based on the temporally upscaled output. This involves two key technologies: Optical Flow and Frame Interpolation.

• Optical Flow (enhanced from AMD Fluid Motion Frames - AFMF): The FSR 3 Optical Flow workload, derived and improved from AMD’s Fluid Motion Frames (AFMF) technology, plays a critical role in frame generation. Optical flow algorithms perform a dense analysis of consecutive upscaled frames to estimate per-pixel motion between them. This process generates a set of optical flow vectors, which are typically more detailed and can capture more nuanced screen-space motion (e.g., from shadows, reflections, volumetric effects, and transparencies) than the often object-based or sparser motion vectors provided by the game engine for the upscaling pass.6 The FidelityFX SDK includes a dedicated FfxOpticalFlow component for this task, which takes the current and previous (upscaled) frames as primary inputs and outputs these detailed motion vectors.
• Frame Interpolation Techniques: Using the dense motion vectors generated by the Optical Flow pass, along with the two surrounding “real” (rendered and upscaled) frames, the frame interpolation stage synthesizes one or more entirely new frames to be inserted between them. The objective is to create visually plausible intermediate frames that significantly enhance the illusion of smooth motion, effectively doubling (or more, depending on the implementation) the displayed frame rate. The quality of these interpolated frames is heavily dependent on the accuracy of the optical flow field and the complexity of the on-screen motion. Fast, erratic, or unpredictable motion, as well as significant disocclusions or complex interacting transparent surfaces, can pose challenges for optical flow estimation and may lead to visual artifacts in the generated frames (such as warping, morphing, or object discontinuities). To mitigate the visibility of such artifacts, AMD recommends that the game should be running at a minimum of 60 FPS before frame generation is applied (i.e., after the upscaling pass), as interpolation artifacts tend to be more noticeable and disruptive at lower base framerates.

The quality and robustness of the Optical Flow algorithm are paramount for the success of FSR 3.1’s frame generation. Frame interpolation is fundamentally an “inventive” process, relying entirely on the information from surrounding real frames and the guidance of motion vectors to construct what happened “in between.” If the optical flow accurately captures the true motion of all on-screen elements, the resulting interpolated frame is likely to be of high quality and free from significant visual disturbances. Conversely, errors or inaccuracies in the optical flow—such as incorrect motion vectors or misinterpretations of complex occluding edges—will directly translate into visual artifacts in the generated frames. This underscores the importance of the FfxOpticalFlow component 20 as a specialized, and potentially computationally intensive, cornerstone of the FSR 3.1 frame generation pipeline.

Graphics Pipeline Integration

Proper integration of FSR 3.1 into the graphics pipeline is crucial for achieving optimal image quality and performance. This involves careful placement of both the upscaling and frame generation stages, as well as managing dependencies with other rendering passes.

• Upscaler Placement:
  • The FSR 3.1 upscaler, much like FSR 2.x, is designed to replace the game engine’s conventional Temporal Anti-Aliasing (TAA) pass. It should be integrated into the pipeline after all main scene opaque geometry has been rendered and essential G-buffer data (such as depth and motion vectors for the current frame) have been generated, but before any TAA would typically run
  • Post-processing effects that rely on or benefit from an anti-aliased input (e.g., certain types of bloom, lens flares, depth of field that assumes clean edges) should be applied after the FSR upscaling pass, operating on the high-resolution upscaled output.
  • Conversely, effects that require the raw, lower-resolution depth buffer or other non-anti-aliased G-buffer information (e.g., Screen Space Ambient Occlusion (SSAO), screen-space reflections if they are not re-rendered or adapted for the output resolution) should generally be executed before FSR upscaling.
• Frame Generation Placement & Swapchain Management:
  • The frame generation process (Optical Flow and Frame Interpolation) occurs after the FSR upscaling pass has produced a high-resolution frame. It also typically happens after the User Interface (UI) has been rendered, or the UI is composited separately onto the final displayed frames to avoid interpolation artifacts on UI elements.
  • A significant integration requirement for FSR 3.1 with frame generation is the management of the swap chain. Developers must either replace their existing swap chain or create a new one using the AMD FidelityFX API’s DX12 or Vulkan Frame Generation Swapchain interfaces. This custom swapchain is essential for correctly pacing and presenting the interleaved sequence of rendered (upscaled) frames and synthetically generated frames.
  • The Optical Flow and Frame Generation workloads are computationally intensive but can often be scheduled to run on an asynchronous compute queue if the hardware supports it. This helps to hide their latency and reduce their impact on the main graphics rendering pipeline.

Official Presentations

HIGH PERFORMANCE RENDERING IN SNOWDROP USING AMD FSR 3 - GDC 2024

TEMPORAL UPSCALING: PAST, PRESENT, AND FUTURE - GDC 2023

FIDELITYFX SUPER RESOLUTION 2.0

References