Deferred Renderer (#9258)

# Objective

- Add a [Deferred
Renderer](https://en.wikipedia.org/wiki/Deferred_shading) to Bevy.
- This allows subsequent passes to access per pixel material information
before/during shading.
- Accessing this per pixel material information is needed for some
features, like GI. It also makes other features (ex. Decals) simpler to
implement and/or improves their capability. There are multiple
approaches to accomplishing this. The deferred shading approach works
well given the limitations of WebGPU and WebGL2.

Motivation: [I'm working on a GI solution for
Bevy](https://youtu.be/eH1AkL-mwhI)

# Solution
- The deferred renderer is implemented with a prepass and a deferred
lighting pass.
- The prepass renders opaque objects into the Gbuffer attachment
(`Rgba32Uint`). The PBR shader generates a `PbrInput` in mostly the same
way as the forward implementation and then [packs it into the
Gbuffer](https://github.com/DGriffin91/bevy/blob/ec1465559f2c82001830e908fc02ff1d7c2efe51/crates/bevy_pbr/src/render/pbr.wgsl#L168).
- The deferred lighting pass unpacks the `PbrInput` and [feeds it into
the pbr()
function](https://github.com/DGriffin91/bevy/blob/ec1465559f2c82001830e908fc02ff1d7c2efe51/crates/bevy_pbr/src/deferred/deferred_lighting.wgsl#L65),
then outputs the shaded color data.

- There is now a resource
[DefaultOpaqueRendererMethod](https://github.com/DGriffin91/bevy/blob/ec1465559f2c82001830e908fc02ff1d7c2efe51/crates/bevy_pbr/src/material.rs#L599)
that can be used to set the default render method for opaque materials.
If materials return `None` from
[opaque_render_method()](https://github.com/DGriffin91/bevy/blob/ec1465559f2c82001830e908fc02ff1d7c2efe51/crates/bevy_pbr/src/material.rs#L131)
the `DefaultOpaqueRendererMethod` will be used. Otherwise, custom
materials can also explicitly choose to only support Deferred or Forward
by returning the respective
[OpaqueRendererMethod](https://github.com/DGriffin91/bevy/blob/ec1465559f2c82001830e908fc02ff1d7c2efe51/crates/bevy_pbr/src/material.rs#L603)

- Deferred materials can be used seamlessly along with both opaque and
transparent forward rendered materials in the same scene. The [deferred
rendering
example](https://github.com/DGriffin91/bevy/blob/deferred/examples/3d/deferred_rendering.rs)
does this.

- The deferred renderer does not support MSAA. If any deferred materials
are used, MSAA must be disabled. Both TAA and FXAA are supported.

- Deferred rendering supports WebGL2/WebGPU. 

## Custom deferred materials
- Custom materials can support both deferred and forward at the same
time. The
[StandardMaterial](https://github.com/DGriffin91/bevy/blob/ec1465559f2c82001830e908fc02ff1d7c2efe51/crates/bevy_pbr/src/render/pbr.wgsl#L166)
does this. So does [this
example](https://github.com/DGriffin91/bevy_glowy_orb_tutorial/blob/deferred/assets/shaders/glowy.wgsl#L56).
- Custom deferred materials that require PBR lighting can create a
`PbrInput`, write it to the deferred GBuffer and let it be rendered by
the `PBRDeferredLightingPlugin`.
- Custom deferred materials that require custom lighting have two
options:
1. Use the base_color channel of the `PbrInput` combined with the
`STANDARD_MATERIAL_FLAGS_UNLIT_BIT` flag.
[Example.](https://github.com/DGriffin91/bevy_glowy_orb_tutorial/blob/deferred/assets/shaders/glowy.wgsl#L56)
(If the unlit bit is set, the base_color is stored as RGB9E5 for extra
precision)
2. A Custom Deferred Lighting pass can be created, either overriding the
default, or running in addition. The a depth buffer is used to limit
rendering to only the required fragments for each deferred lighting
pass. Materials can set their respective depth id via the
[deferred_lighting_pass_id](https://github.com/DGriffin91/bevy/blob/b79182d2a32cac28c4213c2457a53ac2cc885332/crates/bevy_pbr/src/prepass/prepass_io.wgsl#L95)
attachment. The custom deferred lighting pass plugin can then set [its
corresponding
depth](https://github.com/DGriffin91/bevy/blob/ec1465559f2c82001830e908fc02ff1d7c2efe51/crates/bevy_pbr/src/deferred/deferred_lighting.wgsl#L37).
Then with the lighting pass using
[CompareFunction::Equal](https://github.com/DGriffin91/bevy/blob/ec1465559f2c82001830e908fc02ff1d7c2efe51/crates/bevy_pbr/src/deferred/mod.rs#L335),
only the fragments with a depth that equal the corresponding depth
written in the material will be rendered.

Custom deferred lighting plugins can also be created to render the
StandardMaterial. The default deferred lighting plugin can be bypassed
with `DefaultPlugins.set(PBRDeferredLightingPlugin { bypass: true })`

---------

Co-authored-by: nickrart <[email protected]>

Author: DGriffin91

Cargo.toml

@@ -611,6 +611,16 @@ description = "Illustrates bloom configuration using HDR and emissive materials"
 category = "3D Rendering"
 wasm = true
 
+[[example]]
+name = "deferred_rendering"
+path = "examples/3d/deferred_rendering.rs"
+
+[package.metadata.example.deferred_rendering]
+name = "Deferred Rendering"
+description = "Renders meshes with both forward and deferred pipelines"
+category = "3D Rendering"
+wasm = true
+
 [[example]]
 name = "load_gltf"
 path = "examples/3d/load_gltf.rs"

assets/shaders/array_texture.wgsl

@@ -1,6 +1,6 @@
 #import bevy_pbr::mesh_vertex_output    MeshVertexOutput
 #import bevy_pbr::mesh_view_bindings    view
-#import bevy_pbr::pbr_types             STANDARD_MATERIAL_FLAGS_DOUBLE_SIDED_BIT
+#import bevy_pbr::pbr_types             STANDARD_MATERIAL_FLAGS_DOUBLE_SIDED_BIT, PbrInput, pbr_input_new
 #import bevy_core_pipeline::tonemapping tone_mapping
 #import bevy_pbr::pbr_functions as fns
 
@@ -16,7 +16,7 @@ fn fragment(
 
     // Prepare a 'processed' StandardMaterial by sampling all textures to resolve
     // the material members
-    var pbr_input: fns::PbrInput = fns::pbr_input_new();
+    var pbr_input: PbrInput = pbr_input_new();
 
     pbr_input.material.base_color = textureSample(my_array_texture, my_array_texture_sampler, mesh.uv, layer);
 #ifdef VERTEX_COLORS

crates/bevy_core_pipeline/src/core_3d/main_opaque_pass_3d_node.rs

@@ -1,7 +1,7 @@
 use crate::{
     clear_color::{ClearColor, ClearColorConfig},
     core_3d::{Camera3d, Opaque3d},
-    prepass::{DepthPrepass, MotionVectorPrepass, NormalPrepass},
+    prepass::{DeferredPrepass, DepthPrepass, MotionVectorPrepass, NormalPrepass},
     skybox::{SkyboxBindGroup, SkyboxPipelineId},
 };
 use bevy_ecs::{prelude::*, query::QueryItem};
@@ -34,6 +34,7 @@ impl ViewNode for MainOpaquePass3dNode {
         Option<&'static DepthPrepass>,
         Option<&'static NormalPrepass>,
         Option<&'static MotionVectorPrepass>,
+        Option<&'static DeferredPrepass>,
         Option<&'static SkyboxPipelineId>,
         Option<&'static SkyboxBindGroup>,
         &'static ViewUniformOffset,
@@ -53,12 +54,24 @@ impl ViewNode for MainOpaquePass3dNode {
             depth_prepass,
             normal_prepass,
             motion_vector_prepass,
+            deferred_prepass,
             skybox_pipeline,
             skybox_bind_group,
             view_uniform_offset,
         ): QueryItem<Self::ViewQuery>,
         world: &World,
     ) -> Result<(), NodeRunError> {
+        let load = if deferred_prepass.is_none() {
+            match camera_3d.clear_color {
+                ClearColorConfig::Default => LoadOp::Clear(world.resource::<ClearColor>().0.into()),
+                ClearColorConfig::Custom(color) => LoadOp::Clear(color.into()),
+                ClearColorConfig::None => LoadOp::Load,
+            }
+        } else {
+            // If the deferred lighting pass has run, don't clear again in this pass.
+            LoadOp::Load
+        };
+
         // Run the opaque pass, sorted front-to-back
         // NOTE: Scoped to drop the mutable borrow of render_context
         #[cfg(feature = "trace")]
@@ -69,23 +82,17 @@ impl ViewNode for MainOpaquePass3dNode {
             label: Some("main_opaque_pass_3d"),
             // NOTE: The opaque pass loads the color
             // buffer as well as writing to it.
-            color_attachments: &[Some(target.get_color_attachment(Operations {
-                load: match camera_3d.clear_color {
-                    ClearColorConfig::Default => {
-                        LoadOp::Clear(world.resource::<ClearColor>().0.into())
-                    }
-                    ClearColorConfig::Custom(color) => LoadOp::Clear(color.into()),
-                    ClearColorConfig::None => LoadOp::Load,
-                },
-                store: true,
-            }))],
+            color_attachments: &[Some(
+                target.get_color_attachment(Operations { load, store: true }),
+            )],
             depth_stencil_attachment: Some(RenderPassDepthStencilAttachment {
                 view: &depth.view,
                 // NOTE: The opaque main pass loads the depth buffer and possibly overwrites it
                 depth_ops: Some(Operations {
                     load: if depth_prepass.is_some()
                         || normal_prepass.is_some()
                         || motion_vector_prepass.is_some()
+                        || deferred_prepass.is_some()
                     {
                         // if any prepass runs, it will generate a depth buffer so we should use it,
                         // even if only the normal_prepass is used.

crates/bevy_core_pipeline/src/core_3d/mod.rs

@@ -10,6 +10,9 @@ pub mod graph {
     pub mod node {
         pub const MSAA_WRITEBACK: &str = "msaa_writeback";
         pub const PREPASS: &str = "prepass";
+        pub const DEFERRED_PREPASS: &str = "deferred_prepass";
+        pub const COPY_DEFERRED_LIGHTING_ID: &str = "copy_deferred_lighting_id";
+        pub const END_PREPASSES: &str = "end_prepasses";
         pub const START_MAIN_PASS: &str = "start_main_pass";
         pub const MAIN_OPAQUE_PASS: &str = "main_opaque_pass";
         pub const MAIN_TRANSPARENT_PASS: &str = "main_transparent_pass";
@@ -24,13 +27,16 @@ pub mod graph {
 }
 pub const CORE_3D: &str = graph::NAME;
 
+// PERF: vulkan docs recommend using 24 bit depth for better performance
+pub const CORE_3D_DEPTH_FORMAT: TextureFormat = TextureFormat::Depth32Float;
+
 use std::{cmp::Reverse, ops::Range};
 
 pub use camera_3d::*;
 pub use main_opaque_pass_3d_node::*;
 pub use main_transparent_pass_3d_node::*;
 
-use bevy_app::{App, Plugin};
+use bevy_app::{App, Plugin, PostUpdate};
 use bevy_ecs::prelude::*;
 use bevy_render::{
     camera::{Camera, ExtractedCamera},
@@ -50,12 +56,17 @@ use bevy_render::{
     view::ViewDepthTexture,
     Extract, ExtractSchedule, Render, RenderApp, RenderSet,
 };
-use bevy_utils::{nonmax::NonMaxU32, FloatOrd, HashMap};
+use bevy_utils::{nonmax::NonMaxU32, tracing::warn, FloatOrd, HashMap};
 
 use crate::{
+    deferred::{
+        copy_lighting_id::CopyDeferredLightingIdNode, node::DeferredGBufferPrepassNode,
+        AlphaMask3dDeferred, Opaque3dDeferred, DEFERRED_LIGHTING_PASS_ID_FORMAT,
+        DEFERRED_PREPASS_FORMAT,
+    },
     prepass::{
-        node::PrepassNode, AlphaMask3dPrepass, DepthPrepass, MotionVectorPrepass, NormalPrepass,
-        Opaque3dPrepass, ViewPrepassTextures, DEPTH_PREPASS_FORMAT, MOTION_VECTOR_PREPASS_FORMAT,
+        node::PrepassNode, AlphaMask3dPrepass, DeferredPrepass, DepthPrepass, MotionVectorPrepass,
+        NormalPrepass, Opaque3dPrepass, ViewPrepassTextures, MOTION_VECTOR_PREPASS_FORMAT,
         NORMAL_PREPASS_FORMAT,
     },
     skybox::SkyboxPlugin,
@@ -69,7 +80,8 @@ impl Plugin for Core3dPlugin {
     fn build(&self, app: &mut App) {
         app.register_type::<Camera3d>()
             .register_type::<Camera3dDepthLoadOp>()
-            .add_plugins((SkyboxPlugin, ExtractComponentPlugin::<Camera3d>::default()));
+            .add_plugins((SkyboxPlugin, ExtractComponentPlugin::<Camera3d>::default()))
+            .add_systems(PostUpdate, check_msaa);
 
         let render_app = match app.get_sub_app_mut(RenderApp) {
             Ok(render_app) => render_app,
@@ -82,6 +94,8 @@ impl Plugin for Core3dPlugin {
             .init_resource::<DrawFunctions<Transparent3d>>()
             .init_resource::<DrawFunctions<Opaque3dPrepass>>()
             .init_resource::<DrawFunctions<AlphaMask3dPrepass>>()
+            .init_resource::<DrawFunctions<Opaque3dDeferred>>()
+            .init_resource::<DrawFunctions<AlphaMask3dDeferred>>()
             .add_systems(ExtractSchedule, extract_core_3d_camera_phases)
             .add_systems(ExtractSchedule, extract_camera_prepass_phase)
             .add_systems(
@@ -92,6 +106,8 @@ impl Plugin for Core3dPlugin {
                     sort_phase_system::<Transparent3d>.in_set(RenderSet::PhaseSort),
                     sort_phase_system::<Opaque3dPrepass>.in_set(RenderSet::PhaseSort),
                     sort_phase_system::<AlphaMask3dPrepass>.in_set(RenderSet::PhaseSort),
+                    sort_phase_system::<Opaque3dDeferred>.in_set(RenderSet::PhaseSort),
+                    sort_phase_system::<AlphaMask3dDeferred>.in_set(RenderSet::PhaseSort),
                     prepare_core_3d_depth_textures.in_set(RenderSet::PrepareResources),
                     prepare_prepass_textures.in_set(RenderSet::PrepareResources),
                 ),
@@ -101,6 +117,15 @@ impl Plugin for Core3dPlugin {
         render_app
             .add_render_sub_graph(CORE_3D)
             .add_render_graph_node::<ViewNodeRunner<PrepassNode>>(CORE_3D, PREPASS)
+            .add_render_graph_node::<ViewNodeRunner<DeferredGBufferPrepassNode>>(
+                CORE_3D,
+                DEFERRED_PREPASS,
+            )
+            .add_render_graph_node::<ViewNodeRunner<CopyDeferredLightingIdNode>>(
+                CORE_3D,
+                COPY_DEFERRED_LIGHTING_ID,
+            )
+            .add_render_graph_node::<EmptyNode>(CORE_3D, END_PREPASSES)
             .add_render_graph_node::<EmptyNode>(CORE_3D, START_MAIN_PASS)
             .add_render_graph_node::<ViewNodeRunner<MainOpaquePass3dNode>>(
                 CORE_3D,
@@ -118,6 +143,9 @@ impl Plugin for Core3dPlugin {
                 CORE_3D,
                 &[
                     PREPASS,
+                    DEFERRED_PREPASS,
+                    COPY_DEFERRED_LIGHTING_ID,
+                    END_PREPASSES,
                     START_MAIN_PASS,
                     MAIN_OPAQUE_PASS,
                     MAIN_TRANSPARENT_PASS,
@@ -341,12 +369,14 @@ pub fn extract_camera_prepass_phase(
                 Option<&DepthPrepass>,
                 Option<&NormalPrepass>,
                 Option<&MotionVectorPrepass>,
+                Option<&DeferredPrepass>,
             ),
             With<Camera3d>,
         >,
     >,
 ) {
-    for (entity, camera, depth_prepass, normal_prepass, motion_vector_prepass) in cameras_3d.iter()
+    for (entity, camera, depth_prepass, normal_prepass, motion_vector_prepass, deferred_prepass) in
+        cameras_3d.iter()
     {
         if camera.is_active {
             let mut entity = commands.get_or_spawn(entity);
@@ -361,6 +391,13 @@ pub fn extract_camera_prepass_phase(
                 ));
             }
 
+            if deferred_prepass.is_some() {
+                entity.insert((
+                    RenderPhase::<Opaque3dDeferred>::default(),
+                    RenderPhase::<AlphaMask3dDeferred>::default(),
+                ));
+            }
+
             if depth_prepass.is_some() {
                 entity.insert(DepthPrepass);
             }
@@ -370,6 +407,9 @@ pub fn extract_camera_prepass_phase(
             if motion_vector_prepass.is_some() {
                 entity.insert(MotionVectorPrepass);
             }
+            if deferred_prepass.is_some() {
+                entity.insert(DeferredPrepass);
+            }
         }
     }
 }
@@ -428,8 +468,7 @@ pub fn prepare_core_3d_depth_textures(
                     mip_level_count: 1,
                     sample_count: msaa.samples(),
                     dimension: TextureDimension::D2,
-                    // PERF: vulkan docs recommend using 24 bit depth for better performance
-                    format: TextureFormat::Depth32Float,
+                    format: CORE_3D_DEPTH_FORMAT,
                     usage,
                     view_formats: &[],
                 };
@@ -445,6 +484,22 @@ pub fn prepare_core_3d_depth_textures(
     }
 }
 
+// Disable MSAA and warn if using deferred rendering
+pub fn check_msaa(
+    mut msaa: ResMut<Msaa>,
+    deferred_views: Query<Entity, (With<Camera>, With<DeferredPrepass>)>,
+) {
+    if !deferred_views.is_empty() {
+        match *msaa {
+            Msaa::Off => (),
+            _ => {
+                warn!("MSAA is incompatible with deferred rendering and has been disabled.");
+                *msaa = Msaa::Off;
+            }
+        };
+    }
+}
+
 // Prepares the textures used by the prepass
 pub fn prepare_prepass_textures(
     mut commands: Commands,
@@ -458,6 +513,7 @@ pub fn prepare_prepass_textures(
             Option<&DepthPrepass>,
             Option<&NormalPrepass>,
             Option<&MotionVectorPrepass>,
+            Option<&DeferredPrepass>,
         ),
         (
             With<RenderPhase<Opaque3dPrepass>>,
@@ -467,8 +523,12 @@ pub fn prepare_prepass_textures(
 ) {
     let mut depth_textures = HashMap::default();
     let mut normal_textures = HashMap::default();
+    let mut deferred_textures = HashMap::default();
+    let mut deferred_lighting_id_textures = HashMap::default();
     let mut motion_vectors_textures = HashMap::default();
-    for (entity, camera, depth_prepass, normal_prepass, motion_vector_prepass) in &views_3d {
+    for (entity, camera, depth_prepass, normal_prepass, motion_vector_prepass, deferred_prepass) in
+        &views_3d
+    {
         let Some(physical_target_size) = camera.physical_target_size else {
             continue;
         };
@@ -489,7 +549,7 @@ pub fn prepare_prepass_textures(
                         mip_level_count: 1,
                         sample_count: msaa.samples(),
                         dimension: TextureDimension::D2,
-                        format: DEPTH_PREPASS_FORMAT,
+                        format: CORE_3D_DEPTH_FORMAT,
                         usage: TextureUsages::COPY_DST
                             | TextureUsages::RENDER_ATTACHMENT
                             | TextureUsages::TEXTURE_BINDING,
@@ -544,10 +604,56 @@ pub fn prepare_prepass_textures(
                 .clone()
         });
 
+        let cached_deferred_texture = deferred_prepass.is_some().then(|| {
+            deferred_textures
+                .entry(camera.target.clone())
+                .or_insert_with(|| {
+                    texture_cache.get(
+                        &render_device,
+                        TextureDescriptor {
+                            label: Some("prepass_deferred_texture"),
+                            size,
+                            mip_level_count: 1,
+                            sample_count: 1,
+                            dimension: TextureDimension::D2,
+                            format: DEFERRED_PREPASS_FORMAT,
+                            usage: TextureUsages::RENDER_ATTACHMENT
+                                | TextureUsages::TEXTURE_BINDING,
+                            view_formats: &[],
+                        },
+                    )
+                })
+                .clone()
+        });
+
+        let deferred_lighting_pass_id_texture = deferred_prepass.is_some().then(|| {
+            deferred_lighting_id_textures
+                .entry(camera.target.clone())
+                .or_insert_with(|| {
+                    texture_cache.get(
+                        &render_device,
+                        TextureDescriptor {
+                            label: Some("deferred_lighting_pass_id_texture"),
+                            size,
+                            mip_level_count: 1,
+                            sample_count: 1,
+                            dimension: TextureDimension::D2,
+                            format: DEFERRED_LIGHTING_PASS_ID_FORMAT,
+                            usage: TextureUsages::RENDER_ATTACHMENT
+                                | TextureUsages::TEXTURE_BINDING,
+                            view_formats: &[],
+                        },
+                    )
+                })
+                .clone()
+        });
+
         commands.entity(entity).insert(ViewPrepassTextures {
             depth: cached_depth_texture,
             normal: cached_normals_texture,
             motion_vectors: cached_motion_vectors_texture,
+            deferred: cached_deferred_texture,
+            deferred_lighting_pass_id: deferred_lighting_pass_id_texture,
             size,
         });
     }

crates/bevy_core_pipeline/src/deferred/copy_deferred_lighting_id.wgsl

@@ -0,0 +1,19 @@
+#import bevy_pbr::utils
+#import bevy_core_pipeline::fullscreen_vertex_shader FullscreenVertexOutput
+
+@group(0) @binding(0)
+var material_id_texture: texture_2d<u32>;
+
+struct FragmentOutput {
+    @builtin(frag_depth) frag_depth: f32,
+
+}
+
+@fragment
+fn fragment(in: FullscreenVertexOutput) -> FragmentOutput {
+    var out: FragmentOutput;
+    // Depth is stored as unorm, so we are dividing the u8 by 255.0 here.
+    out.frag_depth = f32(textureLoad(material_id_texture, vec2<i32>(in.position.xy), 0).x) / 255.0;
+    return out;
+}
+