Gaussian Splat SPZ Support #12582

keyboardspecialist · 2025-04-25T16:50:57Z

Description
Opening as a draft as some things are still in progress, and this is a long running feature branch.

This adds support for rendering Gaussian splats from 3D Tiles 1.1 with SPZ compression using a proposed draft extension KhronosGroup/glTF#2490

Another PR will remove old Model pipeline code

Affected files

GltfLoader - Incorporates SPZ extension support
GltfSpzLoader - Loads and decodes SPZ buffers within a glTF file
GltfVertexBufferLoader - SPZ loading support
PrimitiveLoadPlan - SPZ post-processing. Mostly for interleaving color with alpha and ensuring correct model orientation.
GaussianSplat3DTilesContent - Manages loading glTF with SPZ Gaussian Splats
GaussianSplatPrimitive - new primitive for managing a tileset's splats, sorting, and rendering.
GaussianSplatRenderResources - Container for common render settings and shader building
PrimitiveGaussianSplatVS.glsl - reworked vertex shader to run outside Model pipeline stages
PrimitiveGaussianSplatFS.glsl - reworked fragment shader to run outside Model pipeline stages

Known Issues

~~View matrix projection is currently incorrect. Splats are rendered in wrong orientation in relation to the camera.~~ Fixed
Handling tilesets with Empty3DTileContent root tiles.

Issue number and link

Testing plan

Will need to gather some single and multi-tile glTF SPZ assets to incorporate into the general testing suite.

Updated sandcastle, possibly a new asset with more rendering controls.

Author checklist

I have submitted a Contributor License Agreement
I have added my name to CONTRIBUTORS.md
I have updated CHANGES.md with a short summary of my change
I have added or updated unit tests to ensure consistent code coverage
I have updated the inline documentation, and included code examples where relevant
I have performed a self-review of my code

interesting scaling decomposed covariance

fragment shader impl

need to get instanced drawarrays working

…ost processing

…loading. Setup for instanced rendering of quads

Attempting to fix splat scaling

new uniforms for camera data, no more computing in vertex shader renamed splat stage define

some clean up

correctly uses viewProjection when sorting

prop cleanup

shader tweaks

ggetz · 2025-05-20T17:06:57Z

I'm taking an additional pass on code and testing now.

Just to make sure everyone is on the same page, this is my current understanding of the plan for this PR. @keyboardspecialist @weegeekps, please correct me if I misunderstood anything.

The goal is to merge this PR into main after the June 2nd release
There are a few very minor known issues that @keyboardspecialist will be documenting
There will be followups for refining the Sandcastles, CHANGES.md updates, and other messaging

ggetz

Thanks @keyboardspecialist!

I took a pass through the source code and did some initial testing. While everything appears to be working well, I think I need clarity on two things related to the code:

I think I'm missing some context around having both a "model" code path and a separate "primitive" code path– Would you mind filling me in? Is the plan to support both? If so why?
One of the lesson we learned from Voxels was that a separate API for specific data types rather than the existing Cesium3DTileset API was probably not the right move (see Voxel API should conform to Cesium3DTileset interface #12297). While we don't need to reply on a model implementation if it's not needed, ideally splats should still go through the Cesium3DTileset API. What's preventing that at the moment?

The rest of my comments mostly revolve around architecture, inline documentation, and code cleanup.

I did not deeply review GaussianSplatPrimitive.js yet. I think I'm missing some context around the above points.
This is running well for me in Sandcastle. But occasionally, I do see a red flash on startup. Is that expected?

ggetz · 2025-05-20T13:29:31Z

Apps/Sandcastle/gallery/3D Tiles Gaussian Splatting SPZ.html

+      window.startup = async function (Cesium) {
+        "use strict";
+        //Sandcastle_Begin
+        Cesium.Ion.defaultAccessToken =


Typically, we add any Sandcastle assets to the CesiumJS ion account. That way, the default token will work to access them, and there's no need to manually set the token.

Reach out if you need access or any help getting assets uploaded to that ion account.

@ggetz, is it alright if we hold off until we've settled on exactly what asset we're going to use as the demo asset before uploading to the CesiumJS account? It could be this tower, or it could be something else. Of course, we won't merge to main without having the sandcastle asset in the correct place.

To answer your questions:

The model path was the old pathway. We definitely don't want to support both. This code has since been stripped out from this branch. That implementation highlighted the need for our new approach.

This was simply a misunderstanding on my part. Nothing is preventing it. It does rely on Cesium3DTileset events for handling tile loading. It's wrapping the load calls from Cesium3DTileset which it shouldn't do. I do see rectifying this as high priority for this release as we don't want to incur unnecessary tech debt as we continue to add features and any bug fixes.

Got it. Thanks for the cleanup!

Cool, that would definitely be the best move from the user's workflow perspective as well. Happy to help talk through how to best handle this if needed. When you say this release, we expect that to be the June 6th-ish timeframe, correct?

ggetz · 2025-05-20T13:31:33Z

Apps/Sandcastle/gallery/3D Tiles Gaussian Splatting SPZ.html

+          {
+            maximumScreenSpaceError: 1,
+          },


Is setting this to 1 necessary for everything to work right now?

It's pretty atypical to set maximumScreenSpaceError to something other than the default, let alone as low as 1.

Yes, but we are working on fixing this in the tiler, and it will be fixed before launch.

ggetz · 2025-05-20T13:33:54Z

Apps/Sandcastle/gallery/3D Tiles Gaussian Splatting.html

What's the plan with both of these Sandcastles? Is the plan to ship them both, or only the SPZ version?

SPZ will be the only shipped sandcastle. This one will be removed in another PR.

ggetz · 2025-05-20T13:35:05Z

packages/engine/Source/Core/PixelFormat.js

+  RED_INTEGER: WebGLConstants.RED_INTEGER,
+  RG_INTEGER: WebGLConstants.RG_INTEGER,
+  RGB_INTEGER: WebGLConstants.RGB_INTEGER,
+  RGBA_INTEGER: WebGLConstants.RGBA_INTEGER,


These all need their own JSDoc comments.

packages/engine/Source/Scene/Cesium3DTileStyle.js

packages/engine/Source/Shaders/Model/GaussianSplatFS.glsl

ggetz · 2025-05-20T17:59:32Z

packages/engine/Source/Shaders/Model/GaussianSplatVS.glsl

+
+highp vec4 discardVec = vec4(0.0, 0.0, 2.0, 1.0);
+
+void gaussianSplatStage(ProcessedAttributes attributes, inout vec4 positionClip) {


The entry point (I think this function) should be documented.

packages/engine/Source/Shaders/Model/GaussianSplatVS.glsl

packages/engine/Source/Shaders/PrimitiveGaussianSplatFS.glsl

packages/engine/Source/Workers/gaussianSplatSorter.js

…plat-spz-concept

fixed typo causing cache error

lilleyse · 2025-05-22T15:59:07Z

Another PR will remove old Model pipeline code

I think it should happen in this PR so that old code isn't going into main and it's easier to review.

ggetz

Thanks @keyboardspecialist! I took a pass on GaussianSplatPrimitive. My biggest concern (beyond making Cesium3dTileset the top-level entry point in the API) is making sure we account for potential race conditions as pat of the asynchronous sorting operations.

ggetz · 2025-05-27T17:29:35Z

packages/engine/Source/Scene/GaussianSplat3DTilesContent.js

+
+  frameState.afterRender.push(() => true);
+
+  if (!defined(loader) || this._resourcesLoaded) {


if loader is not defined, then the following two lines will throw an error. Was that check supposed to be sooner in this function?

ggetz · 2025-05-27T17:32:28Z

packages/engine/Source/Scene/GaussianSplat3DTilesContent.js

+  this._tile.destroy();
+  this._tileset.destroy();


I don't think this should destroy the tile or the tileset, as these could still be referenced somewhere else.

ggetz · 2025-05-27T17:35:19Z

packages/engine/Source/Scene/GaussianSplat3DTilesContent.js

+};
+
+GaussianSplat3DTilesContent.prototype.destroy = function () {
+  this._gsplatData = undefined;


On the other hand, GaussianSplatPrimitive.js does have resources that will need to be destroyed. It'll need to be explicitly destroyed here by calling gsplatData.destroy().

ggetz · 2025-05-27T17:36:05Z

packages/engine/Source/Scene/GaussianSplatPrimitive.js

+});
+
+GaussianSplatPrimitive.prototype.onTileLoaded = function (tile) {
+  console.log(`Tile loaded: ${tile._contentResource.url}`);


As part of cleanup, make sure all debug logs are removed.

ggetz · 2025-05-27T17:37:15Z

packages/engine/Source/Scene/GaussianSplatPrimitive.js

+  /**@type {boolean}
+@private
+   */


Suggested change

/**@type {boolean}

@private

*/

/**

* @type {boolean}

* @private

*/

As part of cleanup, make sure comments are formatted correctly.

ggetz · 2025-05-27T19:53:39Z

packages/engine/Source/Scene/GaussianSplatPrimitive.js

+  });
+  promise.then((sortedData) => {
+    this._indexes = sortedData;
+    GaussianSplatPrimitive.buildGSplatDrawCommand(this, frameState);


Since this happens when the promise is resolved, this very likely will be called in a different frame then when the sorting process was kicked off. Is that all accounted for?

Keep in mind things can occur such as an older call could potentially resolve after a more recent sort resolves, meaning the new command will be overwritten by old data.

ggetz · 2025-05-27T19:55:08Z

packages/engine/Source/Scene/GaussianSplat3DTilesContent.js

+  is3DTileContent: {
+    get: function () {
+      return true;
+    },
+  },


Is this used anywhere?

ggetz · 2025-05-27T19:55:31Z

packages/engine/Source/Scene/GaussianSplat3DTilesContent.js

+  isTilesetContent: {
+    get: function () {
+      return true;
+    },
+  },


Is this used anywhere?

ggetz · 2025-05-27T19:59:29Z

packages/engine/Source/Scene/Model/GaussianSplatTextureGenerator.js

+  return GaussianSplatTextureGenerator._textureTaskProcessor;
+};
+
+GaussianSplatTextureGenerator.generateFromAttrs = function (parameters) {


Suggested change

GaussianSplatTextureGenerator.generateFromAttrs = function (parameters) {

GaussianSplatTextureGenerator.generateFromAttributes = function (parameters) {

Avoid abbreviations.

ggetz · 2025-05-27T20:52:47Z

packages/engine/Source/Scene/GaussianSplatPrimitive.js

+    tileset._selectedTiles.forEach((tile) => {
+      const gsplatData = tile.content._gsplatData;
+      this.pushSplats({
+        positions: new Float32Array(
+          ModelUtility.getAttributeBySemantic(
+            gsplatData,
+            VertexAttributeSemantic.POSITION,
+          ).typedArray,
+        ),
+        scales: new Float32Array(
+          ModelUtility.getAttributeBySemantic(
+            gsplatData,
+            VertexAttributeSemantic.SCALE,
+          ).typedArray,
+        ),
+        rotations: new Float32Array(
+          ModelUtility.getAttributeBySemantic(
+            gsplatData,
+            VertexAttributeSemantic.ROTATION,
+          ).typedArray,
+        ),
+        colors: new Uint8Array(
+          ModelUtility.getAttributeByName(gsplatData, "COLOR_0").typedArray,
+        ),
+      });
+
+      this._numSplats += gsplatData.attributes[0].count;
+    });
+


Consider batching these operation together.

Suggested change

tileset._selectedTiles.forEach((tile) => {

const gsplatData = tile.content._gsplatData;

this.pushSplats({

positions: new Float32Array(

ModelUtility.getAttributeBySemantic(

gsplatData,

VertexAttributeSemantic.POSITION,

).typedArray,

),

scales: new Float32Array(

ModelUtility.getAttributeBySemantic(

gsplatData,

VertexAttributeSemantic.SCALE,

).typedArray,

),

rotations: new Float32Array(

ModelUtility.getAttributeBySemantic(

gsplatData,

VertexAttributeSemantic.ROTATION,

).typedArray,

),

colors: new Uint8Array(

ModelUtility.getAttributeByName(gsplatData, "COLOR_0").typedArray,

),

});

this._numSplats += gsplatData.attributes[0].count;

});

const tiles = tileset._selectedTiles;

const totalElements = tiles.reduce((total, tile) => tile.content.pointsLength, 0);

const aggregateAttributeValues(componentDatatype, getAttributeCallback) => {

const aggregate = ComponentDatatype.createTypedArray(componentDatatype, totalElements);

let index = 0;

for (const tile of tiles) {

const primitive = tile.content._gsplatData;

const attribute = getAttributeCallback(primitive);

aggregate.set(attribute.typedArray, index);

index = aggregate.length;

}

return aggregate;

};

this._positions = aggregateAttributeValues(ComponentDatatype.FLOAT, (primitive) => ModelUtility.getAttributeBySemantic(

primitive,

VertexAttributeSemantic.POSITION,

));

// etc.

this._numSplats = totalElements;

keyboardspecialist added 30 commits June 7, 2024 16:20

initial vertex calcs for splats

0103673

removed early returns. we seem to hit them all the time

ea53962

interesting scaling decomposed covariance

covariance fixes

60022f4

fragment shader impl

hack to switch primitive type points to triangle_strips

ba77e31

refactor, use inverse matrices

8d67853

final point primitive experiment. moving to quads

9aaf292

quad drawing.

8c3aea4

need to get instanced drawarrays working

new splat pipeline files

7a3f472

shader org

4f9c886

switched rendering to not need vertex buffer attr

b235c09

tweaks

df28196

detect KHR_gaussian_splatting extension and set props and prime for p…

bdf4eb5

…ost processing

Buffer rework. inject attribute buffer for screen space quad at gltf …

7a4c864

…loading. Setup for instanced rendering of quads

splats but bad projection

c6b8d25

shader cleanup, simplification

a03abdd

minor tweaks

7d5b09a

tweaks

0620dca

Fixed position and color buffers

fe340a4

Attempting to fix splat scaling

scaling?

e352ca2

added new GAUSSIAN_SPLAT pass

3e74c74

pipeline execution for splats

d5f4069

new uniforms for camera data, no more computing in vertex shader renamed splat stage define

splats sorted via countSort

13dfa78

some clean up

invalidates splat commands when camera moves

31e8d13

correctly uses viewProjection when sorting

gaurded splat vertex setup in builddrawcommand

947d9e0

prop cleanup

command rebuild under control

9862eeb

guard gaussian pipeline stage

0069114

sorting fixes

427d490

shader tweaks

cleanup

fcfc3df

opacity fix

f4657c6

small tweak to re-enable instanceDivisor check on attr 0

2270dae

keyboardspecialist added 2 commits May 20, 2025 08:10

alphas access fix

914fa68

Merge branch 'main' into splat-spz-concept

41c8fee

ggetz requested changes May 20, 2025

View reviewed changes

keyboardspecialist added 6 commits May 21, 2025 02:23

Merge branch 'main' into splat-spz-concept

bbc553c

Merge branch 'splat-spz-concept' of github.com:CesiumGS/cesium into s…

f434dd7

…plat-spz-concept

missing arg

b091fdf

spec fixes

f9fe6cc

clean up

b426cc4

Merge branch 'splat-spz-concept' of github.com:CesiumGS/cesium into s…

488651c

…plat-spz-concept

ggetz added the priority - high label May 21, 2025

keyboardspecialist added 5 commits May 21, 2025 13:00

Remove non-static fromGltf interface

7fc29f5

remove comment

b5622e0

add pick interface

85a920e

remove commented code

ca6565d

unload after decoding SPZ data.

f55ac05

fixed typo causing cache error

keyboardspecialist added 8 commits May 22, 2025 11:59

revert to merge-base 7f57ecd

8f6cfdd

Removing old GS code and properties

0f416f2

old GS pipeline code removal

292309a

clean up

e6e2cd3

removed old GS pipeline shaders

881bf70

Removed old GS stages

8c6cb76

remove GS specs

5116b84

remove old test data

5bc2a79

weegeekps mentioned this pull request May 22, 2025

KHR_spz_gaussian_splats_compression KhronosGroup/glTF#2490

Open

keyboardspecialist added 3 commits May 23, 2025 03:59

GS primitive updated to handle current visible tiles at any LOD depth

104ad85

typo

0359cc0

removed commented code

c496440

ggetz reviewed May 27, 2025

View reviewed changes


		highp vec4 discardVec = vec4(0.0, 0.0, 2.0, 1.0);

		void gaussianSplatStage(ProcessedAttributes attributes, inout vec4 positionClip) {


		frameState.afterRender.push(() => true);

		if (!defined(loader) \|\| this._resourcesLoaded) {

-  /**@type {boolean}
-@private
-   */
+  /**
+   * @type {boolean}
+   * @private
+   */

	GaussianSplatTextureGenerator.generateFromAttrs = function (parameters) {
	GaussianSplatTextureGenerator.generateFromAttributes = function (parameters) {

Gaussian Splat SPZ Support #12582

Are you sure you want to change the base?

Gaussian Splat SPZ Support #12582

Conversation

keyboardspecialist commented Apr 25, 2025 • edited by lilleyse Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Affected files

Known Issues

Issue number and link

Testing plan

Author checklist

Uh oh!

ggetz commented May 20, 2025

Uh oh!

ggetz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lilleyse commented May 22, 2025

Uh oh!

ggetz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

keyboardspecialist commented Apr 25, 2025 •

edited by lilleyse

Loading