What is Gaussian Splatting and How Does It Work?

Imagine taking a few photos of a historic building and instantly creating a fully interactive, ultra-detailed 3D model that you can explore in real time. Traditionally, 3D reconstruction required either manually building models or using methods like photogrammetry, which could be slow and resource-intensive. But now, with Gaussian Splatting, we have a way to efficiently reconstruct and render 3D scenes at incredible speed and quality.

Gaussian Splatting is a point-based Radiance Field representation that allows for smooth, lifelike 3D reconstruction by representing a scene as soft, overlapping Gaussians rather than rigid geometric shapes.

This Radiance Field method enables real-time rendering through traditional rasterization, making it a compelling alternative to other Radiance Field representations, such as NeRFs (Neural Radiance Fields).

To understand Gaussian Splatting, we need to break down two key terms:

• A 3D Gaussian is a soft, blurry point in space that holds color, transparency, and density information. It’s like a tiny, glowing paint droplet that gently blends with its neighbors rather than having a hard edge.
• A Splat refers to how these Gaussians are projected onto the screen. Each one “splats” onto the image like a soft dot of color, and together, millions of these splats form a complete 3D scene.

A great way to think about this is pointillist paintings, like those by Georges Seurat. Up close, the image is just thousands of tiny dots of color. But when you step back, those dots seamlessly blend into a rich, detailed scene.

Both NeRFs and Gaussian Splatting are Radiance Field representations that can reconstruct 3D scenes from 2D images, but they do so in very different ways:

NeRFs are like a digital camera with a smart AI assistant. Imagine taking a few reference photos of a scene and asking an AI to recreate any possible view, even ones you didn’t capture. NeRFs learn how light behaves in the scene and use a neural network to generate new viewpoints mathematically.

Gaussian Splatting is like a pointillist painting. Every detail is already stored as tiny overlapping color points in 3D space. Because everything is precomputed, you can instantly move around the scene in real time, just like stepping closer or further from a painting.

Gaussian Splatting prioritizes speed and efficiency, making it ideal for real-time 3D rendering, whereas NeRFs, while highly realistic, currently require more processing power and are better suited for precomputed scenes or offline rendering. It is important to note that this will not always be the case.

Gaussian Splatting utilizes pieces of the traditional graphics pipeline, such as Rasterization and an explicit representation. This gives 3DGS a lot of up front advantages to its professional use and is why Gaussian Splatting is the first bridge to industry adoption of Radiance Field representations.

Gaussian Splatting is a way of representing a 3D scene as a collection of tiny, overlapping Gaussians that store color, transparency, and density.

Here’s how it works:

1. Input: Capturing the Scene

Just like with other Radiance Field representations, like NeRFs, Gaussian Splatting begins with a set of 2D images taken from different angles. These images help reconstruct the 3D structure of the scene.

2. Processing: Generating 3D Gaussians

Instead of creating geometric shapes (like meshes or voxels), Gaussian Splatting turns the scene into a collection of soft, overlapping 3D Gaussians. Each Gaussian holds position, size, color, opacity, and density information, allowing it to blend smoothly into the final image.

3. Rendering: Real-Time View Synthesis

The Gaussians are projected onto a 2D image plane and blended dynamically, enabling ultra-fast rendering using rasterization. Since each Gaussian is soft and continuous, the results look smooth and natural, with high-quality lighting effects.

While NeRFs and Gaussian Splatting both Radiance Field representations that reconstruct 3D scenes from images, they differ in how they represent and render those scenes:

NeRFs store a scene as a neural network function that predicts color and density at any 3D point based on input images. This enables high realism but lower rendering rates currently.

Gaussian Splatting directly stores the scene as a collection of 3D points, making it significantly faster to render in real-time while maintaining photorealism.

What Gaussian Splatting Does Better Than NeRFs:

Real-Time Rendering - Gaussian Splatting uses rasterization and is able to render lifelike 3D at well above real time rates (150fps +).

Compatible with industry standards - 3DGS outputs a ply file and is compatible with a wider variety of existing libraries such as React-Three-Fiber and Three.js, in addition to Web-GL and WebGPU.

Where NeRFs Might Be Better:

More Accurate Light Simulation - NeRFs are great at modeling higher fidelity reconstructions than Gaussian Splatting.

Compression - NeRFs such as Instant-NGP are able to be compressed to significantly smaller file sizes than Gaussian Splatting, already dipping into sub megabyte levels.

Ultimately, Gaussian Splatting has some up-front advantages that make it easier to work with modern computer architectures compared to NeRFs, right now. However, it is absolutely critical to note that Gaussian Splatting and NeRFs are not the only Radiance Field representations to exist. There are several incredibly promising representations that continue to be published and the capabilities of existing Radiance Field representations, will continue to exponentially increase across the board.

Gaussian Splatting is one of the most exciting Radiance Field representations because it solves many of the biggest challenges of previous methods:

Real-Time Rendering - Gaussian Splatting renders at well above real-time rates, making it great for interactive applications.

Compact & Efficient Storage - The entire scene is stored as millions of tiny Gaussians, compared to detailed meshes.

Handles Fine Details Better - Gaussian Splatting naturally blends colors, densities, and textures, making it great for real-world 3D captures with subtle details compared to Photogrammetry.

Works on a Wide Range of Devices - Because it’s computationally efficient, Gaussian Splatting can run on more consumer-grade hardware, democratizing high-quality 3D capture.

Seamless Integration with 3D Pipelines - Gaussian Splatting can be converted into meshes or used alongside traditional 3D assets, making it highly flexible.

Gaussian Splatting represents an exciting step forward in 3D reconstruction and rendering that prioritizes speed, realism, and efficiency. Instead of relying on heavy neural networks or complex polygonal structures