Neural Networks Taxonomy - 17. Point Cloud and 3D Data Networks

17. Point Cloud & 3D Data Networks

A point cloud is an unordered set of 3-D points — the raw output of a LiDAR sensor or depth camera — so the network must give the same answer whatever order the points arrive in. PointNet, in Fig 19, solves this by applying a shared MLP to every point independently and then collapsing them with a symmetric max-pool into a single global descriptor. The architecture is a direct application of the set-invariant principle, and its success demonstrated that deep learning on raw point clouds is possible without the explicit structure of a grid.

Origins and rise

PointNet introduced the shared-MLP-plus-symmetric-pooling recipe.^[244] PointNet++ added hierarchical local grouping so the model could capture fine geometry, not just a global summary.^[245] A different line, neural radiance fields (NeRF), fits a small network mapping a 3-D coordinate and viewing direction to colour and density, rendering photorealistic novel views.^[246]

Convolutions on points

A global max-pool discards local geometry, so a family of models redefines convolution to act on an irregular neighbourhood of points. PointCNN learns an X-transformation that reorders and weights local points before a standard convolution, recovering the exploitation of spatial layout.^[247] PointConv treats the convolution kernel as a continuous function of relative position, approximated by an MLP and reweighted by the local density so it can be applied at any coordinate.^[248] KPConv instead places a set of learnable kernel points in space, letting each carry a weight that deforms to the local geometry.^[249] SpiderCNN parameterises its filters as a family of polynomial functions over neighbours,^[250] and PAConv assembles each kernel dynamically by mixing a learned weight bank with coefficients predicted from point positions.^[251] Countering this trend, PointMLP showed that a plain residual MLP with a geometric affine module rivals these operators, suggesting elaborate local extractors are not strictly necessary.^[252]

Graph-based point networks

Because a point cloud is naturally a neighbourhood graph, some models borrow directly from graph neural networks. Dynamic Graph CNN introduced EdgeConv, which builds a k-nearest-neighbour graph in feature space and recomputes it at every layer, so points that are semantically — not just spatially — close exchange information.^[253] This dynamic recomputation lets the receptive field follow the shape's structure rather than a fixed spatial radius.

Point transformers

Self-attention is permutation-invariant by construction, making it a natural fit for unordered points. Point Cloud Transformer and Point Transformer both apply attention within local neighbourhoods, the latter using vector attention so each channel is weighted independently.^[254][255] Stratified Transformer samples distant points sparsely and nearby points densely as attention keys, capturing long-range context for scene segmentation without quadratic cost,^[256] while OctFormer uses an octree to sort points into windows of equal count, making attention scale to millions of points.^[257]

Voxel and sparse-convolution backbones

A complementary route rasterises points into a 3-D grid so ordinary convolution applies. VoxNet was an early dense-voxel CNN for real-time object recognition,^[258] but a dense grid wastes computation on empty space. Submanifold sparse convolution fixes this by storing and convolving only occupied voxels, keeping the active sites sparse,^[259] and MinkowskiNet generalised sparse convolution to arbitrary dimensions, enabling high-resolution 4-D spatio-temporal networks.^[260] For autonomous driving, SECOND brought sparse convolution to LiDAR object detection, greatly speeding up the voxel backbone.^[261]

Implicit neural fields

Rather than store geometry as points or voxels, implicit models represent a shape as a continuous function learned by a network — a decoder in the spirit of the generative families. DeepSDF regresses the signed distance to the surface from any 3-D coordinate, so a whole class of shapes is compressed into one conditioned network.^[262] Occupancy Networks predict instead whether a point lies inside or outside the object, giving a watertight surface at any resolution.^[263] Convolutional Occupancy Networks add local convolutional features so the representation scales from single objects to whole scenes.^[264]

Registration and odometry

Aligning two overlapping scans is the classical registration problem, and learned methods now replace hand-tuned iterative closest point. PointNetLK unrolls a Lucas–Kanade alignment on PointNet global features, back-propagating through the optimisation steps.^[265] Deep Closest Point matches learned per-point features with attention and solves for the rigid transform in closed form,^[266] and RPM-Net adds a learned annealing that makes matching robust to noise and partial overlap.^[267] PointDSC then prunes wrong correspondences by enforcing spatial consistency, sharpening the final alignment.^[268]

Applications

• LiDAR perception for self-driving cars and robotics.
• 3-D shape classification, part segmentation, and registration.
• Novel-view synthesis and 3-D scene reconstruction.

Strengths and limitations