Most 3D object detectors are welded to their sensors: a camera-only model and a camera+radar model share almost nothing, so adding radar means retraining a different network from scratch. FUTR3D[1] breaks that coupling with one idea: the detection head never learns which sensors produced the scene. Cameras, radar, LiDAR, or any mix of them all feed a single modality-agnostic feature sampler, and a transformer decoder refines a fixed set of object queries against whatever came out of it. Drop a modality and the same weights still run. This article walks through a compact, self-contained implementation of the camera + radar variant — no dataset download, no trained checkpoint, just the plumbing running end to end.
One detector head, any sensor mix
FUTR3D descends from the DETR family of detectors. DETR[3] replaced hand-tuned anchors and non-max suppression with a fixed set of learnable object queries that a transformer decoder refines directly into boxes. DETR3D[2] lifted that to 3D by giving each query a point in space and sampling multi-view camera features at its projection. FUTR3D's contribution is to make that sampling step modality-agnostic: instead of asking only the cameras, it asks every sensor that happens to be present and fuses the answers before the decoder ever sees them.
Concretely, three things enter the model and one set of boxes comes out, as shown in Fig 1. The Camera_Backbone turns six surround images into multi-scale feature maps; the Radar_Encoder lifts each raw return into the fusion embedding; and 128 object queries each own a 3D reference point. The MAFS (Modality_Agnostic_Feature_Sampler) reads a fused feature at every reference point, and three Fusion_Decoder layers refine the queries and their points into class scores and oriented boxes.
Setup
python -m venv .venv
.venv\Scripts\activate # Windows
pip install -r requirements.txt # torch, torchvision, numpy, matplotlib, pillow
# For an NVIDIA GPU, install the CUDA build of PyTorch instead:
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
python src/futr3d_fusion_demo.py # one forward pass, prints the top-8 detectionsLine 1 creates the virtual environment; line 2 activates it on Windows; line 3 installs the CPU dependencies; lines 5–6 are the one-line CUDA-build alternative for an NVIDIA GPU; line 8 runs the whole pipeline. No dataset download and no trained checkpoint are needed — the scene is synthetic, so the run is reproducible on any machine.
The stack is PyTorch[4] and torchvision[5] for the model and its resnet18 backbone, with NumPy[6] building the synthetic sensor rig and Matplotlib[7] rendering the bird’s-eye-view figures. Every script runs from the project root and uses CUDA when it is available, falling back to the CPU otherwise.
The two sensor encoders
Each sensor gets its own encoder, and their only job is to produce features rich enough to sample — neither one knows what a query is. The Camera_Backbone runs all six views through a shared ResNet-18[8] trunk and a small FPN[9] that emits feature maps at strides 8/16/32, so a projected point can be sampled at three resolutions. The radar side is even simpler: radar is sparse but measures range and radial velocity directly — exactly what a camera cannot — so each raw return is lifted into the shared embedding by a point-wise MLP.
class Radar_Encoder(nn.Module):
def __init__(self, in_channels: int, embed_dim: int):
super().__init__()
self.encode = nn.Sequential(
nn.Linear(in_channels, embed_dim),
nn.LayerNorm(embed_dim),
nn.ReLU(inplace=True),
nn.Linear(embed_dim, embed_dim),
nn.ReLU(inplace=True),
)
def forward(self, radar_points: torch.Tensor) -> torch.Tensor:
# radar_points: (batch, num_points, in_channels)
# returns per-point features: (batch, num_points, embed_dim)
return self.encode(radar_points)Lines 4–10 build the point-wise MLP: two linear layers with a LayerNorm and ReLUs that lift each raw six-channel return — (x, y, z, rcs, vx, vy) — to the shared embed_dim of 256. Lines 12–15 apply it to every return independently (line 15), so a sweep of 256 radar points becomes 256 feature vectors the sampler can pool. Nothing here couples the returns to one another; all of the geometry lives in MAFS.
MAFS: sampling cameras at a 3D point
Every object query owns one 3D reference point, expressed as a normalized coordinate in [0, 1]. To turn that single point into a camera feature, MAFS first maps it into metres, then projects it into all six image planes and keeps only the views where the point is actually in front of the lens.
def reference_to_metric(reference_points: torch.Tensor, pc_range: tuple) -> torch.Tensor:
"""Map normalized [0, 1] reference points into metric BEV coordinates."""
lo = reference_points.new_tensor(pc_range[:3])
hi = reference_points.new_tensor(pc_range[3:])
return reference_points * (hi - lo) + lo
def project_to_images(points_metric: torch.Tensor, lidar2img: torch.Tensor):
"""Project metric 3D points into every camera image plane."""
ones = points_metric.new_ones(*points_metric.shape[:2], 1)
points_h = torch.cat([points_metric, ones], dim=-1) # (b, q, 4)
projected = torch.matmul(
lidar2img.unsqueeze(2), points_h[:, None, :, :, None]
).squeeze(-1) # (b, cam, q, 4)
depth = projected[..., 2:3]
uv = projected[..., :2] / depth.clamp(min=1e-4)
return uv, (depth[..., 0] > 1e-4)Lines 1–5 rescale each normalized reference point into metres inside the pc_range box (a ±51.2 m square, line 5). Lines 8–19 project those metric points into every camera at once: lines 10–11 make the points homogeneous by appending a 1, lines 13–15 apply each camera’s lidar2img matrix, and lines 17–18 do the perspective divide by depth to get pixel coordinates uv. The clamp on line 18 keeps that divide finite, and line 19 also returns a visibility mask that is True only where the depth is positive — i.e. the point is in front of the camera, not behind it. Downstream, MAFS bilinear-samples the multi-scale maps at these uv and averages only the views where the point was visible.
A masked softmax that returns NaN
The radar half of MAFS pools the encoded returns that land near the query in bird’s-eye view, distance-weighted so the nearest returns dominate. It is three lines of tensor code — and it hides the single nastiest bug in the whole pipeline.
def gather_radar_features(points_metric, radar_xy, radar_feats, radius):
"""Distance-weighted pool of radar returns within `radius` of each point."""
query_xy = points_metric[..., :2]
distance = torch.cdist(query_xy, radar_xy) # (b, q, points)
within = distance <= radius
# Softmax over -distance so nearer returns dominate; returns outside the
# radius get -inf and vanish. A query with no neighbour has an all -inf row,
# whose softmax is NaN — clamp that back to 0 so empty rows pool nothing.
scores = distance.neg().masked_fill(~within, float("-inf"))
weights = torch.nan_to_num(torch.softmax(scores, dim=-1), nan=0.0)
return torch.matmul(weights, radar_feats) # (b, q, embed)Line 3 takes each query’s BEV (x, y); line 4 computes the distance from every query to every radar return; line 5 flags the ones inside the radar_gather_radius of 4 m. Line 9 turns distances into scores — negated so the nearest return scores highest — and pushes everything outside the radius to -inf so it drops out of the softmax. That is where the trap springs: a query with no radar neighbour has an entire row of -inf, and the softmax of an all--inf row is NaN, not zero. Multiplying by the boolean mask does not rescue it either, because NaN × 0 == NaN — and one NaN propagates through the fuse MLP and poisons the entire forward pass on the very first decoder layer. Line 10 is the fix: nan_to_num rewrites those NaN weights back to 0, so a query with nothing nearby simply pools nothing and stays finite. Line 11 is the actual weighted sum over the encoded returns.
Refining queries and decoding boxes
The decoder is a stack of three DETR-style layers. Because MAFS produces one fused vector per query regardless of modality, the decoder is identical whether the scene had one sensor or two. Each layer lets the queries attend to each other, injects the freshly sampled sensor feature, and then nudges the reference point closer to a real object — so the sampling location improves every pass.
def forward(self, query, reference_points, sampler, sensors):
for layer in self.layers:
sensor_feat = sampler(
reference_points=reference_points,
feature_maps=sensors["feature_maps"],
lidar2img=sensors["lidar2img"],
radar_xy=sensors["radar_xy"],
radar_feats=sensors["radar_feats"],
image_hw=sensors["image_hw"],
)
query = layer(query=query, sensor_feat=sensor_feat)
delta = self.reg_branch(query)
reference_points = torch.sigmoid(inverse_sigmoid(reference_points) + delta)
centers = reference_to_metric(reference_points=reference_points, pc_range=self.pc_range)
box = self.box_head(query)
return {
"class_logits": self.cls_head(query),
"centers": centers + box[..., :3],
"sizes": box[..., 3:6].exp(),
"yaw": torch.atan2(box[..., 6], box[..., 7]),
"velocity": box[..., 8:10],
"reference_points": reference_points,
}Line 2 loops over the three layers. Each pass re-samples a fused feature at the query’s current reference point (lines 3–10); line 11 lets the layer update the query; line 12 predicts a small offset; and line 13 walks the reference point toward an object — inverse_sigmoid (the logit, the inverse of sigmoid) maps the point back to an unbounded space, the offset is added there, and sigmoid squashes it back into [0, 1]. That is the iterative box-refinement trick from Deformable DETR[10]. After the loop, line 15 converts the final point to metres, and lines 17–24 decode the last layer’s query: class logits (line 18), a box centre as the reference point plus a learned offset (line 19), positive sizes via exp (line 20), a yaw from a sin/cos pair (line 21), and a per-object velocity (line 22).
A synthetic scene, end to end
To exercise the whole thing without a dataset, src/futr3d_fusion_demo.py builds a nuScenes[11]-like ring of six pinhole cameras — each with a 90° field of view, so their wedges overlap into full 360° coverage — and scatters 256 radar returns on a ring 8–40 m out, each carrying an RCS and a radial velocity. That input scene is drawn in Fig 2.
Running the demo prints a coverage check that proves both sensor paths actually fire before any boxes are decoded:
$ python src/futr3d_fusion_demo.py
device : cuda (NVIDIA GeForce RTX 3060 Laptop GPU)
queries : 128
camera-visible : 128 queries project into >=1 view
radar-associated : 56 queries pooled a nearby return
finite output : TrueLine 2 reports the device (a CUDA GPU here, the CPU otherwise); line 3 the 128 object queries. Lines 4–5 are the coverage check: all 128 queries project into at least one camera view, and 56 of them pooled a radar return within 4 m. Line 6 confirms the decoded boxes are finite — the direct payoff of the nan_to_num guard from the radar pool. Overlaying the top-16 decoded boxes gives Fig 3.
Key takeaways
- FUTR3D’s whole trick is that the decoder never sees raw sensors — only the single fused vector MAFS samples at each query’s 3D reference point, so the same weights run for any subset of modalities.
- Cameras contribute through geometric projection and bilinear sampling of multi-scale maps; radar contributes through a distance-weighted pool of nearby returns. The two are concatenated and mixed by one MLP.
- The decoder borrows DETR’s object queries and Deformable DETR’s sigmoid-space reference-point refinement, improving the sampling location on every one of its three passes.
- Watch masked softmaxes: an all--inf row returns NaN, not 0, and masking cannot undo it because NaN × 0 == NaN — nan_to_num after the softmax is the fix.
- A synthetic surround-view scene is enough to run the entire pipeline end to end, with no dataset or trained weights, and to verify both sensor paths fire and the output stays finite.
The complete, runnable code for every figure above — the two sensor encoders, the modality-agnostic sampler, the decoder, and the synthetic-scene demo — lives in the companion repository under src/. Run src/futr3d_fusion_demo.py to reproduce the coverage check and the top-8 detections.
Resources
Every paper and library the implementation builds on — numbered in order of first appearance. No external photos or datasets are used: the scene is generated in code, so the figures in this article are original renders rather than derivatives of any licensed image. Click any bracketed marker such as [1] in the text to jump to its entry here.
- [1] FUTR3D — Chen et al., FUTR3D: A Unified Sensor Fusion Framework for 3D Detection, CVPRW 2023. arXiv:2203.10642
- [2] DETR3D — Wang et al., 3D Object Detection from Multi-view Images via 3D-to-2D Queries, CoRL 2021. arXiv:2110.06922
- [3] DETR — Carion et al., End-to-End Object Detection with Transformers, ECCV 2020. arXiv:2005.12872
- [4] PyTorch — Paszke et al., An Imperative Style, High-Performance Deep Learning Library, NeurIPS 2019 (BSD-3-Clause style). pytorch.org
- [5] torchvision — models, weights & transforms (BSD-3-Clause). github.com/pytorch/vision
- [6] NumPy — Harris et al., Array Programming with NumPy, Nature 2020 (BSD-3-Clause). numpy.org
- [7] Matplotlib — J. D. Hunter, Matplotlib: A 2D Graphics Environment, CiSE 2007 (Matplotlib license, BSD-style). matplotlib.org
- [8] ResNet — He et al., Deep Residual Learning for Image Recognition, CVPR 2016. arXiv:1512.03385
- [9] FPN — Lin et al., Feature Pyramid Networks for Object Detection, CVPR 2017. arXiv:1612.03144
- [10] Deformable DETR — Zhu et al., Deformable Transformers for End-to-End Object Detection, ICLR 2021. arXiv:2010.04159
- [11] nuScenes — Caesar et al., A Multimodal Dataset for Autonomous Driving, CVPR 2020. Dataset released under CC BY-NC 4.0 (non-commercial); this project uses no nuScenes data — only its surround-camera layout as inspiration — so the licence does not restrict reuse here. The devkit is Apache-2.0. nuscenes.org