From Agents to Atoms
For the past few years, the center of gravity in artificial intelligence has been the screen: large language models reading, writing, and orchestrating tools inside a digital sandbox. The next frontier is different. It is AI that perceives, reasons, and acts in the physical, three-dimensional world. We call this Physical AI: robots, autonomous machines, and self-driving vehicles contending with gravity, momentum, occlusion, and the unforgiving consequences of a wrong decision.
A few years ago, I would have argued that model architecture was still the primary differentiator. Working with various autonomous driving teams and watching them hit the same bottleneck, again and again, changed my view. As scaling laws increasingly dictate modeling choices, architectures converge, and the differentiator shifts to data: its quality, coverage, and grounding in real-world behavior. A language model can be trained on the open internet; there is no equivalent corpus of high-quality 3D data describing how the physical world behaves and how an expert system should respond to it. What exists is fragmented, inconsistently labeled, and rarely validated against a real operating system. This is a categorical gap, not a volume gap: the next generation of models demands a type of ground truth that does not yet exist at scale.
HyperFoundry Data Factory was built to close that gap. It is a single pipeline that turns real-world driving into machine-generated ground truth, spanning raw multi-sensor data, 3D scene labels, reasoning and action traces, and reconstructed 3D environments, all validated through a closed-loop autonomous driving stack developed for L4 operation. Here is why this data is now the rate-limiting input for Physical AI, and how we produce it.
Why Autonomous Driving Is the Proving Ground
The most demanding application of Physical AI today is autonomous driving, whose technical architecture has gone through three generations. AV 1.0 was modular: separate, hand-engineered components for perception, prediction, planning, and control, stitched together by rules. AV 2.0 increasingly shifted those hand-engineered components toward learned representations and end-to-end optimization: sensor data in, trajectory out. At PlusAI, AI models now implement the vast majority of driving functionality that historically required hand-engineered perception and planning logic. AV 3.0 is now emerging: world foundation models and Vision-Language-Action (VLA) systems that don’t just map sensors to trajectories but reason about novel situations in something closer to natural language. To be clear, we are not arguing that a single architecture will win; we are describing the data requirements that emerge across these competing paradigms.
This is not just our reading of the trend; the CVPR record over the past five years tells the same story, year by year.
In 2021 and 2022, the field was dominated by BEV perception and 3D detection benchmarks, with BEVFormer and its successors turning multi-camera input into a unified bird’s-eye-view representation.
The inflection came in 2023, when UniAD (“Planning-oriented Autonomous Driving”) won the CVPR Best Paper Award, the first ever given to autonomous driving, for unifying detection, tracking, mapping, occupancy forecasting, and planning in a single end-to-end network.
By 2024, the CVPR Autonomous Grand Challenge ran under the banner “Foundation Models for Autonomous Systems,” with tracks for end-to-end driving at scale, driving with language, and occupancy and flow.
In 2025, the challenge theme moved to world models and what the organizers call the “end-to-end 2.0” paradigm. Modular perception, to end-to-end, to world-model-plus-VLA: the research community has walked exactly this path.
But research and production are different disciplines. A leaderboard rewards raw accuracy; safety-critical OEM programs like our partnerships with TRATON GROUP, IVECO, and Hyundai reward something harder: functional safety and interpretability, achieved under real constraints on latency, compute cost, and power and thermal budgets. A 10-billion-parameter reasoning model that wins on a benchmark cannot run on automotive-grade silicon at highway speed.
This is why, in practice, PlusAI runs a hybrid architecture rather than a single monolithic model. A fast end-to-end REFLEX path handles the millisecond-level driving task. A REASONING layer, built on a fine-tuned vision-language model, interprets ambiguous, long-tail situations; it operates asynchronously and shapes behavior over a longer horizon, while the reflex path remains solely responsible for real-time control. A rule-based GUARDRAILS layer codifies human driving standards into verifiable safety constraints around both. Speed where you need speed, reasoning where you need adaptability, and a hard safety envelope around everything.
Do you still need BEV labels?
A fair question follows: if the model learns its own representation from sensors to action, do explicit 3D scene labels still matter? The honest answer is that it depends, because “end-to-end” and “world model” each now span two quite different design philosophies, and only some of them consume BEV labels in training.
Sample sequence of 3D object tracks
On the end-to-end side, the split is between explicit and implicit designs. Modular end-to-end architectures (UniAD as a representative example) keep an interpretable BEV representation in the middle of the network and supervise it with auxiliary perception losses for 3D detection, tracking, and mapping before the planning head. The labels are not the final training target; the trajectory is. But they are the intermediate supervision that make the pipeline learn faster, generalize better, and stay debuggable. For a safety-critical program, that interpretability is frequently a requirement. Engineers need to know why the system made a decision, and whether the functional-safety case works.
Pure implicit designs (sensors straight to action, no explicit perception layer) derive less direct supervisory benefit from BEV labels; their training signal is the human action itself, and the model learns its own latent scene representation. The industry is drifting toward this implicit, camera-first camp, especially among passenger OEMs, and it is worth being precise about where it is winning: in driver-supervised L2+ systems, where the human remains the safety backstop and the safety case can be statistical, built on shadow-mode comparisons, intervention rates, and large-scale simulation.
Remove the driver, and the bar changes structurally. A driverless system, above all an 80,000-pound truck, must be able to answer why it acted, to regulators, to OEM partners, and to its own safety engineers. That is not nostalgia for modular stacks; it is a consequence of removing the human. Implicit end-to-end is winning where the driver is the guardrail. Explicit intermediates persist where the system is.
And even the implicit camp does not actually escape 3D labels; it moves them out of the loss function. Open-loop metrics and closed-loop benchmarks are computed against labeled ground truth, and scenario mining (finding the cut-ins, construction zones, and rare VRU interactions worth training on) depends on knowing what is in each frame. The most prominent implicit-camp practitioner is the existence proof: Tesla runs one of the largest auto-labeling operations in the industry, and its public AI Day presentations showed multi-trip scene reconstruction and offline 3D labeling pipelines feeding occupancy training, evaluation, and data triggers even as its planner went implicit. The implicit camp does not refute the idea that labels are lifecycle infrastructure. It simply built the label factory in-house. Labels leave the training objective but stay in the loop for evaluation, diagnostics, and data selection.
The picture for world models is similar. A base driving world model predicts future observations conditioned on actions, self-supervised from video and action signals, with no labels in the core objective. But two increasingly common variants depend on them. Structured, semantic world models that forecast future states in BEV space (occupancy, agent positions) rather than future pixels often use BEV labels as a central supervision signal. And action-conditioned world models deployed as neural simulators rely on BEV and semantic conditioning for controllable generation, the ability to place a pedestrian or move a vehicle on demand, which is exactly what scenario generation for validation requires.
The throughline: as long as a production AV program demands interpretability, structured forecasting, controllable simulation, or rigorous evaluation (and safety-critical programs demand all four), high-quality 3D labels remain a core input, not a legacy artifact. The strategic question is who produces them. Building an in-house label factory at fleet-leader scale requires millions of vehicles on the road and a dedicated auto-labeling organization. For everyone else, machine-generated ground truth is how you compete with those who have both.
One Pipeline, Four Products
HyperFoundry produces four data products from a single integrated pipeline. Each is sold independently, so a team buys exactly the training signal it needs.
Raw Data. Synchronized, multi-modality sensor streams (surround camera, LiDAR, long- and short-range radar, IMU, GPS), temporally and spatially aligned. Not dashcam footage: full 360° surround perception with depth and velocity, from a fleet with over 1 million miles of real-world driving across the U.S., Europe, Australia, and Japan. Customers access the catalog with zero fleet CapEx, or commission on-demand collection with custom sensor configurations and vehicle platforms.
HD Scene Label (BEV). Complete, multi-sensor-fused bird’s-eye-view annotation. 3D object labels carry bounding boxes, tracking IDs, velocities, and semantic classes; static-scene labels cover vectorized lanes, road-surface segmentation, traffic lights, and HD map elements. Produced by an auto-labeling engine and validated against the L4 stack rather than human judgment alone, this is the interpretable 3D supervision described above, delivered at scale.
VLA Label. To our knowledge, no broadly available commercial dataset combines planner-derived reasoning traces with expert refinement at this scale. Inside every keyframe: structured decision traces, tokenized action trajectories, instruction-following pairs and driving VQA, visual grounding, and temporal context with causal event structure. The reasoning traces are derived from planner decisions and intermediate planner context (the trajectory candidates, cost evaluations, and constraints behind each action), then refined through expert human-in-the-loop workflows. The result is supervision grounded in the decision process of an L4-capable system driving on public roads, rather than an annotator’s approximation of what a system should have been thinking.
3D Reconstruction. Photorealistic 3D environments reconstructed from real driving data using Gaussian splatting and NeRF techniques, geometrically consistent with the captured sensor observations. These support novel-viewpoint rendering and closed-loop sensor simulation at substantial savings over hand-built, game-engine asset modeling.
What ties these together is that they come off one pipeline, from collection to raw to BEV to VLA to 3D reconstruction, rather than disconnected annotation tasks. Because the chain is machine-generated and L4-validated, it achieves roughly 98% first-pass acceptance and approximately 10x the throughput of manual annotation in our internal production benchmarks. Acceptance is measured against internal QA criteria across production labeling workflows. The throughput comparison is against conventional human-first annotation pipelines, where hundreds of seconds per frame and far lower acceptance rates are typical.
Evaluate It Yourself
Talking about data quality is easy; the only honest test is to put the data into your own training pipeline. If you are building perception, planning, or VLA models for Physical AI and your bottleneck is the quality, scale, or speed of your training ground truth, I would welcome the conversation. The architectures of the next generation demand a fundamentally new kind of ground truth.
Learn more at plus.ai/solutions/hyperfoundry or reach me at joe.yuan@plus.ai.