By: Chen Shen, Senior Manager, AI/ML Engineer at GM

Summary: Road to Sim is a pipeline that converts logs into high-fidelity simulation scenarios, reconstructing road geometry, traffic actors, and calibrated sensor feeds so the autonomy stack can experience realistic road conditions in simulation. Road testing alone cannot provide the repeatable, large-scale exposure needed to mathematically prove performance, measure rare edge cases, and validate safety metrics at scale before vehicles reach customers. A cloud execution pipeline turns road events into regression-ready tests, while behavioral metrics, human ground truth, and AI-assisted grading safeguard accuracy. In this post, we’ll explore how Road to Sim powers this data-to-simulation pipeline and why it is vital for both developer velocity and customer safety. The result is faster iteration, stronger safety evidence, and more comfortable autonomy experiences—with more learning extracted from every mile of road data.

From road truth to scalable evidence

When our Vice President of Autonomy, Rashed Haq, spoke at GM’s Autonomous Vehicle (AV) Day in May 2026, an internal employee event for GM’s Autonomy organization, he framed our work with a story from the earliest days of aviation. Back then, most inventors working on flight tested ideas by taking fragile prototypes straight into the open air. Each experiment meant a full-scale build and a real, risky flight. The Wright brothers did it differently. Instead of iterating only in the sky, they built a small wind tunnel in their bicycle shop and systematically tested more than 200 wing shapes. That shift to rapid, repeatable, controlled experimentation helped the Wright brothers refine their designs to build the first practical airplane.

Autonomous driving is at a similar inflection point. Building consumer trust requires a disciplined, step-by-step approach to validation. Real-world road testing remains essential—it delivers the ultimate ground truth of physical driving conditions and real traffic interactions, as well as insight into complex edge cases that may be encountered. However, road testing alone cannot provide the sheer volume or broader scenario coverage needed to validate an autonomy stack. Pure simulation gives us unlimited scale, but if designed in isolation, it risks testing synthetic conditions that don't statistically reflect what happens on real roadways.

Road to Sim closes the gap between the fidelity of road testing and the scale of simulation. If simulation is our wind tunnel, Road to Sim serves as a primary "wind generator"—converting actual road logs into calibrated synthetic sensor streams and scenario building blocks. While purely synthetic scenarios let us explore unencountered possibilities, Road to Sim anchors our simulation ecosystem in real-world driving behavior. In doing so, it helps our wind tunnel generate "wind" that reflects the complexity, nuances, and statistical distribution our vehicles face on physical roads. By grounding scalable simulation directly in real-world ground truth, we can iterate faster, validate rare events with statistical rigor, and deploy our vehicles to public roads with confidence.

Our end-to-end simulation environment is a powerful ecosystem composed of multiple complementary data and scenario sources. Road to Sim focuses on converting messy, real-world driving logs into high-fidelity semantic environments. It also produces calibrated sensor feeds that can be executed in a physically based rendering (PBR) simulator. Rather than treating a road log as a rigid, static recording, Road to Sim decomposes the drive into editable semantic components—such as road topology, actor trajectories, weather, and sensor models. By parameterizing these components, we can easily manipulate, reconfigure, and stress-test the underlying scene. As a result, a single on-road event can be multiplied into dozens of test variants—spanning different vehicle configurations, lighting conditions, weather patterns, and trajectory perturbations. While it represents just one part of our broader testing strategy, the ability to perturb and amplify real drives is a crucial link that transforms isolated events into the scalable, repeatable experiments that made the Wright brothers successful.

From real drives to simulation scenarios

Road to Sim starts with the raw material: real-world driving. Every manual data collection drive and on-road supervised test in a GM AV generates a detailed record of the environment around the vehicle and the actions taken by the human driver or our software. This includes sensor streams (camera, lidar, radar), vehicle state (IMU, speed, steering, control inputs), and detailed map context (lanes, curvature, traffic controls, road edges, elevation). On their own, these logs are just data. Road to Sim turns them into simulation-native building blocks the autonomy stack can train on and be tested against.

The pipeline begins by selecting events of interest from driving logs. Some are clear standouts—near-misses, tricky merges, construction, lane obstructions, unusual interactions with vulnerable road users (e.g., pedestrians, cyclists, motorcyclists), or events where our safety drivers retook control from our software. Others are intentionally boring: They are sampled from everyday driving to keep the test set grounded in the real-world distribution and preserve the baseline driving experience customers expect. Separate, targeted scenarios provide focused stress tests for rare events.

Once we’ve chosen a log, Road to Sim brings together three components to rebuild the drive in simulation: Environment Builder, Actors Builder, and our 3D end-to-end simulator. Working together, these systems recreate real-world road environments. The Environment Builder starts with a high-definition 2D semantic map and generates a full corresponding 3D geometry—including lane surfaces, curvature, elevation, overpasses, barriers, and roadside infrastructure. The result is a simulated roadway that closely mirrors the conditions the vehicle encountered.

Through AI models, Road to Sim then brings the scene to life in two steps. Motion reconstruction algorithms recreate the motion of every road user in the log, preserving trajectories, speeds, and timing relative to our AV. Asset matching algorithms then assign 3D assets that match each actor’s classification, shape, and appearance, so a truck looks and behaves like a truck, not a generic box.

Our 3D simulator then takes this reconstructed world and inserts a simulated ego vehicle—the vehicle under test, typically the autonomous vehicle whose systems are being evaluated—with realistic sensors, producing synthetic camera, lidar, and radar feeds calibrated to the real hardware. The autonomy stack processes sensor inputs through the same interfaces whether they come from a road test or a simulator.

The result is a reusable, self-contained description of the world, actors, and timeline that we can rerun while varying weather, lighting, traffic, or timing. Like a wind generator, Road to Sim turns a single slice of road data into many different “gusts” at the autonomy stack, probing behavior under realistic perturbations.

We then package these scenarios into test suites that run in continuous integration (CI) and pre-merge gates. New autonomy stack versions are exercised on curated mixes of challenging Road to Sim scenarios and everyday driving and other synthetic tests, and any issue we’ve found to be challenging for our software becomes a permanent regression test. Because this pipeline is automated, the test universe scales with fleet data, supporting tens of thousands of high-fidelity runs built directly from what our vehicles see.

Why scale is the story

After turning real drives into simulation scenarios, the next question is: How many tests do you need? For autonomy, the answer is far more than road testing alone can provide—and achieving statistical confidence requires two distinct types of scale: breadth and depth.

In autonomy validation, statistical confidence means supporting our core performance and safety metrics—such as time-to-collision and trajectory agreement—with sufficient sample sizes rather than small, isolated road samples. Measuring these metrics across millions of synthetic miles allows us to evaluate the statistical strength of our safety evidence.

Breadth establishes our baseline statistical coverage by converting millions of hours of everyday driving into simulation scenarios. This routine data forms the distribution baseline that helps our vehicles remain smooth, reliable, and comfortable during standard operation. By fanning out across thousands of cloud workers, our pipeline can convert millions of hours of road logs into simulation-ready regression tests.

Depth, on the other hand, provides focused amplification of safety-critical events. When an intervention, near-miss, or complex interaction occurs on the road, Road to Sim reconstructs it into a rich semantic model—allowing us to cleanly manipulate, enhance, and stress-test every element of the scene. We can then generate a whole family of variants by altering parameters like cut-in distance, vehicle spacing, traffic density, and lighting. Even for critical situations that are extraordinarily rare—perhaps occurring only once in tens of millions of miles—this semantic framework allows engineers to systematically stress-test real-world parameters. By synthetically amplifying real-world logs, we ensure our test suites thoroughly evaluate high-consequence edge cases without waiting to encounter them again on physical streets.

Together, this breadth and depth allow us to validate both baseline behavior and edge cases at a statistical scale unachievable on physical roads alone.

Keeping simulation grounded in reality

Road to Sim’s scale only matters if converted scenarios are trustworthy and representative of our environment. If going from road logs to 3D simulation changes the behavior the autonomy stack sees, then we are no longer testing what really happens on the road—we are testing artifacts of our own conversion pipeline. We’ve covered how we reconstruct scenes and scale up testing. How can we measure whether that reconstruction is sufficiently accurate for large-scale testing?

Behavior Similarity: Does the stack behave the same?

Our primary quantitative lens on conversion fidelity is a metric called Behavior Similarity. Intuitively, behavioral similarity asks a simple question: If we feed the same autonomy stack the same situation, once with road sensor data and once with synthetic sensor data from Road to Sim, does it plan to do the same thing?

Concretely, we compare two flavors of simulation:

• Replay: The AV stack runs in simulation but consumes real sensor logs from the road.
• 3D Simulation: The same stack runs in the 3D Simulator, consuming synthetic sensor streams generated by the Environment Builder and Road to Sim for the same event.

Behavioral similarity measures how closely the stack’s planned trajectories in Replay and 3D-Simulation match. High similarity indicates that the stack makes similar plans in both conditions; low similarity means conversion or simulation differences are pushing the stack toward a different decision. In those cases, we look first to Road to Sim or the simulator—missing geometry, mis-modeled actors, sensor artifacts—rather than the autonomy code. Aggregated across many events, behavioral similarity tracks the simulator’s road-likeness and pinpoints problematic maps, content, or scenario types.

Sim Test/Trust: Human judgments as ground truth

There are many aspects of a scenario that are obvious to humans but hard to capture in a single scalar measurement: a misaligned lane, a misplaced traffic cone, a vehicle exhibiting physically implausible motion, or a subtle mismatch in how a vulnerable road user moves.

To anchor metrics, we run a human-labeling and analysis program called Sim Test/Trust. In Sim Test/Trust, reviewers look at matched pairs of road and simulation runs to understand questions like:

• Does the simulated world geometry match what we see on the road?
• Are key actors present, in the right place, and moving in a plausible way?
• Would we trust this conversion to stand in for the original road event when evaluating autonomy behavior?

These human judgments provide the ground truth to interpret and calibrate metrics like behavioral similarity. When the metric looks good but reviewers flag problems, we may have found a gap in the metric. When both the metric and the reviewers indicate that a conversion is solid, our confidence in using that scenario for large-scale testing increases.

AI and vision-language models: Scaling trust with AI-assisted grading

Even with good metrics and a structured review program, manually inspecting conversions does not scale. To complement behavioral similarity and help human reviewers work more efficiently, we use AI agents.

The AI-powered agents compare road and simulation videos using vision-language models. They flag mismatches, describe them in natural language, and assign a coarse quality score to each conversion. To automatically grade conversion quality, the agents:

• Identify discrepancies (missing actors, misaligned objects, inconsistent traffic lights, or signage).
• Describe those discrepancies in natural language, example: “The truck in simulation is smaller and in a different lane than the truck on road.”
• Assign a coarse quality score that reflects issue severity.

This provides two benefits. First, agents can quickly surface the worst-converted scenarios and cluster common failure modes, so human reviewers spend their time where it matters most. Second, natural-language descriptions of mismatches make it easier for simulation teams to debug and prioritize fixes than raw numbers alone.

Metrics, humans, and models working together

Keeping simulation grounded in reality requires a layered system. Behavioral similarity quantifies how similarly the autonomy stack behaves under real versus synthetic sensor data. Sim Test/Trust provides human ground truth on whether the simulation world matches what happens on the road. AI and vision-language models help us scale review and turn road–simulation differences into actionable, human-readable feedback.

Together, they help ensure that Road to Sim’s “wind generator” produces what feels like the real thing, providing trustworthy evidence that we’re accurately simulating how the AV stack behaves on the road at scale.

Impact today, and where we go next

The Road to Sim pipeline accelerates how we develop, validate, and deploy autonomous-driving systems at GM. By bridging physical road drives and virtual simulation, we can run far more tests than are practical on roads alone. That lets us catch regressions earlier, iterate on autonomy code faster, and build a robust body of safety evidence before software reaches customers.

One powerful impact of this bridge comes from semantic reuse. When a road event is converted into a simulation scenario, we reconstruct the underlying world geometry, traffic controls, vehicles, and vulnerable road users. Because this representation is platform-agnostic, a challenging scenario originally encountered by an SUV can immediately become part of the regression test suite for a sedan.

The frontier of simulation is expanding rapidly. As rendering and generative capabilities evolve—from traditional game engines toward neural representations like 3D Gaussian Splatting (3DGS) and learned world models—the bridge between the real world and simulation becomes even stronger and more seamless.

At GM, Road to Sim is one of several specialized "wind generators" in our testing engine. We are continuously developing and integrating next-generation rendering and generative world modeling to work alongside our semantic pipelines. Bringing these complementary technologies together allows us to combine precise semantic control, high physical fidelity, and rich neural sensor-realism.

By continually improving the bridge between road data and virtual scale, our autonomy wind tunnel helps us test more realistically, learn more quickly, and build stronger evidence as we evaluate autonomy systems across GM. Every mile driven becomes an opportunity to learn—and every lesson moves us closer to a future of autonomous driving that is safer, smoother, and more capable, with greater confidence in every journey.