Performance benchmarks¶
Roadmap item 0.2-H6. The benchmarks measure the costs that bound how fast a run can go: the
scheduler’s overhead per module step, signal bus access, and lidar rays per second on each
capability tier. They live in tests/benchmarks/ (package lagrange_benchmarks, Google Benchmark).
1. What is measured¶
Benchmark |
What one item is |
Notes |
|---|---|---|
|
one module step |
N chained modules, each reading the previous one’s output and writing its own (one read, one write per step). With |
|
one |
1000 of each per step inside a module, so the scheduler’s share is negligible |
|
one ray |
Every beam against every shape, exact, in double, one thread: the definition Tier 0 must match bit for bit. A ground plane plus N random shapes within 20 m; 32 768 beams in all directions (a 32-channel, 1024-column scan) |
|
one ray |
What the lidar module runs on the CPU: the same exact tests, culled by Embree ( |
|
one ray |
The compute kernel over a bounding-volume hierarchy, same scene and beams. Wall time of a whole scan: shape conversion, hierarchy build on the CPU, upload, dispatch and read-back, as the module does at each update |
LAGRANGE_TEST_GPU=<name> picks the Vulkan device for Tier 1, as in the tests.
2. Running¶
colcon build --packages-up-to lagrange_benchmarks --cmake-args -DCMAKE_BUILD_TYPE=RelWithDebInfo
tools/ci/benchmarks.sh MACHINE [OUT_DIR] # three repetitions, medians against baseline.json
MACHINE is a key of tests/benchmarks/baseline.json. A result fails when it is slower than
its baseline by more than MAX_SLOWDOWN (default 1.25). Skipped benchmarks (no Vulkan device)
do not fail; a baseline with no result does. For a machine with no baseline, the script prints
the results. To record them:
install/lagrange_benchmarks/lib/lagrange_benchmarks/check_benchmarks.py \
--baseline tests/benchmarks/baseline.json --machine MACHINE --record "description" \
OUT_DIR/bench_core.json OUT_DIR/bench_lidar.json
colcon test runs each benchmark once as a smoke test only, because timings on a shared
machine are not pass/fail.
The nightly workflow has a benchmarks job for the hosted runner (machine key
ci-ubuntu-latest, Tier 1 on llvmpipe, MAX_SLOWDOWN 2.0 because shared runners vary).
Its baseline is recorded from the job’s first log. GitHub Actions workflows are disabled for now
(2026-10-24), so it has not run yet. The roadmap’s self-hosted NVIDIA runner gets its own key.
3. Baselines (2026-10-24, with the Embree Tier 0)¶
Intel Core i9-13900H laptop on Ubuntu 26.04, RelWithDebInfo, median of three:
Benchmark |
Result |
Per item |
|---|---|---|
Scheduler, 1 module |
25.3 M steps/s |
40 ns |
Scheduler, 10 modules |
22.5 M steps/s |
44 ns |
Scheduler, 100 modules |
21.8 M steps/s |
46 ns |
Scheduler, 100 modules, multirate |
21.4 M steps/s |
47 ns |
Bus read or write |
249 M/s |
4 ns |
Lidar reference, 50 shapes |
1.69 M rays/s |
19.4 ms per scan |
Lidar reference, 400 shapes |
0.145 M rays/s |
225 ms per scan |
Lidar Tier 0, 50 shapes |
178 M rays/s |
0.19 ms per scan |
Lidar Tier 0, 400 shapes |
61 M rays/s |
0.54 ms per scan |
Lidar Tier 1, Iris Xe, 50 shapes |
34.3 M rays/s |
0.96 ms per scan |
Lidar Tier 1, Iris Xe, 400 shapes |
36.5 M rays/s |
0.90 ms per scan |
Lidar Tier 1, llvmpipe, 50 shapes |
40.3 M rays/s |
0.81 ms per scan |
Lidar Tier 1, llvmpipe, 400 shapes |
10.4 M rays/s |
3.14 ms per scan |
The Iris Xe is Intel’s integrated GPU (Mesa 26.0.8). llvmpipe is Mesa’s software Vulkan device
(LLVM 21.1.8), which runs the same kernel on the CPU’s threads. The machine keys are
i9-13900H-iris-xe and i9-13900H-llvmpipe. Two runs agreed within 3 %.
What the numbers say:
The scheduler costs about 45 ns per module step, nearly flat from 1 to 100 modules. A 1 kHz physics module with ten 100 Hz sensors spends under 0.1 % of real time in it.
The reference scales with the shape count, because every beam tests every shape: 400 shapes are 12 times slower than 50. Tier 0 gives the same results bit for bit, about 100 times faster at 50 shapes and 400 times at 400: Embree leaves each beam only the shapes near its path, and the beams spread over the 20 hardware threads.
A Tier 1 scan has a fixed cost of about 0.8 ms (submit, wait, read-back), so with few shapes a hardware GPU is no faster than llvmpipe. The hierarchy keeps it flat as shapes grow: 400 shapes cost the Iris Xe nothing more, while llvmpipe slows four times.
On this machine, Tier 0 beats the integrated GPU by 1.7 to 5 times for these scans and is deterministic, so
--lidar autostays on the CPU. Tier 1 (--lidar gpu) is for lidars with far more beams, or a discrete GPU; it needs re-measuring there.Re-measured on a discrete GPU (2026-11-15, an RTX 4060 Laptop GPU, Tier 2, beside the same CPU family;
bench_lidar --benchmark_filter=LidarTier, medians of 3, the machine busy with a build, load average about 18, so both columns are slower than in the table above):Scan of 32 768 beams
Tier 0 (CPU)
Tier 1 kernel on the RTX 4060
50 shapes
1.90 ms
3.96 ms
400 shapes
3.40 ms
3.76 ms
The discrete GPU is still not faster: its round trip (upload, dispatch, read-back) dominates. So
--lidar autotakes the CPU on every tier, on purpose, and the server’s log says so (“lidar on the CPU (–lidar auto, on any GPU …)”). TODO(verify): an idle-machine run, and a lidar with far more beams, where the GPU may win.
4. Whole runs: where the server’s time goes (2026-11-12)¶
LAGRANGE_SERVER_STATS=1 makes lagrange_server print, once a second on stderr, the real-time
factor, the scheduler’s module time and what it spends besides (capture, record, world update,
the ROS bridge, emit), the ten costliest modules (ms per wall second, steps, ms per step), and with
viewers connected the loop rate and the scene deltas sent (per second, size, build time, skipped
for congestion). Timing is never part of the state.
Measured headless with --no-ros --rtf 0 (as fast as it goes) for 25 s, the median of the 1 s
windows after the first four, each scenario run before and after back to back. Machine: the
i9-13900H laptop with an RTX 4060 Laptop GPU, RelWithDebInfo; shared with about eight other
builds and test runs (load average 15 to 30 on 20 threads), so the absolute numbers are low and
noisy (±30 % between repeats); the ratios held across two full repeats.
Run |
Before (RTF) |
After (RTF) |
What changed it |
|---|---|---|---|
Sample AMR, warehouse ( |
7.3 |
23.2 |
no sleep while time is owed |
Quadcopter, open field |
14.5 |
52.9 |
no sleep while time is owed |
Thermal camera AMR, 6 people |
3.6 |
4.6 |
colour not drawn, palette in parallel |
Sedan, city block |
1.1 |
4.7 |
camera model on the GPU |
Sedan, city block, with ROS |
1.0 |
5.8 |
as above, and no images built for nobody |
Sedan in the town, 24 cars, 36 people |
0.8 |
2.8 |
camera model on the GPU, radar in parallel |
Town, no robot, 24 cars, 36 people |
3.7 |
4.8 |
no sleep while time is owed |
Per step (ms, same runs): the sedan’s front camera 76.6 → 11.6 (city) and 73.8 → 11.7 (town); its radar 2.3 → 1.05 in the town; the thermal camera 11.1 → 8.9; the ROS bridge with the sedan 34 → 3 ms per simulated second when nothing subscribes to the images.
What each change does:
The run never sleeps while it owes time.
runOnce()polled its sockets with a 5 ms timeout every iteration, also when the run was behind real time or ran with--rtf 0; a run that cannot keep up lost those 5 ms each time (the quadcopter spent 0.66 s of every second asleep). Now the poll waits only when the run is on time.The camera model runs on the GPU (
gpu/kernels/camera/camera_model.slang) right after the render, in the same submission; only the output’s size comes back. The CPUCameraModel::apply(18 to 50 ms for the sedan’s 960 × 540 camera: six Box-Muller normals per pixel) stays the reference (LAGRANGE_CAMERA_MODEL=cpu). sensors.md §2 has the agreement test.The radar casts and works out its echoes in parallel, summed in ray order (bit-identical, checked on a recording of the sedan), and on the GPU with
--lidar gpu.The thermal camera draws no colour (
RasterCamera::setColour(false)) and paints its palette in parallel.The ROS bridge builds camera images and lidar clouds only when someone subscribes (or the topic rides on a radio link or sensor delay, whose random draws must not change).
Viewers get scene deltas every 8 ms while a long batch of ticks runs, not once per batch: in the town at real time, 105 to 135 deltas a second instead of 40 to 50 (42 kB each, 0.1 ms to build; writing them to the socket is most of the cost).
What is left, largest first:
The camera render itself: 4 to 10 ms per frame for the sedan (8 samples, the realistic look’s shadow cascades), waited for on the simulation thread. Overlapping it with the next ticks would cost a frame of latency.
MuJoCo with many kinematic bodies: the town’s 655 bodies cost 0.16 to 0.25 ms per 1 ms step, of which
mj_step1(positions, broad phase) about 150 µs, the bus reads and writes of every body’s pose, twist and contact force about 80 µs,mj_rnePostConstraint30 µs.The CPU lidar, 1.3 to 4 ms per scan; Tier 1 (
--lidar gpu) is still not faster for these scans.
4.1 Second round (2026-11-12, later)¶
Same scenarios and method (--no-ros --rtf 0, 25 s, the median of the 1 s windows after the
first four), the build before (the first round’s state) and after run back to back per scenario,
three times. The table is the third round (load average 1 to 11 on 20 threads, the quietest);
the ratio column gives the range over all three rounds (load 2 to 21).
Run |
Before (RTF) |
After (RTF) |
Ratio (3 rounds) |
What changed it |
|---|---|---|---|---|
Sample AMR, warehouse ( |
31.1 |
31.8 |
0.99 to 1.02 |
nothing it uses |
Quadcopter, open field |
61.9 |
66.8 |
1.08 to 1.19 |
bus commits (most likely: nothing else it uses changed) |
Thermal camera AMR, 6 people |
11.4 |
13.2 |
1.16 to 1.8 |
render overlapped |
Sedan, city block |
10.0 |
12.9 |
1.29 to 1.45 |
render overlapped |
Sedan in the town, 24 cars, 36 people |
2.5 |
3.7 |
1.11 to 1.45 |
render overlapped, MuJoCo |
Town, no robot, 24 cars, 36 people |
8.3 |
18.6 |
1.87 to 2.39 |
MuJoCo, bus commits |
Per step (third round): MuJoCo in the town 0.091 → 0.035 ms, in the town with the sedan 0.095
→ 0.075 ms; the sedan’s front camera 6.4 → 4.1 ms (city) and 22.4 → 12.7 ms (town), the
thermal camera 4.2 → 3.3 ms (a split step’s cost is its two halves, without what runs between).
Every signal’s value is the same bit for bit before and after (all signals hashed every 100 ms
over 20 s of each scenario, cameras off as in lagrange determinism); lagrange determinism
passes for amr_sample_urdf, sedan_city and the sedan in the town.
What each change does:
The camera renders while the tick’s other modules run. The scheduler has split steps (
SplitStep, architecture.md §6): the camera submits its render instep(), the lidar, the radar and the rest of the tick run while the GPU draws, and the camera waits for it right before the first module that reads its images (none, in these runs: the end of the tick).The camera model reads the render from device memory: the images no longer go to host memory and back (labels and copies 1.4 → 0.45 ms on the GPU for the sedan’s camera).
MuJoCo leaves far kinematic bodies out of its broad phase (rigid_body.md): it makes no contact between two bodies without degrees of freedom but sorted and paired them anyway, about half of the town’s step. A world without degrees of freedom skips MuJoCo’s step (the town without a robot): only the time and the kinematic bodies’ poses change.
Bus commits touch less memory: a module’s outputs commit together, in runs; commit times in their own array; previous values kept only where a reader uses them. In the town 2700 body signals commit every tick.
The thermal camera remembers each surface’s apparent temperature by its temperature and emissivity (0.15 → 0.01 ms a frame for the sample AMR’s 138 surfaces).
The thermal camera’s image is drawn on the GPU after the render (
thermal_camera.slang, sensors.md §2), measured after the table above: the sample AMR’s camera 4.0 → 3.5 ms a frame, RTF 12.1 → 13.1 (the same build withLAGRANGE_THERMAL_CAMERA=cpuand without, twice each).
Measured and left alone: allocation per step is small (an LD_PRELOAD malloc counter: under one
per tick in the town, about four per tick with the sedan’s camera, radar and lidar); the humans
(0.04 ms per 10 ms step for 36 people) and the traffic (0.015 ms) are a few per cent of the town.
What is left, largest first:
The camera’s GPU work, 3 to 12 ms a frame in the town, set mostly by the laptop GPU’s clocks (a camera at 10 Hz of simulated time leaves it idle in between, and it runs slow): the same frame took 2.9 ms of GPU time in one run and 10.3 ms in the next. The CPU lidar after it in the tick hides 2 to 4 ms of it.
MuJoCo with a robot among many kinematic bodies: each step still runs its kinematics, velocities and
mj_rnePostConstraintover all 664 bodies (0.075 ms in the town with the sedan).The scheduler’s own work in the town, about 15 µs a tick: filling 1300 input slots and committing 2700 outputs, mostly memory traffic.
The CPU lidar, 1.7 to 4 ms per scan.
4.2 Third round (2026-11-12, day)¶
Same scenarios and method (--no-ros --rtf 0, 20 s, the median of the 1 s windows after the
first four), the build before (71880eb, the second round’s state) and after run in the order
before, after, after, before per scenario; load average 1.4 to 8 on 20 threads. Camera runs are
set by the laptop GPU’s clocks (below), so for them the table also gives the real-time factor
without the camera’s time (each window’s factor divided by one less the camera’s share of it).
Run |
Before (RTF) |
After (RTF) |
Without the camera, before → after |
What changed it |
|---|---|---|---|---|
Sample AMR, warehouse ( |
27.3, 29.2 |
31.3, 31.5 |
caster hierarchy, lidar noise |
|
Quadcopter, open field |
61.9 to 65.9 |
59.3 to 63.9 |
3 to 5 % slower: bus bookkeeping (won back in §4.3) |
|
Thermal camera AMR, 6 people |
12.3, 17.4 |
12.7, 12.8 |
21.8, 21.3 → 25.8, 25.8 |
caster hierarchy, lidar noise |
Sedan, city block |
10.4, 12.5 |
11.1, 11.7 |
19.6, 25.7 → 28.6, 32.2 |
as above |
Sedan in the town |
3.0, 3.0 |
3.6, 3.0 |
5.3, 5.7 → 7.0, 5.8 |
as above, MuJoCo writes |
Sedan in the town, |
2.6, 2.5 |
3.7, 3.4 |
2.9, 2.9 → 4.5, 4.0 |
as above, BVH refit |
Town, no robot, 24 cars, 36 people |
16.4, 16.6, 16.9 |
21.7, 22.0, 22.9 |
bus, MuJoCo writes |
Per step (medians): the AMR’s top lidar 1.03 → 0.56 ms; the sedan’s roof lidar 2.1 → 0.97 ms
(city) and 3.1 → 2.0 ms (town), its radar in the town 1.0 → 0.8 ms; MuJoCo in the town 0.040 →
0.028 ms, with the sedan 0.096 → 0.087 ms. Every signal is the same bit for bit before and after:
LAGRANGE_DETERMINISM_HASHES=1 lagrange determinism prints every state hash, and the before and
after builds give the same 201 hashes over 20 s for amr_sample_urdf, sedan_city,
quad_x500_hover, the sedan in the town and the town without a robot (and each passes
lagrange determinism).
What each change does:
The ray caster builds the hierarchy of shapes that never moved once.
ShapeCasterrebuilt Embree’s hierarchy over every shape whenever one moved: each lidar scan of a robot in a world rebuilt the world’s (about 10 % of the main thread with the sedan in the town). Shapes that have never moved now sit in a scene built once, the moved ones in a second scene rebuilt when they move; a top scene instances both.The lidar draws its range noise in parallel: a Gaussian per beam on one thread was about a millisecond of a 32 768-beam scan. Each beam has its own Philox stream, so threads change nothing.
Held inputs are copied only after a new commit, and commits copy only written signals (architecture.md §6): in the town MuJoCo’s 655 pose-command inputs, written every 10 ms, were copied every 1 ms (input preparation 9.5 → 3.5 % of the main thread).
MuJoCo writes a kinematic body’s unchanged values once (rigid_body.md): its pose when its command changes, its twist and contact force when they change, its zero acceleration once.
The GPU lidar refits its hierarchy instead of rebuilding it every scan (rebuilt when the inner boxes’ area has grown by a quarter, or after 30 scans).
Measured and not fixed:
The camera, and the GPU lidar, are set by the GPU’s clocks. With
nvidia-smisampled once a second during the sedan in the town: for the first 10 s the GPU stays at 2010 MHz (P0) and the front camera takes 3.0 ms a frame; then the driver drops it to 600 to 990 MHz (P5) at 30 to 40 % utilisation and the same frame takes 12 to 13.5 ms, for the rest of the run. The driver keeps the utilisation in a band by lowering the clock, so batching the same work differently does not raise it; only overlapping the render with the next ticks (a frame of latency) or a GPU held at its clocks would. Pinning clocks needs root (nvidia-smi -lgc); whether the PowerMizer “prefer maximum performance” setting holds them for a Vulkan compute client on Wayland is TODO(verify). Not changed here.Why Tier 1 (
--lidar gpu) is not faster. Timed per phase: building the shapes and the tree is 0.02 to 0.05 ms; the dispatch with its wait is the rest. The sample AMR’s lidar (14 400 beams, 13 shapes, trivial work) takes 1.1 ms per dispatch, and a second dispatch straight after the first as long (spinning on the fence instead of sleeping: 0.95 ms), at 400 to 870 MHz: the GPU is latency-bound at low clocks, where the CPU caster over 20 threads takes 0.56 ms. In the town the lidar’s and the radar’s dispatches also wait behind the camera’s render in flight (6.4 ms for 1 350 radar beams as for 32 768 lidar beams).--lidar autokeeps the CPU caster.The bus bookkeeping costs the quadcopter 3 to 5 % (62.6 → 59.9 and 65.7 → 63.1 RTF in two sets of four runs): its 100 µs ticks commit small signals that are all written every step, so the dirty flags and the per-slot checks are pure overhead there (merging nearby written signals into one copy took back part of it). Won back in §4.3.
MuJoCo with a robot among the kinematic bodies: the broad-phase culling already applies with a robot present (it is per moving tree), so what is left is MuJoCo’s per-body passes (kinematics, velocities,
mj_rnePostConstraint) over all 664 bodies, which a backend cannot restrict to a subset of anmjModel.
4.3 The quadcopter’s commits (2026-11-15)¶
The third round’s dirty flags (§4.2) made each commit look at every output of the module one by
one: its flag, then its record (a SignalRecord, a cache line each, for its offset and size), with
a branchy merge of neighbours. The quadcopter’s electrical network commits about 40 small outputs
every 100 µs tick, all of them written, so that was pure cost. Now (SignalBus::commitAll,
architecture.md §6):
A module that wrote every output and held none commits with one copy. At wiring the bus notes, per module, whether its outputs are plain values with consecutive ids (so one stretch of storage); then a commit checks the module’s dirty and held flags in one pass and, when all are written, copies the stretch, clears the flags and stamps the timings, without per-signal branches. The electrical network, MuJoCo and the controllers take this way.
The other commits read a 12-byte record per signal (
CommitInfo: offset, size, plain, previous value read) instead of the signal records, in one pass with the stores to the flags not forcing the compiler to reload the bus’s arrays.
Input preparation is unchanged (its held-copy check costs the quadcopter nothing measurable).
Measured as in §4 (--no-ros --rtf 0, 25 s, the median of the 1 s windows after the first four),
three builds of the same tree that differ only in SignalBus and its two lines in the scheduler:
old is the bus before the third round (71880eb), before the bus as it was (dirty flags), after
this one. Run in the order old, before, after, after, before, old, three times for the quadcopter
and twice for the town; load average 1.9 to 10.5 on 20 threads (other agents’ builds), so the
spread within a build is up to 10 %. The overhead column is the scheduler’s own time per tick, the
modules’ time less the ten costliest modules’ step times (in the town more than ten modules run,
so there it also holds the cheaper ones’ steps).
Run |
Old (RTF) |
Before (RTF) |
After (RTF) |
Overhead per tick, old / before / after |
|---|---|---|---|---|
Quadcopter, open field |
47.3, 52.1, 52.0, 46.5, 51.8, 52.0 |
43.0, 50.6, 46.5, 46.0, 49.7, 50.2 |
52.1, 52.2, 46.3, 46.5, 51.4, 52.1 |
335 to 384 / 392 to 472 / 340 to 391 ns |
Town, no robot, 24 cars, its walkers |
16.1, 14.6, 14.7, 10.5 |
18.2, 18.4, 16.4, 16.0 |
20.1, 19.8, 17.4, 18.0 |
21.2 to 31.3 / 13.4 to 15.0 / 8.7 to 10.3 µs |
Paired by round, the quadcopter is back to the old bus’s speed (in rounds 1 and 3 at their
quietest both builds run at 52 RTF, where before ran 3 to 5 % slower as in §4.2; round 2 ran at
46 for all three), and the town keeps the third round’s gain and adds 6 to 12 % to it (--robot none --traffic vehicles=24,pedestrians=36; the town’s own walkers walk, its lanes took no
traffic pedestrians). A synthetic run of the same shape (a
module at 100 µs writing 39 scalars, three at 1 ms, 2 × 10⁵ ticks per sample, the fastest of 40
samples) gives 466 ns per tick for the old bus, 517 for before and 427 for after.
Every signal is the same bit for bit: LAGRANGE_DETERMINISM_HASHES=1 lagrange determinism (whose
state hash covers every committed, previous, pending and input value and every commit’s time)
prints the same 201 hashes over 20 s with the before and after builds for amr_sample_urdf,
sedan_city, quad_x500_hover, the sedan in the town and the town without a robot (24 cars),
and each passes. test_scheduler has a case for the three ways a commit goes (every output
written, some written, one held) against a model of what the readers must see.
4.4 Lagrange’s own work in the MuJoCo backend (2026-11-15)¶
The MuJoCo 4 spike (mujoco-4.md) put Lagrange’s share of a step of
the 200-box pile at about 15 % on MuJoCo 3.12 (RTF 0.156 in Lagrange, 0.183 for boxes.xml in
MuJoCo alone with Lagrange’s options) and suspected the split step, the contact force and
acceleration pass and the bus writes. A timed build of MujocoBackend::step (steady-clock laps
around each stage, not committed) says otherwise:
Per 1 ms step (µs) |
Box pile, 5 s |
Sample AMR, warehouse |
Sedan in the town |
|---|---|---|---|
|
5546 |
10.6 |
5.3 |
|
370 |
6.4 |
45.2 |
|
88.4 |
0.68 |
18.4 |
|
13.1 |
0.17 |
0.25 |
Inputs read and applied |
1.3 |
0.14 |
10.8 |
Outputs written |
12.7 |
0.96 |
10.8 |
Kinematic culling |
0.06 |
0.03 |
2.2 |
Outside the module until its next step (scheduler, Python) |
3.9 |
21 (other modules) |
153 (other modules) |
On the pile, Lagrange’s own work is about 2 % of the step (120 of 6040 µs), and most of it is
mj_rnePostConstraint, which the IMU’s acceleration andjoint/<j>/reactionneed. The split step costs nothing: Lagrange’s compiled model (saved withmj_saveModel) stepped alone by a C program runs at RTF 0.167 withmj_stepand 0.166 withmj_step1/mj_step2, where Lagrange ran it at 0.166.The gap to
boxes.xmlis the model, not the step. Lagrange adds its ground plane at the top of the file’s own ground box (as it does when the file is given as the world too), and makes that box a static body of its own after the 200 boxes.boxes.xmlwith the plane added runs at 0.206 alone (1219 contacts at 5 s); with the ground box also moved into a body at the end, as Lagrange builds it, at 0.167 (1227 contacts, Lagrange’s 1227). The pile is chaotic, so where its geoms sit in the model changes how it falls and how hard the solver works by ±20 %. The spike’s 15 % is that, not Lagrange’s work.The doubled floor is gone (2026-11-15). Lagrange now adds its plane only to a file with no floor of its own (
hasOwnFloor: an upward plane, a height field, or a level slab at least 10 m wide; docs/guide/cli.md), for every loader and as robot or world. The pile then runs at 0.123 on MuJoCo (2.71 on Jolt, from 2.40): with no plane the boxes rest on the ground box in its static body, whose contacts merge MuJoCo’s constraint islands (34.8 per step against 43.9 with the box on the world body, 3.0 solver iterations against 2.2). MuJoCo alone with that model: 0.125 (853 contacts), against 0.184 with the box on the world body. Done in §4.5: 0.180.In the town the backend’s own work was large: 42 of 93 µs with the sedan (MuJoCo’s own steps 51), reading 663 bodies’ external wrenches (1 of them connected) and 654 pose commands, writing every kinematic body’s pose, twist and contact force, and
mj_rnePostConstraint.
What changed (MujocoBackend, bit-identical by construction):
Inputs nothing writes are read once. An unconnected optional input reads its default for the whole run (
SignalBus::allocate), soinitnotes which bodies’external_wrenchandpose_commandhave a writer, puts the others’ defaults in place once (again afterrestoreState, after its accelerations, which use the saved ones as before), andstepreads only the connected ones. Town with the sedan: 10.7 → 6.1 µs.A kinematic body that stands keeps its twist. Its twist comes from its last two poses; when both of the last two steps left its pose unchanged, the twist is the one already on the bus (the same poses give the same bits) and is not worked out again. Town with the sedan: 10.8 → 7.9 µs.
Not changed, and why:
Computing contact forces and
mj_rnePostConstraintonly when a module reads them. The bus knows a signal’s module readers, but not its other readers: Python’ssim.read, the ROSget_entity_stateservice (it readsbody/<b>/accelerationby name), the viewer’s scopes, the recorder, and the state hash. A skipped output would read stale values there, and the first read cannot know. On the pile it would save 1.7 % (88 + 13 µs of 6040); in the town 18 µs. A lazy path would need the bus to call back into the writer on a read: not worth it for this.A partial
mj_rnePostConstraintover the moving trees only (in the town it spends 18 µs on 654 kinematic bodies whose results Lagrange never reads): it would copy MuJoCo 3.12’s internals bit for bit, and 3.14 already changed that function.
Measured as in §4 (--no-ros --rtf 0, 25 s, the median of the 1 s windows after the first four;
the MuJoCo column is rigid_body/mujoco’s median ms per step), the 1.3 install (old) against
this build (new) of the same tree, in the order old, new, new, old, old, new; load average
5 to 26 on 20 threads (other agents’ builds; the AMR rounds at 17 to 26, the town’s at 5 to
25). The box pile is bench_lagrange.py (Python, 5 s, RTF of the stepping), load 5.6 to 10.4.
Run |
Old (RTF) |
New (RTF) |
MuJoCo per step, old / new |
|---|---|---|---|
Town, no robot, 24 cars, its walkers |
20.0, 18.6, 19.0 |
24.3, 24.7, 21.7 |
33 to 36 / 24 to 28 µs |
Sedan in the town, 24 cars |
2.61, 3.01, 2.80 |
2.98, 3.14, 3.06 |
110 to 127 / 96 to 107 µs |
Sample AMR, warehouse ( |
15.8, 24.9, 23.8 |
17.3, 25.3, 22.6 |
19 to 28 / 19 to 24 µs |
200-box pile, 5 s (Python) |
0.147, 0.160, 0.165 |
0.147, 0.148, 0.166 |
— |
The town without a robot (no degrees of freedom: the step is the inputs, kinematics and outputs) runs 20 to 25 % faster; with the sedan 5 to 10 %. The AMR (10 bodies, nothing connected but its wheels’ torques) and the pile (200 free bodies, no inputs) were within the load’s noise, as the per-stage times above predict.
Every signal is the same bit for bit: LAGRANGE_DETERMINISM_HASHES=1 lagrange determinism prints
the same 201 hashes over 20 s with old and new for amr_sample_urdf, sedan_city,
quad_x500_hover, the sedan in the town, the town without a robot and the box pile, and each
passes. test_mujoco_backend has a case for a kinematic body that stands and moves on, restored
while it stands.
4.5 Static scenery on MuJoCo’s world body (2026-11-15)¶
The MuJoCo backend built every world body as a MuJoCo body, so static scenery (a body welded to
the world, as the MJCF importer makes of worldbody geoms, or a world file’s fixed links) was a
static body of its own, and contacts with it slowed the pile’s solver (§4.4). Now
(staticScenery in MujocoBackend.cpp) a body that never moves (movingBodies: no joint to
the world, not kinematic, nor below one that is), in an articulated tree where no moving body has
geometry, has its geoms built on MuJoCo’s world body at their world poses, with the same
friction; such a body with nothing moving below it is left out of the model. Nothing referred to
it: static bodies have no signals (no pose, no contact force), and ray casts, the viewer and
--export-map use the world tables. A tree with moving geometry (a fixed-base arm) keeps its
static bodies, so its links’ collisions with its base stay excluded. The floating origin and
streamed tiles rebuild the run on the shifted or extended tables, so the geoms are placed anew
(tiles’ colliders were on the world body already). Jolt already makes every collider of a body
that never moves a static Jolt body on its static layer; it is unchanged.
Measured as in §4.4, the build before (dev at pull request 369, old) and this one (new) in
the order old, new, new, old, three rounds; load average 1.7 to 11.4 on 20 threads. Six runs of
each; the pile is bench_lagrange.py (5 s), the others --no-ros --rtf 0 for 25 s with the
median of the 1 s windows after the first four.
Run |
Old (RTF) |
New (RTF) |
MuJoCo per step, old / new |
|---|---|---|---|
200-box pile, MuJoCo, 5 s (Python) |
0.113 to 0.123 (median 0.122) |
0.176 to 0.181 (median 0.180) |
— |
200-box pile, Jolt, 5 s (Python) |
2.50 to 2.71 |
2.67 to 2.72 |
(Jolt unchanged) |
Sample AMR, warehouse ( |
25.7 to 29.5 |
28.2 to 29.4 |
16.8 to 18.5 / 17.0 to 17.5 µs |
Sedan, city block |
11.3 to 12.4 |
11.6 to 12.4 |
12.1 to 14.2 / 12.2 to 12.7 µs |
Sedan in the town, 24 cars, 36 people |
3.08 to 3.25 |
3.10 to 3.67 |
86.8 to 93.8 / 86.0 to 91.1 µs |
The pile runs 1.48 times as fast, as MuJoCo alone with the floor box on its world body (0.184). The other runs are within the noise: the warehouse’s static links carry only its seven collision boxes, and the city block and the town (USD worlds) already had their scenery on the world body.
State hashes change for runs whose world has static bodies: the MuJoCo state
(mjSTATE_INTEGRATION) holds an applied wrench per MuJoCo body, and the left-out bodies are no
longer in it. Where the scenery was already world geometry nothing changes:
Sample AMR in the warehouse (Python, 10 s; also
lagrange determinismofamr_sample_urdf,sedan_cityandquad_x500_hover, which load the default warehouse): new hashes, the robot’s pose after 10 s the same bit for bit. The quadcopter over the warehouse now gives the hash of the quadcopter alone (e72044ab06f46b7f).Sedan in the city block, sedan in the town, quadcopter alone: the same hashes.
The 200-box pile: new hashes and a different pile. The floor’s geom is now first in MuJoCo’s geom list, so contacts are found and solved in another order; the pile is chaotic, so after 1 s boxes differ by 1e-6 m at the bottom and up to 3 mm at the top (box 199).
Each run is still deterministic: lagrange determinism passes for all of them with both builds,
and Jolt’s pile gives the same hashes as before. test_mujoco_backend
(StaticSceneryIsWorldGeometryAndFixedBasesKeepTheirBodies) checks the split, the geom’s world
pose, a box resting on a welded shelf of a turned static tree, and a fixed-base arm whose link
overlaps its base and turns freely.
5. Start-up and Python throughput (2026-11-12)¶
The target (§13.4 of the project rules) is under 5 s from a cold start to the viewer’s first frame with a sample robot on a warm disk. Every sample is far below it; what is left is mostly the GPU.
LAGRANGE_TIMING=1 makes the server (and the Python Simulation) print on stderr how long each
load stage took (compose: the robot and world files read into world tables; ros; streaming;
lidar; cameras; build: the modules made and wired), each module’s declare and init, and
each step of making the camera scene (gpu device, pipelines, meshes, textures). Timing is
never part of the state.
Measured on the same laptop as §4 (i9-13900H, RTX 4060 Laptop GPU, RelWithDebInfo), shared with
other builds and test runs (load average 8 to 23). Before is night/integration as built, after is
this work; each is run in a clean environment, alternating, five times for the load and four
for the first frame. The tables show medians. The GPU was awake: see the cold GPU below.
Run |
Server load, before → after (s) |
First frame with |
|---|---|---|
Sample AMR, warehouse |
0.12 → 0.07 |
0.81 → 0.42 |
Sedan, city block (camera, radar, lidar) |
0.80 → 0.47 |
1.65 → 0.93 |
Sedan in the town |
0.81 → 0.52 |
1.73 → 1.30 |
Quadcopter, open field |
0.13 → 0.15 |
1.02 → 0.67 |
Tiled town, |
0.15 → 0.16 |
1.00 → 0.58 |
The load is lagrange_server --no-ros from exec until its socket listens. The first frame is
measured as the lagrange command starts things, with ROS: before, the server until it listened
and then the viewer (--headless --frames 1); after, both at once.
Rebuild |
Before (s) |
After (s) |
|---|---|---|
Spawn an AMR into the sedan’s city run ( |
0.70 to 0.80 each |
0.09 to 0.15 each |
Apply a profile (quadcopter hover), a spawn without cameras |
0.02 to 0.05 |
unchanged |
What changed:
The viewer starts beside the server.
lagrangewaited for the server’s socket and only then started the viewer, so the first frame came after both start-ups, one after the other. Now both start at once. Until its first connection the viewer’s client tries every 50 ms (for 20 s), not after an exponential backoff that has grown to seconds by the time the server listens. This saves 0.3 to 0.7 s on every run.A rebuilt run keeps its camera device. Spawning or deleting a robot, or a profile that rebuilds the run, made a new Vulkan device and decoded the realistic textures again for the cameras.
gpu::RasterScene::create(..., reuse)now shares the previous scene’s device, queue lock and textures. The meshes are uploaded again because they changed.The camera model’s ray table is computed in parallel. It holds the undistorted ray and the vignetting of every output pixel: 0.26 to 0.36 s on one core for the sedan’s 960 × 540 camera, now 20 to 50 ms. Each row’s reach is kept separately and combined in order, so the table is the same on any number of cores.
The sixteen surface photographs are decoded at once (one thread per JPEG): 100 to 175 ms became 50 to 90 ms, for camera runs and for the viewer’s realistic look.
Python:
Server::readfinds signals in a name index instead of walking every signal, andcommandJointkeeps the run’s drive modes instead of working them out on every call.LagrangeEnvbuilds its observed signal names once, andimage()no longer lists every signal before each frame. The API is unchanged.
Results are unchanged. lagrange determinism passes (AMR, quadcopter). Python state_hash()
after the same steps is equal before and after for the AMR, the quadcopter, the sedan and a
spawn sequence. The LagrangeEnv episodes give identical observations and hashes.
The cold GPU. On this laptop the NVIDIA GPU powers down when idle. The first Vulkan device
after that takes 2.1 to 2.3 s instead of 0.15 to 0.28 s; it was seen in about one run in five
here, after a pause. The server’s cameras and the viewer each pay it. Starting the two together
lets them pay it at the same time instead of one after the other. Nothing else in a start-up
comes close. A VkPipelineCache on disk was considered and not added: the camera pipelines
take 0.3 to 1.3 ms to build, because the NVIDIA and Mesa drivers keep their own shader caches.
Python throughput (sample AMR with joint_control, CPU lidar, medians of 2000 calls):
Call |
Cost |
|---|---|
|
0.1 to 0.25 µs |
|
0.1 to 0.35 µs |
|
18 to 38 µs |
One 1 ms tick through |
16 to 46 µs, the same whether called per tick or once for many |
|
1240 → 1420 per second (before → after, same run) |
|
10 900 → 11 100 per second |
|
~9 900 per second: the pipe to the worker costs ~45 µs a step |
The Python layer adds about 1 % to an RL step. The rest is the simulation itself (§4): the AMR’s
two lidars, MuJoCo and the electrical network. Under the shared load, ParallelEnvs with four
workers gave 1.5 to 1.8× one worker at 20 ms steps. Each worker’s CPU lidar already uses every
core, and pinning workers to separate cores was slower here.