Verification log¶

Dated measurements behind development.md: the rows of the scale harness and the hand-run and scripted verifications of the transport rules. New rows are appended at the bottom of each table; the how-to (commands, budgets, what each run records) stays in development.md.

Budget history¶

The budgets of Scale verification changed while the rows below were taken; a row is judged by the rules of its date.

• --watch frame: the run without --stats joined the three runs in #135. Before #177 the p95 alone was judged at 250 ms (over: watch_frame_p95_ms in the older rows): a 60 s run drew 16-22 frames once the settle rule of #168 delayed the first one, the p95 of that few is the worst frame, and the host is not still within a harness run (the load's statistics writers work while the tool is matched, so every run heats the host for the next one - /proc/loadavg climbed from 3.6 to 24 on 8 CPUs over six back-to-back runs). The median is the frame a user sees; the p95 keeps a stall from hiding. At large the load alone takes 6.7 to 7.5 of the 8 cores, so the frame time there says more about the host than about the tool.
• --watch statistics coverage replaces the "dropped statistics samples = 0" budget (stats_dropped_samples in the older rows, #147): the counters are cumulative and the tool prints last - first, so a lost sample costs nothing until it leaves a pair without a measurement, and a run could lose 925130 of 1040228 samples while measuring every pair. The dropped samples are still recorded - growth of sample_lost - sample_lost_latency + sample_rejected after the first --watch frame, the higher of the two --stats runs, with the late-join losses of the one-shot runs (sample_lost_at_start, #134) "at start" - and judged by nothing.
• --stats coverage: a delivery proof alone counted until #153; since then only measured packets do. Until #168 the default one-shot was a fixed --timeout 5; now it stops once every matched RTPS_SENT writer has been heard and the measured entries stop growing for 3 s, at 30 s at the latest. The pairs with a delivery proof but no packet are recorded as delivered_unmeasured_default / _30s, and how the default run stopped as default_stopped_on / settled_at_s / default_wall_s; judged by nothing.

Scale results¶

Columns: - participants / topics / pairs: from the 30 s --stats document; - one-shot: the discovery wait / table time, with the lowest pair count of the three runs; - host: architecture, CPUs, memory and, since #177, the whole-VM cores the load took before the tool started (load_before.vm_cores_load_only, 5 s sample); - --watch frame: median / p95. Rows from before #177 were judged on the p95 alone, at 250 ms (over: watch_frame_p95_ms), and are left as they were; since #177 the median is judged at 250 ms and the p95 at 500 ms; - tool CPU (cores) / peak RSS; - dropped samples: growth during --watch, with the late-join losses of the one-shot --stats runs "at start", the one-shot coverage (a fixed 5 s window before #168) and, since #147, the --watch coverage. Rows from before #147 were judged on the dropped samples (over: ... stats_dropped_samples) and are left as they were; the number is record-only since. Rows from before #153 counted latency-only pairs (a delivery proof without a measured packet) as measured in the 5 s coverage, which is why a Lyrical row can show a 5 s coverage of 1.0 next to a --watch coverage of 0.0; they too are left as they were; - -v lines; - the load's RSS and the memory still free.

The rows dated 2026-09-17 were measured after #141 (the drain thread and the per-topic reader QoS); the earlier rows are before it.

Web viewer on the 30 s --stats documents, measured on the Docker host (Apple M3, 8 CPU, 24 GB, macOS 26.6) in headless Chrome 153. The times are medians of five loads, and the budget is 100 ms for every action at large. The rows dated 2026-09-20 were taken after #136: their filter cells include the viewer's 100 ms typing debounce (the work is the cell minus 100 ms), and the harness waits for the viewer's render counter rather than two frames; the select column did not exist before:

Replaying a recording (#82), measured the same way on 2026-09-22 with headless Chrome through web/test/cdp.js: the medium document of 2026-09-21 (2400 pairs) written compactly 60 times with observed_at one second apart (5.5 MB per frame, 332 MB), opened with ?src=, three loads. The first frame was on screen after 0.6-0.8 s, the whole file read after 1.4-1.6 s, and a step to the next frame (▶, measured until the render counter moved plus one animation frame) took 33 ms in all 24 steps, the two-frame floor. The page's JavaScript heap was 45 MB after the scan (126 MB in the first load, before a collection): the viewer keeps byte ranges and a few numbers per pair and frame, never the documents.

The live history (#218), measured on 2026-09-22 with headless Chrome through web/test/cdp.js: transport_viz_web running a stand-in producer that printed the same medium document with a fresh observed_at every second (5.7 MB per frame as serve.py sends it), ?live=1 with the default 512 MB bound, 30 frames followed and then 70 more after ◀, the synchronous work of each document event timed in the page. Following: 17 ms median, 20 ms max; paused: 12 ms median, 17 ms max. The bound was reached at frame 91: nine frames were dropped in that event, 14 ms in all, and 90 stayed kept. The JavaScript heap was 49 MB after 30 frames and 38 MB after 100 (the frames' text is in Blobs, outside the heap), and switching match by to guid read the 90 kept frames again in 0.8 s.

The Prometheus endpoint (#83), measured on 2026-09-22 in the Jazzy container: metrics_text() of web/serve.py over the medium document of 2026-09-21 (502 topics, 2400 pairs, --stats) built 24 413 series, 7.5 MB (344 KB gzipped; the endpoint does not compress), in 46.7 ms median and 53.0 ms max; a scrape of /metrics with curl through a running transport_viz_web took 45-77 ms.

The limit_p135 document has no pair to draw. First render stays far below 3 s. Until #136, each input event (every keystroke, and also a click on an arrow or node) rebuilt the model and rendered every visible arrow again, with one getTotalLength per label: a filter that hid most arrows was fast, clearing it rendered them all at about 0.16 ms per arrow, which is the 136 ms of the 2026-09-16 large row. Since #136 the model is built once per document, a click only toggles the selected classes and re-renders the panel, the label sits at a midpoint computed from the curve's control points, and typing is debounced by 100 ms; on 2026-09-20 clearing a filter on large takes 46 ms above the debounce, and a selection 35 ms, of which 33 ms are the two-frame floor of the measurement at 60 Hz. The medium and large documents were regenerated that day, hence their sizes.

The one-shot pair counts below the total are the --quiet 1 stops described in Scale verification (#133). What falls over first:

• small holds every budget.
• medium:
• the --watch frame was over budget (344 / 667 ms): collect 218 ms (apply_stats 93 ms, resolve 50 ms, the drain collect still does 26 ms) plus update 115 ms. Since #135 it holds it, at 152 / 204 ms with --stats and 54 / 81 ms without: the ROS graph is queried only while discovery goes on (2 to 4 of the 28 frames of a run, resolve 0 ms on the others), and update no longer copies the frame and applies the statistics to the copy (8 ms). What is left is apply_stats, 76 ms of a 137 ms collect;
• the counter samples lost over a 60 s --watch run vary between runs of one build: 0 in the --stats run and 5823 in the -v run of the same row (#147). Since #147 that number is recorded and no longer judged: what is judged is the --watch coverage, the pairs with measured packets among those HISTORY_LATENCY proves delivered, and it is 1.0 (2000 of 2000 /scale pairs at the last frame of both --stats runs, one counter sample lost, no stats-samples-lost). A first attempt judged the RTPS_SENT instances holding two or more samples instead and read 0.85 to 0.91 in runs that lost nothing: an instance with one sample is most often a locator nothing was sent to since the tool started (113 of 1270), so only a delivery proof tells a starved pair from an idle one;
• statistics used to go missing here - 26 % of the /scale pairs were not measured within 5 s. Since #141 every one of them is (coverage 1.0, from 0.74) and the one-shot table shows all 2400 pairs instead of one. What is left is 23 counter samples over a 60 s --watch run, which cost no pair its measurement: a counter is read as last - first, so only a loss that leaves an instance with fewer than two samples in the window does. The best-effort HISTORY_LATENCY losses are counted apart (stats.samples_lost_latency) and not judged;
• the --stats one-shot used to stop at 5 s, and the statistics writers had not finished handing their transient-local history to the tool's readers by then: coverage 0.0 on both Jazzy and Lyrical (#168). The last RTPS_SENT writer is first heard from at 16-20 s and measured entries keep arriving until 22-27 s, with pauses of up to 2 s in between. Since #168 the one-shot ends on the settle rule (every matched RTPS_SENT writer heard, measured entries unchanged for 3 s), at 16.6-20.1 s on Jazzy (coverage 0.953, 1.0, 0.9575 over three runs) and 16.9-22.7 s on Lyrical (1.0 in three runs), with --timeout 30 as the cap;
• the --watch frame gate judged the worst frame of a run on a host the harness itself heats (#177): once the settle rule of #168 delayed the first frame, a 60 s run drew 16-22 frames, whose p95 is the max, and the load's statistics writers work only while the tool is matched, so every run of the harness raises the host load for the next one (/proc/loadavg 3.6 to 24 on 8 CPUs over six back-to-back 60 s runs; the -v run, late in the order, read 753 / 925 ms once, and 198-216 ms median on its own). Since #177 --watch --stats starts at discovery quiet and 5 s (27-29 frames per run), the median is judged at 250 ms and the p95 at 500 ms, and the row says what the host was doing. -v itself costs a flat 27 ms per frame of render (33-38 ms against 8 ms) on 2000 pairs. Six runs of one build (Jazzy and Lyrical, three each): median 184-218 ms with --stats, 209-235 ms with -v, p95 227-328 ms, all within budget;
• the settle rule of #168 counted every RTPS_SENT entry with a delta, and on Lyrical the entries to multicast metatraffic (239.255.0.1:7400), to the SHM metatraffic ports 7000-7019 and to the tool's own port all move within the first seconds of any participant: one run settled at 8.0 s with 47 such entries, every writer heard from, and no /scale pair measured (coverage 0.0, #179). Since #179 only entries towards a discovered reader's unicast (kind, port) count (stats.measured_instances): coverage 1.0 in three Lyrical runs (settled at 27.7 s, 24.3 s, and once the 30 s cap with 545 entries measured and the handoff still producing them) and one Jazzy run (15.4 s). The Lyrical handoff finishes later than the Jazzy one, so its one-shot takes 24-30 s with the default --timeout 30;
• the first of the three one-shot runs can still see no pair at all ("pairs 0"): it starts 20 s after the load and the graph is not complete yet (#133 reports this), while the two that follow both show 2400.
• large:
• the load alone keeps 6.7 of the 8 cores busy, and the tool gets about a third of a core;
• a frame took 6.6 to 8.9 s (resolve 2.9 s: two rmw graph queries per topic every frame). Since #135 the median is 1.6 s with --stats and 0.44 s without, resolve 0 ms on all but 3 to 6 frames of a run and update 0.11 s; the rest is apply_stats (0.44 s) and the drain (0.38 s) on the third of a core the load leaves, with a p95 of 6.6 to 12 s. The frame budget is not judged at this size;
• the statistics readers lose millions of samples and used to measure no /scale pair at all. The tool says so (stats.samples_lost, the stats-samples-lost warning, #134), and since #141 the loss no longer costs the measurements: the 5 s coverage was 0.62, 0.97 and 1.0 over three runs of the same build, where before it was 0.0 in all three. The counter samples that are still lost (1.2 M in the last run) are nearly all intermediate values of a last - first the tool never needed, which is why a run can lose a million of them and still measure every pair;
• this rung is not reproducible on this host, and the coverage above is the spread it produces: the load alone takes 6.7 to 7.5 of the 8 cores before the tool starts, and the 0.62 is the run that started from 7.5. The best-effort HISTORY_LATENCY losses (1.7-1.9 M) are counted apart in stats.samples_lost_latency and judged by nothing;
• the one-shot table still takes 0.7 s.
• large_multi behaves the same way.
• large on the other distributions:
• on Kilted and Lyrical the nodes no longer subscribe to /parameter_events, so there are 4000 pairs instead of 5600;
• Humble has no statistics (the statistics budgets are n/a, and drain is empty), and a frame still takes 550 ms, 370 ms of it in resolve: the graph queries alone exceed the budget at this size (#135);
• Lyrical behaves like Jazzy (frames of 4.5–6.9 s, 2.1–3.5 s of them in drain) and its peak RSS is 312 MB, in the 30 s --stats run;
• on Kilted the tool's view is unstable: the 30 s --stats document holds 37 of 40 participants, and the --watch frames lose and regain pairs (159–3327 of 4000), while the one-shot tables before them saw all 4000. The load's VM CPU dropped from 6.5 to 3.3–4.8 cores during the --watch runs (#138, closed without a fix: Kilted is out of scope from 1.1.0 on, the cause was not found, and Jazzy, Humble and Lyrical do not show it).
• RATE of sporadic writers: a writer that sends one burst gets a single PUBLICATION_THROUGHPUT sample of megabytes per second. Examples: /map in Nav2 shows 11.5 MB/s, and /parameter_events in large shows 267 MB/s on Jazzy and 1.7 GB/s on Lyrical (#137). Fixed by removing the column and the subscription: the statistic is a per-write payload/interval sample, not a rate. A rate that is actually a rate is #143.
• nav2:
• frames and memory are small;
• with --stats, the tool uses a whole core in every run. All of it is one Fast DDS UDP receive thread (dds.udp.<port>: 0.995 cores; the tool without --stats uses none): the statistics readers take only UDP (#106), and one thread per port is also the ceiling behind the losses at large (#141);
• in a first run, both 60 s --watch runs saw no pair at all while the one-shot runs before them saw 1195. Not reproduced by hand or in the second run; this is why the harness now fails such a run.
• limit:
• the load alone keeps all 8 cores busy from 60 processes on, so these steps show a starved tool;
• 60 processes: the table takes 4.3 s after discovery, peak RSS is 385 MB in the --stats runs, and a --watch frame takes 40 s (drain 27 s, resolve 10 s): two frames per minute;
• 90 processes: the tool's view is incomplete and unstable: one-shots show 600–1075 of 13 500 pairs, --watch frames 0 to 15 000, and the 30 s --stats run needs 129 s;
• 135 processes: discovery practically stops. One-shots see 0–4 pairs, the 30 s --stats document is empty and that run was killed after 180 s. The load's summed RSS (11.8 GB, shared pages counted per process) leaves 676 MB free, which ends the ladder.

Verification results¶