Repository navigation
Count overlaps in bench and correct documents that had drifted from the code - #24
Merged
Merged
Conversation
bench prints a coll column, tests/tools.rs requires it to be zero and the README says "Zero vehicle overlaps in every run" under the controller table. But bench never turned the safety check on: Sim::check_safety is off in a new simulation, and collisions is only counted by check_overlaps, which runs when it is on. The column could only be zero. * bench now runs every strategy of a table with the check on. The sensor study, which reports no overlaps, still runs without it. * The column is a total over the seeds instead of a mean, so that one overlap in one run can't round away to 0. * A unit test in bench moves one vehicle on top of another and requires the run to count it. It fails with the check left off. * run_scenario's comment says when the count it returns is real. * docs/architecture.md: bench joins the list of places the check runs, which also lacked the green-wave test; the bench row says what the column holds; and the test list gains the new unit test. With the check on, bench all 20 6 prints the same figures as before in every column and 0 overlaps in all 32 rows (192 runs of 25 simulated minutes), so the README's table and sentence stand as they are. Signed-off-by: Jim Schreckengast <schreck@gmail.com>
* ACCESSIBILITY.md said everything the app measures can be produced from the command line. The tools print the average and 95th-percentile delay, throughput, stops, person delay, pedestrian waits and near misses, corridor trips and the safety counts. CO2 per vehicle, the level of service, the breakdowns, the charts and a selected vehicle's figures are only in the app (src/gui/mod.rs), and only the cooperative example prints the maximum delay. The heading and the introduction made the same claim and change with it. * docs/assurance-case.md said a controller whose weights file is rejected starts from a fresh network. Since 75341af the loader skips a copy that doesn't load for the next one, ending with the built-in copy (src/control/weights.rs), and a Hybrid file whose network doesn't fit loads without it and keeps its base tuning (src/control/hybrid.rs). * ROADMAP.md planned a keyboard way to place sensors. The app has no such tool: its tools are Select, Vehicle and Pedestrian, and sensors are set with checkboxes and sliders (src/gui/mod.rs). * docs/architecture.md said every run of tests/cooperative.rs must show no monitor violations. Three tests run the crossing without that check, and the other runs must show more than that: no overlap between vehicle bodies, margins and times within a microsecond, and at least one crossing measured (safety_failures in tests/cooperative.rs). Signed-off-by: Jim Schreckengast <schreck@gmail.com>
The picture in section 9.4 of docs/architecture.md still showed 0.5.1, though the section says that pictures of parts that changed were retaken with the code released as 0.6.0. Retaken with TS_ABOUT=1, as Appendix A lists it, and cut at the same place: only the version number differs from the earlier picture. Signed-off-by: Jim Schreckengast <schreck@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
An audit of the documents against the code as released in 0.6.0 gave seven statements to check. Five were wrong and one picture showed the previous version; they are corrected here, each tied to the code that shows it. The seventh agrees with the code and the other documents on
main, and is left as it is.ACCESSIBILITY.md: "Everything the app measures can be produced from the command line"src/gui/mod.rs:1099-1111), breakdowns by approach, movement and class (src/gui/mod.rs:1188-1228) and charts.benchprints none of those (src/bin/bench.rs:166-169), and of the examples onlycooperativeprints the maximum delay (examples/cooperative.rs:106). Nothing outsidesrc/guireadsco2_per_vehor callslosdocs/assurance-case.md: a controller whose weights file is rejected "starts from a fresh network"src/control/weights.rs:51-65). A Hybrid file whose network doesn't fit loads without it and keeps its base tuning (src/control/hybrid.rs:143-156). A fresh network is what Deep RL does only when no copy loads (src/control/rl.rs:107-119).tests/weights_files.rs:152-170checks the orderROADMAP.md: keyboard ways to "place sensors"src/gui/mod.rs:33-37); sensors are set with checkboxes and sliders (src/gui/mod.rs:790-817)docs/architecture.md, section 9.4: the About picture shows 0.5.1CARGO_PKG_VERSION(src/gui/mod.rs:1561), which is 0.6.0 (Cargo.toml:4)TS_ABOUT=1, as Appendix A lists it, and cut at the same place: 389 pixels differ from the earlier picture, all in the version number. The alt text followsdocs/architecture.md, section 10.3, thetests/cooperative.rsrow: "Every run must show no monitor violations"safety_failures(tests/cooperative.rs:35-60) also requires no overlap between vehicle bodies, margins and times within a microsecond and at least one crossing measured. Three tests run the crossing without it (tests/cooperative.rs:71,tests/cooperative.rs:351,tests/cooperative.rs:369)README.md: "Zero vehicle overlaps in every run" under the controller table (README.md:271), and thetests/tools.rsrow: "without collisions"check_safetyis off in a newSim(src/sim/mod.rs:385). Overlaps are counted only incheck_overlaps(src/sim/mod.rs:2217-2261), which runs only when it is on (src/sim/mod.rs:1649-1651).benchnever turned it on, so itscollcolumn could only be 0benchnow turns the check on for its strategy tables (src/bin/bench.rs:22-28), reports the overlaps as a total over the seeds, not a mean that could round one away, and has a unit test that fails with the check off (src/bin/bench.rs:226-252). With that, the README sentence and the test row are true as written (see Results), so neither changessrc/sim/coop.rs:116)maindescribe the nominal gap the same way: it assumes no vehicle failsBecause of 6,
docs/architecture.mdalso changes in three places:benchjoins the list of where the safety check runs in section 6.1 (the list lacked the green-wave test too), thesrc/bin/bench.rsrow says what thecollcolumn holds, and section 10.3 lists the new unit test.Results
Traffic behavior doesn't change: the safety check only counts.
cargo run --release --bin bench all 20 6prints the same figures with the check on as it did with it off, in every column of all 32 rows (the run time at the end of each row aside), andcollis 0 in every row: 192 runs of 25 simulated minutes without an overlap. The README's controller table is reproduced exactly.The check added no run time that could be told from the noise of a busy machine: three alternating pairs of that command took 32 to 40 s without it and 35 to 38 s with it, apart from one run of 118 s during a load spike.
Checklist
cargo test --locked --releasepasses (zero vehicle overlaps)cargo fmt --allcargo clippy --locked --release --all-targets -- -D warningsis clean (and forwasm32-unknown-unknownif the app changed)THIRD_PARTY_LICENSES.mdis regeneratedgit commit -s) under the Developer Certificate of OriginTested on macOS only, with Rust 1.99.0; CI runs Rust 1.100.0, so its Clippy is the first to see the new lines in
src/bin/bench.rs. Coverage measured locally is 95.77% of lines, withsrc/bin/bench.rsat 100%.cargo deny --locked checkandpackaging/licenses.sh --checkpass; no dependency changes.By submitting this pull request, I agree that my contribution is dual licensed under MIT OR Apache-2.0, as described in CONTRIBUTING.md.