feat(evals): ground-truth store, predicate library, xarm7 tabletop suite - #4040
feat(evals): ground-truth store, predicate library, xarm7 tabletop suite#4040Jerrybery wants to merge 3 commits into
Conversation
…ring MujocoSimModule gains publish_ground_truth (default off) and ground_truth_hz (20 Hz). When enabled, world poses of every free-joint scene body (robot root excluded — odom already covers it) publish on a new gt_object_poses Out[PoseStamped] stream, frame_id = body name. Ground truth is privileged scoring data for the evals framework (issue dimensionalOS#3594): never consumed by the agent. Part 1 of 2; part 2 adds the eval-side gt_store() and predicate library.
Part 2 of 2 for issue dimensionalOS#3594, stacked on the MujocoSimModule GT stream. - GTRecorder: Recorder subclass subscribing /gt_object_poses over LCM into its own per-case db — ground truth stays out of the agent's recording. The eval runner deploys it in-process for cases declaring ground_truth=True and passes --mujoco-publish-ground-truth (new GlobalConfig flag) so any mujoco blueprint emits GT. - EvalRunner.gt_store() + two-arg GTScore scorers: score(store, gt) alongside the existing score(store); sample() wires the GT store in when the scorer asks for it. - dimos/evals/predicates.py: spatial predicates over the GT store (inside_region, lifted, ...); grasped() is a placeholder until the GT stream carries contact data. - suites/xarm7_tabletop.py: regression suite scoring pick/place against GT object poses.
github-actions
Bot
added
the
first-time-contributor
 |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| names topics after it. | ||
| """ | ||
|
|
||
| gt_object_poses: In[PoseStamped] |
There was a problem hiding this comment.
This multiplexed input is recorded through the inherited recorder callback, which retains only the latest unprocessed message. MuJoCo sends one pose message per object in succession, so a later object's update can replace an earlier update when SQLite recording falls behind. The tabletop scorer can then receive no history for the cup or a bystander and fail an otherwise valid evaluation. Record this stream through a lossless serialized queue, or publish each tick's object poses as one atomic message.
Knowledge Base Used: Robot memory services
| # consumed by the agent. Off = zero overhead, zero behavior change. | ||
| # The global --mujoco-publish-ground-truth flag enables this too. | ||
| publish_ground_truth: bool = False | ||
| ground_truth_hz: float = 20.0 |
There was a problem hiding this comment.
ground_truth_hz accepts arbitrary floats, but the enabled post-step publishing path divides by it. A value of 0.0 raises ZeroDivisionError and stops the simulation loop; negative and infinite values publish every step, while NaN prevents publication. Require this value to be finite and greater than zero before starting ground-truth publication.
Knowledge Base Used: Simulation and digital twins
Artifacts
Ground-truth rate check script
- The script executes the checked-in publication branch with valid and invalid rate values.
- This capture contains the complete source used to exercise the publication-rate behavior.
- The output shows a divide-by-zero error for zero and incorrect publication behavior for negative, infinite, and NaN rates.
Ground-truth rate code location
- This capture identifies the changed declaration and the enabled publication branch that divides by the configured rate.
Codecov Report❌ Patch coverage is @@ Coverage Diff @@
## main #4040 +/- ##
==========================================
- Coverage 77.86% 76.43% -1.43%
==========================================
Files 1394 1433 +39
Lines 131706 140140 +8434
Branches 11430 12738 +1308
==========================================
+ Hits 102548 107115 +4567
- Misses 25900 29702 +3802
- Partials 3258 3323 +65
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 119 files with indirect coverage changes 🚀 New features to boost your workflow:
|
1 participant
Summary
Part 2 of 2 for #3594 (privileged ground-truth channel for interactive eval scoring), stacked on #3960 (the
MujocoSimModuleopt-in GT pose stream). The diff will shrink to just the eval-side commit once #3960 lands.This PR adds the eval-side consumption of the GT stream introduced in #3960:
GTRecorder(dimos/evals/gt_recorder.py):Recordersubclass subscribing/gt_object_posesover LCM into its own per-case db — ground truth stays out of the agent-visible recording. Uses directsubscribe()rather thanhandle_<stream>auto-subscription, per the single-slot LATEST mailbox caveat noted in feat(simulation): opt-in ground-truth object pose stream for eval scoring #3960.EvalRunner.gt_store()+ two-argGTScorescorers (dimos/evals/runner.py):score(store, gt)alongside the existingscore(store);sample()wires the GT store in when the scorer asks for it. The runner deploysGTRecorderin-process for cases declaringground_truth=Trueand passes--mujoco-publish-ground-truth(newGlobalConfigflag) so any mujoco blueprint emits GT during eval runs.dimos/evals/predicates.py: spatial predicates over the GT store (inside_region,lifted, ...).grasped()is kept as a placeholder until the GT stream carries contact data.suites/xarm7_tabletop.py: regression suite scoring pick/place against GT object poses.Ground truth is privileged scoring data — never consumed by the agent.
Test plan
dimos/evals/test_predicates.py(predicates against a synthetic GT store),dimos/evals/test_mem2_wiring.py(GTRecorder wiring), extendeddimos/evals/test_evals.py(two-arg scorer dispatch, gt_store lifecycle)pytest dimos/evals/— 125 passed locally (Linux, lockfile-pinned MuJoCo 3.10.0)Stacked on #3960.