isaac_harness: build a self-loop (autonomous spawn/check/adjust) on top of structured scene introspection

[wilsonchenghy]

Context

tools/isaac_harness/ (the persistent Isaac Sim session daemon + structured scene
introspection: bbox/overlap/distance/scene_tree) currently requires a human (or
Claude, driven step-by-step) to manually iterate: spawn something, check it, decide
what to adjust, repeat. That works but doesn't scale — building a scene correctly
tonight took many manual round trips of "guess a value, screenshot, eyeball, guess
again."

The architecture already matches the pattern used in production robotics/agent
tooling — a persistent sim server exposed over an API, with structured numeric
feedback as the primary signal (not vision) — but is missing the actual self-loop
layer: given a target condition expressed in bbox/overlap/distance terms, spawn →
check → adjust → recheck automatically until it passes or a bound is hit, without a
human in the loop for every step.

Work items

1. Fix bbox staleness for moving dynamic objects properly, not just flag it.
   Currently bbox reads raw USD stage transforms, which don't reliably sync from the
   physics tensor/Fabric layer — for an object that's actively moving under physics,
   bbox can silently return a stale spawn-time position. The daemon currently just
   attaches a "warning" field to the response for tracked dynamic objects rather than
   solving it (a first attempt — reconstructing world bbox from a cached local bbox +
   live tracked pose — produced numbers that didn't match query()'s known-good pose
   either, and was reverted rather than ship something subtly wrong in a different
   way). A self-loop is only as good as the numbers it trusts, so this needs to be
   solid before building on top of it.
2. Build the loop primitive itself: a way to express "spawn/adjust until
   bbox/overlap/distance satisfies condition X, bounded by N iterations or a
   timeout," reusable by both Claude and any future automated pipeline.
3. Fix the remove + other-tracked-Articulation corruption bug: removing an
   object via remove (which calls stage.RemovePrim() + sim.reset()) was observed
   to invalidate the physics view of an already-spawned, untouched Articulation
   (ReferenceError: weakly-referenced object no longer exists / Failed to get DOF velocities from backend on the next step/spawn call). This needs to be
   understood and fixed before a self-loop can safely remove-and-retry objects, which
   is a natural pattern for iterative scene correction.

See .claude/skills/isaac-harness/SKILL.md for the full context on how the harness
works and the gotchas already found building it.

Update: another sim.reset()-related bug found (2026-08-25, cabinet task prototype)

4. set_pose changes are silently discarded by the next spawn. Every spawn_primitive
   / spawn_usd call ends with sim.reset(). That reset re-applies every tracked
   Articulation's original spawn-time init_state (pos/rot), which wipes out any
   runtime set_pose change made since — with no error, no warning, nothing. Confirmed
   directly: rotated a cabinet with set_pose, query() showed the new rotation
   immediately and held through several step calls, then spawning one more (unrelated,
   throwaway) object made query() show the original identity rotation again.
   This cost real time — two screenshots in a row looked "unchanged" after an
   apparently-successful set_pose, and it wasn't obvious why until isolated with a
   throwaway spawn + immediate re-query. Workaround for now: bake any needed pose
   directly into the spawn_usd/spawn_primitive call's pos/rot args, and do all
   your spawning before using set_pose on anything, not after. Real fix would be
   either making sim.reset() preserve current runtime state instead of reverting to
   init_state, or having set_pose also update the tracked object's init_state so
   a later reset doesn't discard it.

Update 2 (2026-08-25): remove-corruption fix attempted and reverted, set_pose crash fixed, persistence bug still open

• Item 3 (remove corrupts other Articulations): fix attempted, reverted — too risky.
  Forcing every other tracked object to reacquire its PhysX view
  (obj._is_initialized = False; obj._initialize_impl()) after any sim.reset()
  worked in initial testing, but under slightly different conditions caused a much
  worse failure: a genuine PhysX/CUDA "illegal memory access" that aborted the whole
  process with a core dump, not a recoverable Python exception. Reverted rather than
  ship something that trades a recoverable error for an unrecoverable one. The
  original bug (documented in Update 1) is still present and still just a documented
  workaround (avoid remove while other Articulations are tracked; do a full daemon
  restart instead). A real fix needs to understand why IsaacLab's own timeline
  PLAY/STOP-event re-initialization mechanism doesn't reliably refire for an
  already-initialized object under this daemon's repeated multi-reset usage pattern,
  not just force it externally.

• New bug found and fixed: set_pose on a kinematic object crashed the whole
  process. write_root_pose_to_sim() isn't a valid path for a kinematic body and
  triggered a genuine CUDA "illegal memory access," confirmed reproducible on a fresh
  daemon with nothing else going on (ruling out GPU state degraded by unrelated
  churn, which was the first suspicion). Fixed: set_pose now writes a kinematic
  object's pose directly to its USD prim transform instead. Caveat: query() won't
  see this change (it reads the PhysX tensor view) — use bbox() to verify a
  kinematic set_pose instead, confirmed correct.

• Item still fully open: set_pose changes are silently wiped by the next
  spawn/remove's sim.reset(). Root cause identified precisely — sim.reset()
  re-applies each object's data.default_root_state tensor, cached once at spawn
  time and never touched again. Two fix attempts both failed silently (no crash,
  the pose just still reverted): mutating obj.cfg.init_state (no-op, cfg isn't
  re-read after spawn) and writing directly into data.default_root_state (still
  didn't survive a later reset — why not is not yet understood). Workaround remains:
  bake pose into the spawn call itself, don't set_pose before further spawning.

[wilsonchenghy]

Update: YCB / bare-mesh USD physics spawning — fixed (2026-08-25)

Root cause found. spawn_usd's previous --apply-physics attempt (merged then
reverted in #208) passed rigid_props/collision_props/mass_props into
UsdFileCfg, which routes through isaaclab.sim.schemas.modify_rigid_body_properties()
et al. Those functions only modify an already-applied RigidBodyAPI/CollisionAPI/
MassAPI — for a bare-mesh asset with no physics baked in (e.g. YCB), the schema check
fails and the function just returns False, no exception. So the flag was a silent
no-op at spawn time; the real failure ("Failed to find a rigid body... Please ensure
RigidBodyAPI is applied") only surfaced later and confusingly, inside
RigidObject._initialize_impl() at the next sim.reset().

IsaacLab ships a separate define_rigid_body_properties() / define_collision_properties()
/ define_mass_properties() family that applies the schema first, then delegates to
the same modify_*() — that's the step UsdFileCfg's cfg fields never do. spawn_usd --apply-physics now calls these define_* functions directly on the spawned prim
right after creation (before sim.reset()), instead of routing through UsdFileCfg.

Verified end-to-end, not just "no exception": spawned a YCB cracker box at
pos=[0,0,0.3] with --apply-physics, stepped 60 physics steps, and it fell under
real gravity to z≈-1.165 (ground at -1.2 + the box's own half-height) — confirmed
both via query() and a screenshot showing it resting flat on the ground plane.

Also added along the way (useful independent of this fix):

• spawn_visual — spawns a USD reference as a pure visual prim, no
  RigidObject/Articulation wrapper at all. For anything that just needs to be
  looked at (no physics interaction needed), this sidesteps the whole
  RigidBodyAPI question entirely.
• list_dir — lists a Nucleus (or local) directory via omni.client.list(), for
  finding an asset's real filename/path before spawning it instead of guessing.