Skip to content

fix(agent-server): re-read meta.json when loading a persisted conversation - #5777

Draft
enyst wants to merge 3 commits into
mainfrom
claude/umbrella-bug-fixes-tgz9ai-takeover-keeps-metadata
Draft

enyst wants to merge 3 commits into
mainfrom
claude/umbrella-bug-fixes-tgz9ai-takeover-keeps-metadata

Conversation

@enyst

@enyst enyst commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

HUMAN:
This PR proposes a fix for a bug uncovered during the first full agentic run of verify-agent-server.


AGENT:

Why

A second server could take over a conversation using metadata cached at its own startup, then overwrite newer title, tags, secrets and timestamps.

Summary

 resume persisted conversation
   claim conversation lease
-  use startup catalog's meta.json snapshot
+  re-read meta.json after claim, before any write
+  layer request-supplied resume_secrets on that copy
   initialize EventService
   use loaded metadata for catalog / autotitle / telemetry / response

Reload metadata when resuming an existing conversation. Pass the ChatGPT fallback credential separately so it is applied after the reload rather than lost with the stale catalog snapshot.

Source: event_service.py · conversation_service.py.

Issue Number

Fixes #5629

How to Test

Previously recorded verification

Unit tests (new, in tests/agent_server/test_conversation_service.py; the first two fail on main):

  • test_second_service_load_keeps_metadata_written_after_its_startup: a second ConversationService started before the owner renames and retags; after the owner releases, the second one's load keeps title, tags and updated_at in meta.json and get_conversation.
  • test_load_with_unreadable_meta_fails_without_overwriting_it: on main the load overwrites a corrupt meta.json; now it raises and leaves the file.
  • test_load_layers_resume_secrets_over_reread_meta.

uv run pytest tests/agent_server/test_conversation_service.py tests/agent_server/telemetry 399 passed; tests/agent_server otherwise green here except test_canvas_extension_backend.py::test_exited_leader_descendant_does_not_block_recovery[status], which fails the same way on main in this sandbox. pre-commit passes on the changed files.

Live before/after, the issue's Steps to Reproduce on two real python -m openhands.agent_server processes sharing one OH_CONVERSATIONS_PATH (script run against a main checkout and this branch):

== main, graceful stop of A
meta.json before stop: {"title":"after B","updated_at":"2026-10-13T22:35:01.225747Z"}
B events/count: 200
lease: {"generation":1,"owner_pid":17187} (B pid 17187)
meta.json after B load: {"title":"before B","updated_at":"2026-10-13T22:34:54.126480Z"}
B GET title: "before B"
== main, kill -9 A (takeover)
meta.json before stop: {"title":"after B","updated_at":"2026-10-13T22:35:15.905331Z"}
lease: {"generation":2,"owner_pid":17327} (B pid 17327)
meta.json after B load: {"title":"before B","updated_at":"2026-10-13T22:35:08.820041Z"}
B GET title: "before B"
== this branch, graceful stop of A
meta.json before stop: {"title":"after B","updated_at":"2026-10-13T22:35:30.466990Z"}
lease: {"generation":1,"owner_pid":17564} (B pid 17564)
meta.json after B load: {"title":"after B","updated_at":"2026-10-13T22:35:30.466990Z"}
B GET title: "after B"
== this branch, kill -9 A (takeover)
meta.json before stop: {"title":"after B","updated_at":"2026-10-13T22:35:46.280397Z"}
lease: {"generation":2,"owner_pid":17849} (B pid 17849)
meta.json after B load: {"title":"after B","updated_at":"2026-10-13T22:35:46.280397Z"}
B GET title: "after B"

Map proof (verify-agent-server, real processes via launch --new --share-conversations-from and restart --hard):

  • main (a430c77): control-agent-server map run --file F31 --only F31.takeover-keeps-metadata --fresh --repeat 2 --checkout <main checkout> reports xfail in both attempts.
  • this branch, same command without --checkout: xpass in both attempts.
  • after removing the markers: control-agent-server map check ok, and map run --file F31 --only F31.takeover-keeps-metadata --fresh reports pass.

The reproduction is local processes on Linux, not a Kubernetes/NFS deployment; the shared directory and lease file are the same mechanism.

Video/Screenshots

n/a

Type

  • Bug fix
  • Feature
  • Refactor
  • Breaking change
  • Docs / chore

Notes

A missing metadata file keeps the catalog fallback; an unreadable file fails startup and is left untouched. No metadata schema, REST shape, lease protocol or Docker refresh change. Creation-fixed runtime and credential-binding fields still come from the catalog. Independent comparison: #5745.

Visual explanation by Codex using HumanLayer show-me.


🐳 Agent Server images for this PR — GHCR package, pull/run commands, and all pushed tags (click to expand)

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant Architectures Base Image Docs / Tags
java amd64, arm64 eclipse-temurin:17-jdk Link
python-slim amd64, arm64 python-node-runtime Link
python-minimal amd64, arm64 python-node-runtime Link
python amd64, arm64 python-node-runtime Link
golang amd64, arm64 golang:1.21-bookworm Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:0ccf60b-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-0ccf60b-python \
  ghcr.io/openhands/agent-server:0ccf60b-python

All tags pushed for this build

ghcr.io/openhands/agent-server:0ccf60b-golang-amd64
ghcr.io/openhands/agent-server:0ccf60b337395bb39bec0bb3bbc3fbcf4dd94e11-golang-amd64
ghcr.io/openhands/agent-server:claude-umbrella-bug-fixes-tgz9ai-takeover-keeps-metadata-golang-amd64
ghcr.io/openhands/agent-server:0ccf60b-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:0ccf60b-golang-arm64
ghcr.io/openhands/agent-server:0ccf60b337395bb39bec0bb3bbc3fbcf4dd94e11-golang-arm64
ghcr.io/openhands/agent-server:claude-umbrella-bug-fixes-tgz9ai-takeover-keeps-metadata-golang-arm64
ghcr.io/openhands/agent-server:0ccf60b-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:0ccf60b-java-amd64
ghcr.io/openhands/agent-server:0ccf60b337395bb39bec0bb3bbc3fbcf4dd94e11-java-amd64
ghcr.io/openhands/agent-server:claude-umbrella-bug-fixes-tgz9ai-takeover-keeps-metadata-java-amd64
ghcr.io/openhands/agent-server:0ccf60b-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:0ccf60b-java-arm64
ghcr.io/openhands/agent-server:0ccf60b337395bb39bec0bb3bbc3fbcf4dd94e11-java-arm64
ghcr.io/openhands/agent-server:claude-umbrella-bug-fixes-tgz9ai-takeover-keeps-metadata-java-arm64
ghcr.io/openhands/agent-server:0ccf60b-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:0ccf60b-python-amd64
ghcr.io/openhands/agent-server:0ccf60b337395bb39bec0bb3bbc3fbcf4dd94e11-python-amd64
ghcr.io/openhands/agent-server:claude-umbrella-bug-fixes-tgz9ai-takeover-keeps-metadata-python-amd64
ghcr.io/openhands/agent-server:0ccf60b-python-node-runtime-amd64
ghcr.io/openhands/agent-server:0ccf60b-python-arm64
ghcr.io/openhands/agent-server:0ccf60b337395bb39bec0bb3bbc3fbcf4dd94e11-python-arm64
ghcr.io/openhands/agent-server:claude-umbrella-bug-fixes-tgz9ai-takeover-keeps-metadata-python-arm64
ghcr.io/openhands/agent-server:0ccf60b-python-node-runtime-arm64
ghcr.io/openhands/agent-server:0ccf60b-python-minimal-amd64
ghcr.io/openhands/agent-server:0ccf60b337395bb39bec0bb3bbc3fbcf4dd94e11-python-minimal-amd64
ghcr.io/openhands/agent-server:claude-umbrella-bug-fixes-tgz9ai-takeover-keeps-metadata-python-minimal-amd64
ghcr.io/openhands/agent-server:0ccf60b-python-node-runtime-minimal-amd64
ghcr.io/openhands/agent-server:0ccf60b-python-minimal-arm64
ghcr.io/openhands/agent-server:0ccf60b337395bb39bec0bb3bbc3fbcf4dd94e11-python-minimal-arm64
ghcr.io/openhands/agent-server:claude-umbrella-bug-fixes-tgz9ai-takeover-keeps-metadata-python-minimal-arm64
ghcr.io/openhands/agent-server:0ccf60b-python-node-runtime-minimal-arm64
ghcr.io/openhands/agent-server:0ccf60b-python-slim-amd64
ghcr.io/openhands/agent-server:0ccf60b337395bb39bec0bb3bbc3fbcf4dd94e11-python-slim-amd64
ghcr.io/openhands/agent-server:claude-umbrella-bug-fixes-tgz9ai-takeover-keeps-metadata-python-slim-amd64
ghcr.io/openhands/agent-server:0ccf60b-python-node-runtime-slim-amd64
ghcr.io/openhands/agent-server:0ccf60b-python-slim-arm64
ghcr.io/openhands/agent-server:0ccf60b337395bb39bec0bb3bbc3fbcf4dd94e11-python-slim-arm64
ghcr.io/openhands/agent-server:claude-umbrella-bug-fixes-tgz9ai-takeover-keeps-metadata-python-slim-arm64
ghcr.io/openhands/agent-server:0ccf60b-python-node-runtime-slim-arm64
ghcr.io/openhands/agent-server:0ccf60b-golang
ghcr.io/openhands/agent-server:0ccf60b337395bb39bec0bb3bbc3fbcf4dd94e11-golang
ghcr.io/openhands/agent-server:claude-umbrella-bug-fixes-tgz9ai-takeover-keeps-metadata-golang
ghcr.io/openhands/agent-server:0ccf60b-golang_tag_1.21-bookworm
ghcr.io/openhands/agent-server:0ccf60b-java
ghcr.io/openhands/agent-server:0ccf60b337395bb39bec0bb3bbc3fbcf4dd94e11-java
ghcr.io/openhands/agent-server:claude-umbrella-bug-fixes-tgz9ai-takeover-keeps-metadata-java
ghcr.io/openhands/agent-server:0ccf60b-eclipse-temurin_tag_17-jdk
ghcr.io/openhands/agent-server:0ccf60b-python-minimal
ghcr.io/openhands/agent-server:0ccf60b337395bb39bec0bb3bbc3fbcf4dd94e11-python-minimal
ghcr.io/openhands/agent-server:claude-umbrella-bug-fixes-tgz9ai-takeover-keeps-metadata-python-minimal
ghcr.io/openhands/agent-server:0ccf60b-python-node-runtime-minimal
ghcr.io/openhands/agent-server:0ccf60b-python-slim
ghcr.io/openhands/agent-server:0ccf60b337395bb39bec0bb3bbc3fbcf4dd94e11-python-slim
ghcr.io/openhands/agent-server:claude-umbrella-bug-fixes-tgz9ai-takeover-keeps-metadata-python-slim
ghcr.io/openhands/agent-server:0ccf60b-python-node-runtime-slim
ghcr.io/openhands/agent-server:0ccf60b-python
ghcr.io/openhands/agent-server:0ccf60b337395bb39bec0bb3bbc3fbcf4dd94e11-python
ghcr.io/openhands/agent-server:claude-umbrella-bug-fixes-tgz9ai-takeover-keeps-metadata-python
ghcr.io/openhands/agent-server:0ccf60b-python-node-runtime

About Multi-Architecture Support

  • Each variant tag (e.g., 0ccf60b-python) is a multi-arch manifest supporting both amd64 and arm64
  • Docker automatically pulls the correct architecture for your platform
  • Individual architecture tags (e.g., 0ccf60b-python-amd64) are also available if needed

Jev-Fast-Audit

⚡ Jev fast audit · estimates · 0.43s · commit 86ddad7
Strongest signal: No primary concern selected.
Evidence: No primary concern to locate.
Coverage: complete supplied coverage; 14/14 hunks, 4/4 files.

All estimates and evidence
Estimate Likelihood / value Direct evidence
SQL injection 3.0% No direct hunk selected
Command injection 4.0% No direct hunk selected
Weakened authentication 8.0% No direct hunk selected
Weakened authorization 11.0% No direct hunk selected
Contract regression 13.0% No direct hunk selected
Data loss 14.0% F003H003 · openhands-agent-server/openhands/agent_server/event_service.py:1167–1180
Sensitive data disclosure 11.0% No direct hunk selected
Unexpected data transfer 4.0% No direct hunk selected
Credential misuse 9.0% No direct hunk selected
Untrusted instruction authority 4.0% No direct hunk selected
Package source redirection 4.0% No direct hunk selected
Unverified remote execution 2.0% No direct hunk selected
Privileged environment access 8.0% F003H003 · openhands-agent-server/openhands/agent_server/event_service.py:1167–1180
Security assessment bypass 5.0% No direct hunk selected
Prohibited workload 2.0% No direct hunk selected
Primary concern None selected; confidence 67.0% No primary concern to locate

…ation

A server reads every meta.json once at startup. When it later loads a
conversation (takeover from a dead owner, or a claim after the owner
released it on shared storage) it built the EventService from that
startup copy and saved it straight back, reverting the title, tags,
stored secrets and updated_at the previous owner wrote in between.

A resume now re-reads meta.json in EventService.start() right after the
lease claim and before any write, so the catalog record, save_meta(),
the autotitle check and the returned ConversationInfo all use what is on
disk. The ChatGPT auth fallback a resume request supplies is passed as
resume_secrets and layered on top, instead of being patched into the
catalog record. New conversations and forks are unchanged.

The F31.takeover-keeps-metadata map bullet loses its known-bug markers.

Fixes #5629

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014MKndVj7XWvRQmvW9w2bDg
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014MKndVj7XWvRQmvW9w2bDg
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

REST API breakage checks (OpenAPI) — ✅ PASSED

Result: ✅ PASSED

Action log

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Coverage

Coverage Report •
FileStmtsMissCoverMissing
openhands-agent-server/openhands/agent_server
   conversation_service.py126311991%187–188, 270, 280–281, 285–286, 291, 461–462, 523, 628, 635–636, 737, 828, 879–880, 887, 919–920, 936, 967, 971, 983, 1003, 1024–1025, 1034, 1103, 1113, 1142, 1148–1149, 1153–1154, 1162, 1189, 1195, 1289, 1295, 1300, 1306, 1314–1315, 1324–1327, 1336, 1349, 1367, 1404, 1408–1412, 1435, 1476, 1489, 1497, 1555–1556, 1560, 1762, 1786–1788, 1790–1791, 1794–1795, 1837, 1953, 1956, 1963–1965, 1968–1969, 1973–1975, 1978–1979, 1983–1985, 1988–1989, 2018, 2027, 2086–2088, 2119, 2247, 2250, 2277, 2287, 2292–2295, 2309, 2320, 2332–2333, 2365, 2463, 2529, 2587, 2602–2603, 3009, 3062, 3065, 3078
   event_service.py109010990%162, 168–169, 437, 441, 446, 473, 480, 515, 518–519, 523–524, 535, 541, 669–670, 791, 811, 838, 862–863, 867, 915, 950, 954–955, 990, 997, 1003, 1063, 1066, 1208, 1238, 1241, 1327, 1345, 1359, 1527–1528, 1582, 1616, 1763–1766, 1770, 1813, 1817, 1824, 1838, 1852, 1876, 1923–1925, 2009, 2031, 2034–2035, 2060, 2066, 2068, 2084, 2086, 2093, 2101, 2103–2104, 2108, 2156, 2162, 2171–2172, 2178–2181, 2189–2192, 2303–2304, 2306–2313, 2315–2316, 2325–2326, 2328–2329, 2337, 2353–2354, 2365, 2392, 2398, 2404, 2413–2414, 2436, 2446
TOTAL516552047160% 

@enyst
enyst marked this pull request as ready for review October 13, 2026 22:58
@enyst
enyst requested a review from all-hands-bot October 13, 2026 22:59

@all-hands-bot all-hands-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review was posted by an AI agent (OpenHands).

Verdict: correct, well-scoped fix

The change addresses a real shared-storage bug: a second agent server builds an EventService from the catalog copy of meta.json it read at its own startup and, on first load, saves that stale copy back. Adding reload_meta (set for every load that is not a new conversation or a fork) so start() re-reads meta.json after the lease claim and before any write is the minimal, targeted correction. The reload happens under the lease, so newer metadata from the previous owner is preserved rather than reverted, and a subsequent lease loss still fails closed through guarded_write.

What I verified against the current head

  • Reload placement and failure path. event_service.start() re-reads only when meta.json exists; a missing file keeps the catalog copy, and an unreadable file raises out of load_meta() before any write, so the existing startup-failure cleanup closes the service and leaves the file intact. test_load_with_unreadable_meta_fails_without_overwriting_it exercises exactly that.
  • Downstream consumers use the re-read copy. _start_event_service now branches autotitle on event_service.stored (post-reload) and passes the same loaded copy into _maybe_subscribe_telemetry, and the catalog record / returned ConversationInfo are taken from event_service.stored. So title, tags, updated_at, and secrets written by the owner survive in both meta.json and GET.
  • resume_secrets carries the ChatGPT fallback. The previous in-place mutation of existing_record.stored (and its finally restore) is replaced by layering resume_secrets on top of the re-read secrets inside start(). This preserves the original effective behavior: on the resume path _scrub_persisted_credentials pops binding-managed secret names from the merged map regardless of source, so the fallback is accepted when unscrubbed and dropped when a credential binding owns it. test_load_layers_resume_secrets_over_reread_meta covers the layering.
  • Scope of reload_meta. It is not is_new_conversation, so create and fork paths (is_new_conversation=True) keep the caller-supplied stored and never re-read; only genuine loads re-read. No schema, REST, lease, or Docker-runtime change.

Checks

  • Focused tests pass on the head: tests/agent_server/test_conversation_service.py (134 passed, including the 3 new tests) and tests/agent_server/test_event_service.py (118 passed).
  • pre-commit (ruff format/lint, pycodestyle, pyright, import-dependency and registration checks) passes on the three changed Python files.
  • Current-head CI is green on the reported check runs (agent-server-tests, sdk-tests, coverage-report, and the REST/OpenAPI breakage check reported PASSED).
  • The PR supplies the requested live before/after evidence on two real python -m openhands.agent_server processes sharing one OH_CONVERSATIONS_PATH, plus the F31.takeover-keeps-metadata map xfail->xpass->pass transition, for both graceful-stop and SIGKILL takeover. The reproduction is local processes rather than a Kubernetes/NFS deployment, and the PR states that limitation; the shared-directory and lease mechanism is the same one the change relies on.

No material correctness, security, compatibility, or acceptance-criterion defect found. The linked issue (#5629) is open with ready-for-dev.

✅ APPROVED

@enyst enyst left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To note, I'm not sure this should happen: have two servers running on the same conversations directory - or rather, if it does happen, it's not supported. It's possible that the agent saw the leasing mechanism and thought it's a supported use case; or that it has become a supported use case with docker runtimes; or that I'm wrong and misremember when I think the leasing addresses the same agent-server bad restart.

I'd prefer we don't merge this until we look at bit into it. The PR is not big, but the code design around meta.json is not always great... smells of unnecessary complexity.

…eps-metadata

Co-authored-by: openhands <openhands@all-hands.dev>
@github-actions

Copy link
Copy Markdown
Contributor

📁 PR Artifacts Notice

This PR contains a .pr/ directory with temporary PR-specific documents. The directory will be automatically removed after this PR merges to main through a cleanup PR.

@enyst
enyst marked this pull request as draft October 26, 2026 12:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: A second server's first load of a shared conversation reverts meta.json to its stale startup copy

3 participants