Skip to content

fix: enforce per-request deadlines in child evaluation - #227

Open
Dnyanesh-29 wants to merge 1 commit into
accordproject:mainfrom
Dnyanesh-29:codex/evaluator-per-request-timeout
Open

Dnyanesh-29 wants to merge 1 commit into
accordproject:mainfrom
Dnyanesh-29:codex/evaluator-per-request-timeout

Conversation

@Dnyanesh-29

@Dnyanesh-29 Dnyanesh-29 commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Enforce the original per-request deadline when dispatching queued child-process evaluations.
  • Derive worker execution timeouts from the selected request's remaining budget rather than the queue-processing caller's options.
  • Prevent expired requests from being dispatched after child-process startup and reject late successful completions.

Problem

evalChildProcess() records an absolute expiration deadline for each WorkItem, but the worker previously received the full timeout supplied to processQueue().

This allowed a queued request to exceed its original deadline and, under concurrent queue polling, potentially receive another request's timeout.

The original behavior was reproduced with real child processes and deterministic scheduler tests before implementing the correction.

Fix

The evaluator now calculates the remaining time from the selected WorkItem's deadline at dispatch.

It checks expiration before launching the child and again before sending work, and it prevents successful settlement after the original deadline.

Existing worker cleanup and close-based settlement behavior are preserved.

Tests

Added six focused regression tests covering:

  • Queue waiting consuming the original deadline.
  • Timeout ownership across requests with different timeout values.
  • Both directions of timeout mismatch.
  • Expiration before launch.
  • Expiration during execution.
  • Expiration during child-process startup.

Validation:

  • 199 unit tests passed across 19 suites.
  • 46 snapshots passed without changes.
  • 6 Chromium tests passed separately.
  • TypeScript, lint, license, webpack build, and whitespace checks passed.

No public API, dependency, model, or snapshot changes.

Deadline semantics

The timeout covers the request from enqueue until the parent observes a successful worker close. An IPC result received before the deadline is not sufficient for success if the worker closes at or after the deadline.

The evaluator retains close-based settlement to confirm the worker's exit status and complete worker-slot cleanup.

Related work reviewed

This PR preserves #200's close-based settlement while enforcing each selected WorkItem's own deadline. No existing issue was identified that describes the two timeout defects exactly.

AI assistance

OpenAI Codex assisted with repository investigation, reproduction, implementation, regression testing, and code review. The changes and reported test results were reviewed before submission.

Signed-off-by: Dnyanesh-29 <dnyaneshbharambe2006@gmail.com>
@Dnyanesh-29
Dnyanesh-29 requested review from a team and a balanced review from Copilot November 15, 2026 10:03

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@mira-accord-project

Copy link
Copy Markdown
Contributor

Thank you for the pull request. It enforces each request's original deadline when queued child-process evaluations are dispatched, and it adds regression tests for the timeout cases. We have requested review from Dan Selman, a template-engine maintainer, who will reply here.


Posted automatically by Mira, an Accord Project agent. A maintainer will follow up; they make the decisions on this.

@mira-accord-project mira-accord-project Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for this fix. Each queued request now keeps its own deadline: the worker timeout is the time remaining on that request, and expiry is checked before the fork and again before sending work. The options parameter that let one request borrow another's timeout is gone from processQueue and doWork, and both are private, so there is no public API change. The new tests use fake timers and would fail if the change were reverted. I found nothing blocking, only two optional points below.

Nothing blocking; some suggestions below. Findings: 2 nit.

  • nit · conventions: AGENTS.md asks the PR description to say whether an AI tool helped, and which earlier discussions on the touched file were read. The description covers the tests you ran but not these two points, and it does not link an issue. Earlier work on this file includes #200 (settle on close, not exit) and #180. A line saying whether a tool helped, and how this change fits with #200's close-based settlement, would help reviewers.

Automated review by Mira, an Accord Project agent, at ef57fed. Findings are labelled blocking, should-fix or nit; a maintainer makes the final call.

Comment thread src/JavaScriptEvaluator.ts Resolved
@Dnyanesh-29

Copy link
Copy Markdown
Contributor Author

Thanks for pointing this out. You're right that a result received before the deadline can be followed by a late close.

The current implementation intentionally uses successful worker close as the completion boundary for the request. The message handler collects the result, while close confirms successful termination, releases the worker slot, and settles the promise. The existing fake-clock regression covers an on-time result followed by a late close and expects a timeout.

I agree that measuring at message receipt would be a valid alternative if the intended contract were result availability rather than completion of the worker lifecycle. For this fix, I'd prefer to retain the stricter close-time deadline semantics and document that choice explicitly.

Happy to adjust if the maintainers prefer message-time completion.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants