Repository navigation
Bound HTTP body read waits and report transfer progress - #2141
Tyrie Vella (tyrielv) wants to merge 4 commits into
Conversation
3058dd6 to
2ac6953
Compare
|
[Claude] Clone still has no retry at the operation level; only individual requests retry (MEDIUM) File: Problem: This PR stops stalled body reads from hanging, which is a real improvement. But the only retries are per request: a stall is retried up to Evidence:
Suggestion: Add a bounded retry around the network-bound clone steps. Retry only transient failures, and leave auth failures, missing branches and filesystem errors alone. For example: C# Apply the same pattern to |
Nathan Baird (ShiningMassXAcc)
left a comment
There was a problem hiding this comment.
No significant introduced correctness issue found in the reviewed code. Holding approval pending integration validation.
Medium - Merge readiness: This head currently conflicts with master. Please resolve or rebase and rerun validation against the current target before merge; the existing green run does not establish that the conflict resolution is correct.
Medium (nonblocking) - Differential coverage: The exact-head CI run passes, including the stalled-prefetch deadline and mounted invalid-config tests, but I could not find a changed-line coverage report. Please link differential coverage for the new timeout, cancellation, ownership, and retry branches, or document the remaining coverage gaps in the PR description. This is nonblocking, not evidence that these lines are uncovered.
Stop pending response-body reads after 60 seconds without data. Preserve known error statuses, existing retries and cache fallback, and caller cancellation. Zero disables the read deadline. Load complete retry configuration through one entry point in every production consumer. Warn and use per-setting defaults for invalid or unreadable values instead of aborting. Preserve valid settings, active mount snapshots, and intentional command overrides. Report byte progress and read waits in local logs and telemetry. Correlate pack events and retry outcomes with response and request IDs. Number diagnostic samples and keep diagnostic writes outside the read-state lock. Guarantee response cleanup when diagnostics fail. Drain pending pack finalization work before retry after a failed read. Retain completed timestamps for the next request. Add unit tests for complete config loading, fallback, transport deadlines, cleanup, and diagnostic contracts. Add functional tests for a stalled CLI prefetch and invalid retry settings on a mounted enlistment. Document the deadline, configuration fallback, and diagnostic fields. Assisted-by: GPT-6.1 Sol Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Dispose prepared responses when final diagnostics throw. Keep the fixed anonymous 401 message and tolerate expected tracer teardown during body diagnostics. Preserve caller cancellation through stream copies and temporary-pack writes. Correlate fast pack transfers with retry summaries and report concurrent pack finalization failures. Return complete retry configuration from LoadFromGitConfig. Reuse the configuration during command authentication and retain per-field defaults and command overrides. Add unit tests for response ownership, completed error bodies, credentials, cancellation, configuration consumers, numeric boundaries, and diagnostic ordering. Extend functional tests to assert startup warnings after remount with invalid configuration. These tests disable prefetch offload and restore the original configuration. Document progress timing, configuration ranges, and offloaded prefetch ownership. Assisted-by: GPT-6.1 Sol Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
d035eaf to
847ce53
Compare
Disable further response-body reports after an unexpected diagnostic failure. Attempt one warning through the tracer, then standard error if the tracer fails. Keep the read deadline, caller cancellation, and response cleanup active. Preserve the original read error when final diagnostics fail. Add unit tests for periodic and final diagnostic failures, report construction, warning fallback, listener failures, and continued reads. Assisted-by: GPT-6.1 Sol Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Return exit code 12 when clone succeeds but synchronous prefetch fails. Print a command that retries only prefetch. Preserve the failure code when a requested mount fails. Keep per-request retry budgets and the existing background-prefetch behavior. Add functional tests for stalled clone requests, retry recovery and exhaustion, the partial-success result, and mount-failure precedence. Document clone retry configuration, prefetch recovery, and best-effort body diagnostics. Assisted-by: GPT-6.1 Sol Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
|
Note 🤖 Machine-drafted reply, reviewed and approved by Tyrie Vella. Clone's network steps already use the request retry budget. I kept that budget. An outer loop multiplies the retry budget. The new functional tests exercise a stalled Whole-clone retry and same-directory restart remain outside this change because they need failure classification and safeguards for an existing enlistment. |
| new HttpResponseStream( | ||
| responseStream, | ||
| this.RetryConfig.ReadTimeout, | ||
| cancellationToken, | ||
| this.Tracer, | ||
| bodyMetadata), |
There was a problem hiding this comment.
[KeithBot]: 🟡 Medium · Reliability — Cancelled hydration aborts shared blob download; other ProjFS reads can hang
Body reads now honor the request's CancellationToken, but GVFSGitObjects.TryDownloadAndSaveObject runs one shared download per SHA under the first caller's token. Cancelling that caller mid-body (e.g. its process is killed) fails every coalesced waiter: a second ProjFS GetFileData for the same blob is never completed, so that reader hangs, and a joined git read-object request gets DownloadFailed. At the merge base both waiters succeed.
Evidence:
Scratch NUnit fixture: real GVFSGitObjects + GitObjectsHttpRequestor against a loopback server streaming the object in 64-byte chunks every 50 ms, plus the real WindowsFileSystemVirtualizer on MockVirtualizationInstance; GetFileData commands 1 and 2 for one SHA, then OnCancelCommand(1) mid-body.
> GVFS.UnitTests.exe --test=GVFS.UnitTests.Windows.Virtualization.CoalescedCancelRepro
head: RESULT projfs ... command2Completed=False completions=[] cancelledEvents=2
base: RESULT projfs ... command2Completed=True completions=[2:Ok] cancelledEvents=1
head: RESULT direct secondIsNamedPipe=True ... second=EXCEPTION TaskCanceledException
base: RESULT direct secondIsNamedPipe=True ... second=RETURNED Success
Fix:
In GVFSGitObjects.TryDownloadAndSaveObject, restart the download instead of inheriting another caller's cancellation (add before the existing finally):
catch (OperationCanceledException) when (!cancellationToken.IsCancellationRequested)
{
this.TryRemoveInflightDownload(objectId, lazy);
return this.TryDownloadAndSaveObject(objectId, cancellationToken, requestSource, retryOnFailure);
}🤖 KeithBot · 1/3 models · verified locally
|
[KeithBot]: Validated: ✅ PR unit fixtures, head + merge base · ✅ Full GVFS.UnitTests suite at head · ❌ Coalesced cancel repro, head vs base · 🤖 KeithBot · claude-opus-5.5, gpt-6.1-sol, gemini-4-argon · reply on a thread to dispute |
Problem and Context
An HTTP response can return its headers promptly, then stop delivery of body data. VFS for Git uses
ResponseHeadersRead, so the request timeout does not bound subsequent body reads. A stalled clone or prefetch can therefore wait indefinitely and never reach the retry policy.A long transfer does not necessarily indicate a stall. Existing logs do not distinguish a slow transfer from a pending read without progress.
Changes
gvfs.read-timeout-seconds. Zero disables the body-read deadline. Caller cancellation remains active.LoadFromGitConfigin every production consumer. Reuse the configuration during authentication and download initialization.CloneSucceededPrefetchFailed) when clone succeeds but synchronous prefetch fails. Print a prefetch-only retry command. A requested mount failure takes precedence and returns its own failure code.The read-wait measurements include managed scheduling delay. They distinguish outstanding reads from consumer work, but they do not prove a server or network cause.
Known gaps and deliberate decisions
gvfs.trust-pack-indexes=falsekeeps its existing background behavior. Exit 0 confirms process launch, not prefetch completion.