Skip to content

Bound HTTP body read waits and report transfer progress - #2141

Open
Tyrie Vella (tyrielv) wants to merge 4 commits into
microsoft:masterfrom
tyrielv:tyrielv/response-read-timeout
Open

Tyrie Vella (tyrielv) wants to merge 4 commits into
microsoft:masterfrom
tyrielv:tyrielv/response-read-timeout

Conversation

@tyrielv

@tyrielv Tyrie Vella (tyrielv) commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Problem and Context

An HTTP response can return its headers promptly, then stop delivery of body data. VFS for Git uses ResponseHeadersRead, so the request timeout does not bound subsequent body reads. A stalled clone or prefetch can therefore wait indefinitely and never reach the retry policy.

A long transfer does not necessarily indicate a stall. Existing logs do not distinguish a slow transfer from a pending read without progress.

Changes

  • Add a 60-second deadline to each pending response-body read, including error bodies. Exclude disk writes and pack processing.
  • Add gvfs.read-timeout-seconds. Zero disables the body-read deadline. Caller cancellation remains active.
  • Load the complete retry configuration through LoadFromGitConfig in every production consumer. Reuse the configuration during authentication and download initialization.
  • Warn and use per-field defaults for invalid or unreadable configuration. Preserve valid fields, active mount snapshots, and explicit command overrides.
  • Preserve known HTTP statuses, credential rejection, cache fallback, and cancellation through object-copy and retry paths. Keep the fixed anonymous 401 message.
  • Emit progress reports every 30 seconds after the first body read. Include byte progress, pending-read state, current and maximum read waits, bounded cache identity, and final outcomes.
  • Correlate pack events and retry summaries through response and request IDs. Use sample sequence numbers to order delayed reports.
  • Keep diagnostic writes outside the read-state lock. Make body diagnostics best-effort, including unexpected failures. After a diagnostic failure, disable further reports for that response. Attempt one warning. The read result and cleanup remain unchanged.
  • Before retry, wait for pending pack finalization tasks after a failed read. Retain completed pack timestamps and report concurrent finalization failures.
  • Return exit code 12 (CloneSucceededPrefetchFailed) when clone succeeds but synchronous prefetch fails. Print a prefetch-only retry command. A requested mount failure takes precedence and returns its own failure code.
  • Add unit tests for deadlines, slow transfers, zero-disable behavior, numeric boundaries, and configuration consumers. These tests also assert completed-body handling, credential rejection, cancellation, cleanup, diagnostic failures, and continued reads.
  • Add functional tests for stalled clone requests. These tests assert timeout diagnostics, retry recovery, and retry exhaustion through the CLI.
  • Add functional tests for post-clone prefetch failure. These tests assert exit code 12, the recovery command, a valid HEAD, and mount-failure precedence.
  • Add functional tests for a stalled CLI prefetch and invalid retry configuration. These tests assert the configured deadline, timeout logs, continued mount operation, remount success, and startup warnings.
  • Document the deadlines, configuration ranges, fallback behavior, diagnostic fields, prefetch recovery, and ownership of offloaded prefetch.

The read-wait measurements include managed scheduling delay. They distinguish outstanding reads from consumer work, but they do not prove a server or network cause.

Known gaps and deliberate decisions

  • Whole-clone recovery — Network steps retain per-request retries. This change does not add whole-clone retries or same-directory restart. Those features need failure classification, compatible-option checks, and protection for existing work. No follow-up PR is linked.
  • Background prefetch — gvfs.trust-pack-indexes=false keeps its existing background behavior. Exit 0 confirms process launch, not prefetch completion.

@tyrielv
Tyrie Vella (tyrielv) force-pushed the tyrielv/response-read-timeout branch 2 times, most recently from 3058dd6 to 2ac6953 Compare October 21, 2026 18:05
@tyrielv
Tyrie Vella (tyrielv) marked this pull request as ready for review October 21, 2026 18:25
@YingyHuang

Copy link
Copy Markdown

[Claude] Clone still has no retry at the operation level; only individual requests retry (MEDIUM)

File: GVFS/GVFS/CommandLine/CloneVerb.cs (not in diff) Lines: TryClone 480–537, CreateClone 685–694, 725–728 (master)

Problem:

This PR stops stalled body reads from hanging, which is a real improvement. But the only retries are per request: a stall is retried up to gvfs.max-retries times, then reported as a failure. Clone has no retry of its own. If the network is unstable for longer than one request's retry budget, for example several 60s stalls on the commit pack or on info/refs, gvfs clone fails. The user then has to clean up the partial enlistment and run clone again, which is exactly the transient-failure experience this PR is meant to improve.

Evidence:

  • CloneVerb.Execute → GetRetryConfig(..., FetchAndCloneTimeoutMinutes) → TryAuthenticateAndQueryGVFSConfig → TryClone → QueryInfoRefs / TryDownloadCommit / TryDownloadRootGitAttributes. Each step returns new Result(errorMessage) on failure, and nothing loops around them.

  • The only retries are in RetryWrapper.Invoke (IsHandlableException: IOException, HttpRequestException, RetryableException) and in GitObjects.TrySendProtocolRequest / HandleProtocolException.

  • The new functional test (PrefetchVerbTests) covers a stalled prefetch. No test covers a stalled or transient clone.

Suggestion:

Add a bounded retry around the network-bound clone steps. Retry only transient failures, and leave auth failures, missing branches and filesystem errors alone. For example:

C#

// CloneVerb.TryClone
const int MaxCloneStepAttempts = 3;
GitRefs refs = null;
for (int attempt = 1; attempt <= MaxCloneStepAttempts && refs == null; attempt++)
{
    refs = objectRequestor.QueryInfoRefs(this.SingleBranch ? this.Branch : null);
    if (refs == null && attempt < MaxCloneStepAttempts)
    {
        tracer.RelatedWarning(
            new EventMetadata { { "Attempt", attempt } },
            "info/refs query failed after per-request retries; retrying clone step");
        Thread.Sleep(TimeSpan.FromSeconds(RetryBackoff.CalculateBackoffSeconds(attempt + 1, 60, 2)));
    }
}

Apply the same pattern to TryDownloadCommit in CreateClone. Alternatively, make clone restartable in the same directory so re-running gvfs clone resumes. At a minimum, add a clone functional test with a stalled or erroring server and a line in docs/troubleshooting.md saying that gvfs.max-retries and gvfs.read-timeout-seconds apply to clone.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No significant introduced correctness issue found in the reviewed code. Holding approval pending integration validation.

Medium - Merge readiness: This head currently conflicts with master. Please resolve or rebase and rerun validation against the current target before merge; the existing green run does not establish that the conflict resolution is correct.

Medium (nonblocking) - Differential coverage: The exact-head CI run passes, including the stalled-prefetch deadline and mounted invalid-config tests, but I could not find a changed-line coverage report. Please link differential coverage for the new timeout, cancellation, ownership, and retry branches, or document the remaining coverage gaps in the PR description. This is nonblocking, not evidence that these lines are uncovered.

Stop pending response-body reads after 60 seconds without data. Preserve known error statuses, existing retries and cache fallback, and caller cancellation. Zero disables the read deadline.

Load complete retry configuration through one entry point in every production consumer. Warn and use per-setting defaults for invalid or unreadable values instead of aborting. Preserve valid settings, active mount snapshots, and intentional command overrides.

Report byte progress and read waits in local logs and telemetry. Correlate pack events and retry outcomes with response and request IDs. Number diagnostic samples and keep diagnostic writes outside the read-state lock. Guarantee response cleanup when diagnostics fail.

Drain pending pack finalization work before retry after a failed read. Retain completed timestamps for the next request.

Add unit tests for complete config loading, fallback, transport deadlines, cleanup, and diagnostic contracts. Add functional tests for a stalled CLI prefetch and invalid retry settings on a mounted enlistment. Document the deadline, configuration fallback, and diagnostic fields.

Assisted-by: GPT-6.1 Sol
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Dispose prepared responses when final diagnostics throw. Keep the fixed anonymous 401 message and tolerate expected tracer teardown during body diagnostics.

Preserve caller cancellation through stream copies and temporary-pack writes. Correlate fast pack transfers with retry summaries and report concurrent pack finalization failures.

Return complete retry configuration from LoadFromGitConfig. Reuse the configuration during command authentication and retain per-field defaults and command overrides.

Add unit tests for response ownership, completed error bodies, credentials, cancellation, configuration consumers, numeric boundaries, and diagnostic ordering.

Extend functional tests to assert startup warnings after remount with invalid configuration. These tests disable prefetch offload and restore the original configuration.

Document progress timing, configuration ranges, and offloaded prefetch ownership.

Assisted-by: GPT-6.1 Sol
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
@tyrielv
Tyrie Vella (tyrielv) force-pushed the tyrielv/response-read-timeout branch from d035eaf to 847ce53 Compare October 28, 2026 22:35
Disable further response-body reports after an unexpected diagnostic failure. Attempt one warning through the tracer, then standard error if the tracer fails.

Keep the read deadline, caller cancellation, and response cleanup active. Preserve the original read error when final diagnostics fail.

Add unit tests for periodic and final diagnostic failures, report construction, warning fallback, listener failures, and continued reads.

Assisted-by: GPT-6.1 Sol
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
Return exit code 12 when clone succeeds but synchronous prefetch fails. Print a command that retries only prefetch. Preserve the failure code when a requested mount fails.

Keep per-request retry budgets and the existing background-prefetch behavior.

Add functional tests for stalled clone requests, retry recovery and exhaustion, the partial-success result, and mount-failure precedence.

Document clone retry configuration, prefetch recovery, and best-effort body diagnostics.

Assisted-by: GPT-6.1 Sol
Signed-off-by: Tyrie Vella <tyrielv@gmail.com>
@tyrielv

Copy link
Copy Markdown
Contributor Author

Note

🤖 Machine-drafted reply, reviewed and approved by Tyrie Vella.

Clone's network steps already use the request retry budget. I kept that budget. An outer loop multiplies the retry budget.

The new functional tests exercise a stalled info/refs request through gvfs clone, including recovery and retry exhaustion. If clone succeeds but synchronous prefetch fails, exit code 12 distinguishes that result. The command prints a prefetch-only retry command. A mount failure takes precedence.

Whole-clone retry and same-directory restart remain outside this change because they need failure classification and safeguards for an existing enlistment.

Comment on lines +227 to +232
new HttpResponseStream(
responseStream,
this.RetryConfig.ReadTimeout,
cancellationToken,
this.Tracer,
bodyMetadata),

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[KeithBot]: 🟡 Medium · Reliability — Cancelled hydration aborts shared blob download; other ProjFS reads can hang

Body reads now honor the request's CancellationToken, but GVFSGitObjects.TryDownloadAndSaveObject runs one shared download per SHA under the first caller's token. Cancelling that caller mid-body (e.g. its process is killed) fails every coalesced waiter: a second ProjFS GetFileData for the same blob is never completed, so that reader hangs, and a joined git read-object request gets DownloadFailed. At the merge base both waiters succeed.

Evidence:
Scratch NUnit fixture: real GVFSGitObjects + GitObjectsHttpRequestor against a loopback server streaming the object in 64-byte chunks every 50 ms, plus the real WindowsFileSystemVirtualizer on MockVirtualizationInstance; GetFileData commands 1 and 2 for one SHA, then OnCancelCommand(1) mid-body.

> GVFS.UnitTests.exe --test=GVFS.UnitTests.Windows.Virtualization.CoalescedCancelRepro
head: RESULT projfs ... command2Completed=False completions=[] cancelledEvents=2
base: RESULT projfs ... command2Completed=True completions=[2:Ok] cancelledEvents=1
head: RESULT direct secondIsNamedPipe=True ... second=EXCEPTION TaskCanceledException
base: RESULT direct secondIsNamedPipe=True ... second=RETURNED Success

Fix:
In GVFSGitObjects.TryDownloadAndSaveObject, restart the download instead of inheriting another caller's cancellation (add before the existing finally):

catch (OperationCanceledException) when (!cancellationToken.IsCancellationRequested)
{
    this.TryRemoveInflightDownload(objectId, lazy);
    return this.TryDownloadAndSaveObject(objectId, cancellationToken, requestSource, retryOnFailure);
}

🤖 KeithBot · 1/3 models · verified locally

@KeithIsSleeping

Copy link
Copy Markdown

[KeithBot]: ⚠️ 1 issue · iteration 1 (14b635f)

  1. 🟡 Medium Cancelled hydration aborts shared blob download; other ProjFS reads can hang

Validated: ✅ PR unit fixtures, head + merge base · ✅ Full GVFS.UnitTests suite at head · ❌ Coalesced cancel repro, head vs base · ⚠️ Loopback transport/pack/culture probes · ✅ Build of all PR-touched projects · ⏭️ CLI functional tests (clone/prefetch)

🤖 KeithBot · claude-opus-5.5, gpt-6.1-sol, gemini-4-argon · reply on a thread to dispute

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants