Skip to content

server: recover panics in the ByteStream Write put goroutine so a decoder panic fails one write, not the process - #66

Merged
rcooney-sh merged 1 commit into
patchsetfrom
devin/1793064894-bytestream-put-recover
Oct 27, 2026
Merged

rcooney-sh merged 1 commit into
patchsetfrom
devin/1793064894-bytestream-put-recover

Conversation

@rcooney-sh

@rcooney-sh rcooney-sh commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #65. The recovery interceptors added there only cover the handler goroutine. ByteStream/Write hands the body to putUpload on its own goroutine via an io.Pipe, and for compressed uploads that is where the zstd decoder runs, so a panic there still exited the process.

Recover in the put goroutine

go func() {
    defer s.releaseCASWriteSlot()
    defer func() {
        if r := recover(); r != nil {
            _ = pr.CloseWithError(errPutPanicked)
            putResult <- recoveredPanicStatus("ByteStream/Write put", r)
        }
    }()
    putResult <- s.putUpload(srv.Context(), hash, size, cmp, pr)
}()

The write fails with codes.Internal, the pipe is closed so the recv loop unblocks, and the CAS write slot is still released.

Test

TestGrpcByteStreamWritePutPanicFailsOnlyThatWrite wraps the disk cache so Put panics, sends a write, expects Internal, and checks the server still answers GetCapabilities. Without the recover it crashes the test binary.

Worth including in v2.6.1-blacksmith.14 if it hasn't been cut yet; otherwise it can ride the next tag, since the klauspost bump in #65 is the actual fix.

Link to Devin session: https://app.devin.ai/sessions/c4fbdf7fdaee45c89654f8496af5720b
Open in Devin Desktop: https://app.devin.ai/desktop/session/c4fbdf7fdaee45c89654f8496af5720b?variant=devin
Requested by: @rcooney-sh


View with [code]smith View full Proof
Need help on this PR? Tag @codesmith-bot with what you need. (Staging)


View with [code]smith QA with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

The recovery interceptors only cover the handler goroutine. ByteStream Write hands the body to putUpload on its own goroutine (where the zstd decoder runs for compressed uploads), so a panic there still exited the process. Recover it, fail just that write with codes.Internal, and close the pipe so the recv loop unblocks.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@blacksmith-staging

blacksmith-staging Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Matches Put panic fails one write, server survives

Before, the panicking zstd upload killed the server and every later call was refused. After, only that write failed with Internal, and later calls and writes still succeeded.

Merge-base

Before: process exits

This PR c2c6630b

After: one write fails

Also checked · 1 match

Matches CAS write slot released after panic

Proof on c2c6630b

View full Proof report

@rcooney-sh
rcooney-sh requested a review from ajwerner October 27, 2026 02:54
@rcooney-sh
rcooney-sh merged commit 7e779d3 into patchset Oct 27, 2026
6 checks passed
devin-ai-integration Bot added a commit that referenced this pull request Oct 27, 2026
Merges patchset (#66) into this branch so the Bazel test target covers every
_test.go in server/ and the release build compiles from the current patchset head.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
shreyas-blacksmith added a commit that referenced this pull request Oct 27, 2026
* server: add grpc_recovery.go and its test to BUILD.bazel

* server: list grpc_bytestream_put_panic_test.go in the Bazel go_test too

Merges patchset (#66) into this branch so the Bazel test target covers every
_test.go in server/ and the release build compiles from the current patchset head.

Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>

* server: keep the go_test srcs in gazelle order

---------

Co-authored-by: Codesmith Staging <codesmith-bot@users.noreply.github.com>
Co-authored-by: shreyas@blacksmith.sh <shreyas@blacksmith.sh>
Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants