[ericfirth]

Why

After #6532, a Data Streams pathway stops at any thread, fiber, or external enumerator started after a consume, so that work's produces start new pathways. Reviewers suggested fiber storage, which Ruby 3.2+ inherits into exactly that work.

What

On Ruby 3.2+ the pathway context lives in fiber storage, so work started after a consume links to the consumed message. Checkpoints now replace the context instead of mutating it, so inherited copies never write into each other.

Risk

Two-way door, scoped to Data Streams when enabled; revert this PR. Emitted data changes on Ruby 3.2+: a long-lived thread started from inside a consumer links to that message until its next checkpoint.

Details

Notes

• Stacked on fix(data_streams): scope DSM pathway context per thread instead of per process #6532. The base branch is fix(data_streams): scope DSM pathway context per thread instead of per process #6532's branch; this will be retargeted to master once fix(data_streams): scope DSM pathway context per thread instead of per process #6532 merges. Opened as a draft while tracing settles its own storage choice in Add Thread-bound tracing context #5226, so DSM and tracing can end up consistent.
• Why replace instead of mutate. New threads inherit fiber storage with the same object (Fiber[:x] = obj; Thread.new { Fiber[:x].equal?(obj) } is true on 3.4), and threads checkpoint concurrently. Mutating in place would let a thread started after a consume rewrite the consumer's pathway, which is the bug fix(data_streams): scope DSM pathway context per thread instead of per process #6532 fixed. Each checkpoint now works on a dup of the current context and installs it; the fields are integers, times, and strings, so a shallow copy is enough.
• Concurrent fibers. Fibers that run concurrently (Async tasks, Falcon requests) are expected to get their own storage from the library or scheduler (Fiber.new(storage: nil)), per the review thread on fix(data_streams): scope DSM pathway context per thread instead of per process #6532.
• Ruby < 3.2 keeps fiber-locals (Thread#[]), the behavior from fix(data_streams): scope DSM pathway context per thread instead of per process #6532. The implementation is chosen once, when the file loads.
• Specs in spec/datadog/data_streams/processor_spec.rb: a new thread chains onto the consumed pathway; its checkpoints leave the starting thread's context untouched (fails without the dup); an external enumerator chains onto the consumed pathway; on Ruby < 3.2 a new thread starts a new pathway.
• Test: bundle exec rake spec:core_with_libdatadog_api.

🤖 Generated with Claude Code

[linear-code]

DSM2-484

[datadog-prod-us1-5]

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 94.30% (+0.10%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 4670443 | Docs | View more details | Give us feedback!}