Repository navigation
Conversation
An attempt to gather more information in the case of 1987, which so far has only been seen on extent 0. Keep a bounded record of the jobs that write to or live-repair extent 0: - The block contexts written - What each client was told to do with the job - The dependencies each client was sent - Where live-repair was, - Each client's final state On a read hash mismatch, add every traced job for each mismatched block to the panic message, marking which write produced the expected and the computed contexts. Log the source and target extent info that live-repair compares for each extent, and whether it decided to repair. When live-repair closes a traced extent, log the close's full dependency list and every active job that touches the extent. Stuff all this in the panic string so we are sure to see it.
Focus the trace on whether a write landed after live-repair closed the extent (A), or the extent info it compared was wrong (C). When a client acks live-repair's close of a traced extent, log the writes to that extent, enqueued before the close, that the client has not finished, and whether each is in the close's dependencies. Keep these in the trace for the panic message. Number each client's completions in the order they arrive, and record them for traced jobs. On a read mismatch call out any write to the block that finished on a client after live-repair's close of its extent. Keep the last 4 writes to each traced block rather than the last 4096 jobs. Keep every write that was active when live-repair closed its extent so the writes behind a mismatch are still there no matter how long after the repair it shows up.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is just for debugging, it should not be merged.
An attempt to gather more information in the case of 1987, which so far has only been seen on extent 0.
Keep a bounded record of the jobs that write to or live-repair extent 0:
On a read hash mismatch, add every traced job for each mismatched block to the panic message, marking which write produced the expected and the computed contexts.
Log the source and target extent info that live-repair compares for each extent, and whether it decided to repair.
When live-repair closes a traced extent, log the close's full dependency list and every active job that touches the extent.
Stuff all this in the panic string so we are sure to see it.