Repository navigation
Conversation
Collaborator
|
I think the general approach is pretty good. I like the idea of keeping the instrumentation mechanism(s) out of rr proper. Do we really need to support i386? Do you have a use-case for that? I'd be happy to not support it for now, and if we ever add support for it, make the instrumentation work with the 64-bit value. I think you can use cmpxchg8b on i386 to decrement it atomically. I think it would be helpful for the rr trace header to include metadata describing which instrumentation scheme was used. That might help consumers of rr traces deal with them. Maybe copy the contents of the ELF note in there. What's your plan for system libraries? "in code that is not instrumented (e.g. libc, ld.so) the registers alone must tell repetitions apart, which makes reverse execution through long uninstrumented stretches slow or ambiguous." sounds ... worrying. |
Member
Author
After this commit, I built an instrumented userspace container image. Works fine. In principle the binary instrumentation approach should also work, but I haven't tried it yet. |
Member
Author
I did add Did you want something fancier than that?
Not really, but julia does build binaries for it, so it'd be kinda nice if it worked?
The question is more about the size of the assembly sequence. The sequence I came up with is I think the shortest that works well, but of course one could just write a full 64 bit decrement - doesn't have to be atomic. That said, perhaps if we don't care that much about i386, we do just do the full 64 bit decrement even at the cost of a few extra instructions per tick sequence, but buying back some simplicity. |
Collaborator
Yeah, that doesn't tell us which instrumentation system was used.
Yeah I think that's worth it. |
Member
Author
Ah I see what you're saying. I'm not sure about that because one trace could in theory have multiple different kinds of instrumentation in it, so it would be a per-process value and at that point, I'm not sure if there's much value over just reading it out of the elf section that's in the trace. That said, we could perhaps define the contents of that section to be some capnp structure we define so that the analysis of the section can be tool-independent? Just to be clear though, I'm expecting that the trace itself is replayable fully self contained - i.e. there's no additional tooling assist needed during replay - e.g. the binary instrumentor would be recorded. That said, there may of course be value in a replay tool knowing what the instrumentation was to do the replay more effectively, but that's more fancy than I was thinking at the moment. |
Collaborator
OK, that makes sense. It's not that big a deal, it's just sometimes useful to have this kind of information for debugging an rr bug. I guess that's not too important for this case. |
Add a tick source for machines without a usable PMU (most cloud VMs): the recorded program counts its own ticks. A compiler pass (for LLVM or GCC) places a tick so that no instrumented instruction executes twice in the same stack frame without a tick in between (at function entries, loop headers and after returns_twice calls), which is what (ticks, registers) needs to identify a point in the execution. A tick decrements a countdown at a fixed address in the preload thread-locals page and traps when it reaches zero; include/rr/softticks.h is the tracee ABI. Instrumented modules carry a .note.rrsoftticks note. When the program rr starts carries the note, rr records with the new TicksSemantics value `software` and needs no PMU: - PerfCounters programs the countdown instead of perf events (skid 1). The countdown is per address space; rr runs one task at a time, so start() takes it over for the task it resumes after counting the ticks of the task that had it. - Task::did_waitpid turns a tick trap into the TIME_SLICE_SIGNAL stop the PMU interrupt would have caused, so scheduling, signal recording and replay work unchanged. When recording, a task stopped after a tick's decrement and before the end of its sequence is moved to the end: whether the rest runs depends on the countdown's value, which replay programs differently. - The trace header records the ABI version and the countdown's address (Header.softwareTicks). Replay reads the header before constructing the session, so these traces replay without a PMU too. - The preload thread-locals segment is sized to cover the countdown (the kernel does not keep what is written to a shared mapping's page beyond the end of its file). rr parks the countdown when it creates the page, so instrumented programs also work with PMU ticks, and memory checksums ignore it. rr reads and programs the countdown only in the page it created, while that is mapped and writable: a tracee may map over it, and a task that has not exec'd yet may still have the page of an outer rr. - The syscallbuf's force_tick() also executes the tick sequence, so that a task killed between a record's commit and that point is recognized in replay as with the PMU (bufferedSyscallForcedTick). With PMU ticks the countdown is parked and that tick never traps. - The PMU-only checks (ticks before the initial exec, CPUID-bug detection) are skipped. Built for software ticks (__RR_SOFTTICKS__, which the compiler passes predefine), tests with hand-written loops tick in them, mmap_replace_most_mappings keeps the countdown's page and rdtsc_loop2 keeps its rdtsc patchable; otherwise the tests are unchanged. Ticks only come from instrumented code: in code that is not instrumented (e.g. libc, ld.so) the registers alone must tell repetitions apart, which makes reverse execution through long uninstrumented stretches slow or ambiguous. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The i386 tick sequence used a 32-bit countdown plus a "disarmed" guard word,
so that code running outside rr would not trap when the 32-bit count wrapped,
and rr could arm at most 2^32 - 1 ticks at a time. Replace the guard with the
high word of a 64-bit countdown, decremented one 32-bit word at a time:
subl $1, 0x70001800 # low word
jnz 1f # fast path, unchanged
subl $1, 0x70001804 # low word hit 0: borrow from the high word
jae 1f # high word was nonzero: continue
int3 # the 64-bit count is exhausted
1:
This is the same size (19 bytes) with the same fast path, and gives i386 the
64-bit semantics of the other architectures: the initial value (both words
all ones, which the initializer now writes when it creates the page) never
traps in practice, and rr arms any period on every architecture (2^60 when
it wants no interrupt, as for the PMU) instead of capping it at 2^32 - 1.
A single cmpxchg8b would make the update atomic, but it is microcoded and
needs eax, edx, ebx and ecx and a retry loop at every function entry and
backedge. Atomicity is not needed: rr runs one thread of an address space at
a time and swaps the countdown between them, and a signal handler that ticks
between an interrupted tick's two subl's only subtracts from the same count.
Because a low word of 0 stands for 2^32 and the borrow is a separate
instruction, the ticks to go on i386 depend on where the task is stopped:
hi * 2^32 + lo, with lo = 0 counting as 2^32, less 2^32 while the task is
between the two subl's (at the jnz with ZF set, at the second subl, or just
after a breakpoint's int3 on either). software_ticks_remaining() computes it
when rr reads a task's ticks (including those of the previous owner when the
countdown changes hands), and software_ticks_slot_for() arms such a task with
1 more in the high word, which its pending borrow takes back. A recording
moves a stop inside the sequence to its end after the counters have counted
the tick and parked the countdown, so the skipped borrow no longer applies
to anything; a trap stop has the countdown at 0 as before.
The code of a tick sequence is now read with rr's breakpoints replaced by the
original bytes (when recognizing a tick trap, completing a sequence, or
looking for a pending borrow), so that a debugger breakpoint inside a
sequence does not hide it. software_ticks_break_in_tick replays an i386
program under gdb with a breakpoint at the borrow, which stops there whenever
an armed countdown runs out. overflow_branch_counter's software variant now
runs its 2^32 + 2^28 ticks under one countdown, which on i386 passes 0 in the
low word and borrows from the high word.
The ABI version stays 1: the software-ticks support has not been released.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Task::did_waitpid() calls is_software_tick_trap() for every SIGTRAP signal-stop in software-ticks mode, when recording and when replaying, before it sets wait_status to the new status. is_software_tick_trap() first read the siginfo with get_siginfo(), which asserts that wait_status, still the previous stop's, has a stop signal. So in a Debug build, any SIGTRAP stop (a single-step, one of rr's breakpoints or a tick trap) that followed a stop without a signal (a syscall, ptrace-event or group stop, or a cleared wait status) aborted with "[FATAL src/Task.cc:431:get_siginfo()] stop_sig()". On x86 that includes the single-step of the remote prctl(PR_SET_NAME) that rr makes right after an exec, so every Debug recording of a program built for software ticks failed at startup there. aarch64 doesn't single-step remote syscalls, so recording could hit the assertion only later. Replaying a software-ticks trace hit it on both. Release builds don't check, and pending_siginfo already held the new stop's siginfo. Pass the siginfo that did_waitpid() has just read with PTRACE_GETSIGINFO to is_software_tick_trap() instead. With this, in a Debug build on x86_64, software_ticks_async_kill, software_ticks_replace_page and overflow_branch_counter passed in 5 of 5 runs per variant, and the runs of software_ticks and software_ticks_break_in_tick that failed timed out recording or replaying (software_ticks-32 in 2 of 5 runs, software_ticks_break_in_tick in all 20). Those two take a timer signal every 1 or 0.5 ms, and on this loaded machine, delivering the signals took nearly all of the recording's time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test starts each recording in the background and waits for the
program to print "ready" before it kills it. If the recording ended
before that, as every Debug recording on x86 did before "Don't use
get_siginfo() for a stop did_waitpid hasn't recorded yet", the loop
waited forever, and so did ctest: test-monitor's timeout only covers
rr, not the test script. Now the loop fails the test, showing
rr's output, when the recording has ended, as other tests that wait for
a background recording do ("subshell died").
With a recording that fails right away (RECORD_ARGS=--bogus-option),
the test now fails at once instead of hanging. Otherwise it passes as
before: 5 of 5 runs per variant.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
To reach a target in writable memory (a JIT's code), replay sets a hardware breakpoint there (try_set_hardware_breakpoint), which PerfCounters::stop disables when the task stops. With software ticks, stop returned before doing that, so the breakpoint stayed armed and trapped every time the task executed that address again, e.g. in a loop: replay of the test below didn't finish within 5 minutes (it takes 0.3 seconds with this fix), and replaying a JIT under DynamoRIO reported "Replay got unrecorded signal SIGTRAP". Test: software_ticks_jit_loop runs a ticking loop in an anonymous writable and executable mapping while ITIMER_PROF interrupts it (245 signals in the loop here). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RecordSession::create() reads the software-ticks note of the program it starts with read_software_ticks_note(), which called find_section_file_offsets() without checking that the reader parsed a valid ELF header. When it didn't (the program is a #! script, as in the simple_script tests), ElfReaderImpl's section table and its size are left uninitialized and find_section() walks garbage: recording a script crashed rr whenever that memory wasn't zero. Like read_dynamic(), check ok() first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test ticks by hand, then replaces the countdown's page with a PROT_NONE mapping and must not tick again. Built by a software-ticks compiler pass (__RR_SOFTTICKS__), main also got the pass's ticks, at the loop's head among others, so the program crashed on its first tick after the mmap. Ask the pass to leave main alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When rr records an rr (record_replay: rr records "rr replay" of a
software-ticks trace), the inner rr maps its own preload thread-locals
page over ours in its tracee and programs the countdown there. A tick
trap of that countdown stops the grandchild with SIGTRAP at the end of a
tick sequence, and the outer rr took it for its own time-slice
interrupt: is_software_tick_trap() only looked at the code at the pc.
The trap was recorded as a sched event and never reported to the inner
rr, whose replay then missed its target ("expecting tracee signal or
trap").
rr never reads or arms a countdown in a page it didn't create
(software_ticks_slot_usable()), so a trap there is someone else's: leave
it an ordinary SIGTRAP for the tracee or its ptracer. Likewise
complete_software_tick_sequence() must not skip such a trap.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test's 500us ITIMER_REAL could starve it: when recording one signal takes rr longer than the period (a Debug rr with --check-cached-mmaps, as the test harness runs it, takes over 1ms), the next SIGALRM is due by the time the handler returns, every time, and main's loop never runs again, so the recording never ends. ITIMER_VIRTUAL only counts the program's own user time, so the loop always runs between signals. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RecordSession::create() chose software ticks when the file it was asked to record carries the software-ticks note. A #! script carries none, but what runs is its interpreter: in a system whose programs all count software ticks, "rr record ./script.sh" picked the PMU (and failed where there is none, as in the simple_script tests), and a script whose interpreter counts software ticks was recorded with hardware ticks. Follow #! lines to the interpreter, as the kernel does, and read the note there. software_ticks_script records a script whose interpreter is the software_ticks test program and checks that the trace uses software ticks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GitHub's hosted runners have no PMU that rr can use, so rr's tests can only run there with software ticks. The new workflow runs them in a Linux image in which every binary is built with a software-ticks compiler pass and whose GCC applies the pass to everything it compiles, so rr records each test program with software ticks. The image's container is started by an action that runs the job's later steps in it. The image has no 32-bit toolchain (-Ddisable32bit=ON), no capnproto (built from source and cached) and no lldb (the runner's lldb is run in the container with the runner's own loader and libraries). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
Now with a github actions workflow that runs softticks on CI (pull my linux instrumented linux image). All tests pass (including nested with rr itself softticks instrumented!). |
On aarch64 a tick loads the countdown into x17, decrements it and stores
it back (the commit), and clears x17 with the mov at the end of the
sequence. rr compares x17 between recording and replay, and from the ldr
to that mov x17 holds a countdown value, which differs between the two,
so a stop recorded there could not be found again in replay ("overshot
target ticks ... by 1", x17 mismatched):
- At a tick trap rr continued at the mov after the brk. Continue at the
end of the sequence instead, with x17 = 0, as the mov would leave it.
- complete_software_tick_sequence() moved a recorded signal stop after
the commit to the end of the sequence but left x17 as it was, and
missed a stop at the mov. Clear x17, and also complete a stop at the
mov and between the ldr and the str: there the counters have not
counted the tick yet, so completing the sequence commits it (the
caller adds it to the task's ticks).
- A stop between the ldr and the str that rr does not move (in replay,
at breakpoints and single-steps) has in x17 the value the str will
store, computed from the countdown the ldr loaded. Arming the
countdown there was undone by the str, which counted ticks twice
("overshot ... by 2"). After arming, set x17 as if the ldr had loaded
the new countdown (the aarch64 counterpart of i386's pending borrow).
x86 is unchanged: its decrement is a single instruction and only the
flags, which rr doesn't compare, depend on the countdown.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Member
Author
|
Further experimentation with the instrumented linux distro shows catastrophic cache trashing in threaded code when not running under rr. Will rejigger the tick sequence to fix that. |
Outside rr all threads of a process tick the one countdown in the page the initializer maps, and every tick decremented it: a store from every core the threads run on to one cache line, which kept the line moving between them, and a chain of stores and loads through it in each thread. A loop running in 4 threads was ~10x slower than uninstrumented. Each tick now first tests the countdown's top bit and, while it is set, skips the decrement and the trap without writing anything: on x86 a testb of the countdown's top byte (testb $0x80, 0x70001807; jnz), on aarch64 a tbnz on what the ldr loaded (tbnz x17, rr-debugger#63). Outside rr the countdown is 2^64 - 1 (i386's initializer writes that; on x86-64 and aarch64 the zero-filled page wraps to it at the first tick), so ticks only read a cache line that every core can share: the 4-thread loop is now ~1.8x slower than uninstrumented. rr already arms countdowns in [1, 2^63) (SOFTWARE_TICKS_UNLIMITED is 2^60) and parks the countdown at 2^64 - 1, which the guard now treats as disarmed. 0 is no longer a way to arm it ("never traps"): it wraps to 2^64 - 1 at the first tick, which disarms it; rr never arms 0. The commit is still one instruction, and the trap still comes after it. The sequences grow to 21 bytes (x86-64), 28 (i386) and 32 (aarch64), which moves the trap (to offsets 20, 27 and 24) and the places where rr completes or rebases a tick: - x86: complete_software_tick_sequence() still moves only stops after the commit (x86-64: 18, 20; i386: 16, 18, 25, 27), and x86_borrow_pending() looks 9 bytes further into the sequence. A stop at the guard's jnz or at the decrement is an ordinary one: the flags there depend only on the top bit, which is clear whenever rr runs the task, in recording and in replay. - aarch64: x17 holds the countdown from the tbnz (8) to the mov (28). Completing a stop before the commit (tbnz, sub, str) counts the tick, and rebase_pending_software_tick() rebases it, only if the tick commits: at the tbnz with x17's top bit set the guard skips it. RR_SOFTTICKS_ABI_VERSION stays 1: the ABI is new in this series. software_ticks_jit_loop now assembles its loop from the header's tick instead of hard-coding the bytes, software_ticks_break_in_tick's i386 breakpoint follows the borrow, and the new software_ticks_disarmed checks that ticks leave a disarmed countdown alone (also one of exactly 2^63) and that on x86-64 and aarch64 a zero one disarms itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The image's binaries and compilers now emit the sequence that tests the countdown's top bit first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Keno
added a commit
to ChronitonAI/julia
that referenced
this pull request
Nov 6, 2026
…ware ticks A Julia whose runtime is compiled for rr's software ticks (rr-debugger/rr#4104), by a compiler that predefines `__RR_SOFTTICKS__`, instruments the code it generates as well, but its host platform looked like any other, so Pkg gave it the ordinary, uninstrumented artifacts even for packages that also have builds for software ticks. The runtime now reports the version of the software-ticks ABI it was compiled for (`jl_softticks_abi_version`, 0 if none), and `HostPlatform()` carries it as the tag `rr_softticks`, so that such a package gets its artifacts for, e.g., `x86_64-linux-gnu-rr_softticks+1`. Like `sanitize`, the tag is strict: a platform with it only matches platforms with the same value, so a Julia built for software ticks never falls back to uninstrumented artifacts (a package that has none for it gets no artifact), and an ordinary Julia never selects instrumented ones. The version is read from the runtime whenever `HostPlatform()` is computed, rather than recorded in `Base.BUILD_EXT_TAGS` by the build like `sanitize`, since it is the compiler, not the build system, that decides whether the runtime is instrumented. Assisted-by: Claude Code (Opus 5.5)
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The idea of adding a soft-ticks mode to rr has been discussed many times. I think the main reason it never really went anywhere is that there's too much variation in how people think the necessary instrumentation should get into the instrumented binary. So this is my attempt at adding the feature without having to go into the exact specifics of how the instrumentation happens. We simply define the ABI and whatever and however the user wants to satisfy that is not our problem.
The ABI defined is as follows. There is a counter word in rr thread-local preload space (instrumented binaries are responsible for allocating this if the tracer is not attached). The count word is used as follows:
(size_t)-1).i386 is slightly more complicated because the trap word can overflow (underflow) during record, which would give a spurious SIGTRAP if rr is not attached to handle it. To fix this, x86 gets a separate guard word to disable the trap unless rr is attached. Another option would be to use a 64 bit counter there as well (well internally in rr, it's always 64 bits, but it detects overflow and adds), at the cost of one extra instruction per tick increment, but that didn't seem worth it.
This deliberately does not define when ticks happen. The only invariant is that the same register state cannot recur between ticks. A safe invariant is thus to tick on every function entry, backedge and after any returns_twice functions, but instrumenting compilers are permitted to optimize as long as they uphold the invariant.
To decide whether to use soft-ticks mode, rr recognizes a special ELF section. The idea is that this would be set by either a binary that was compiled with appropriate compiler instrumentation or a dynamic binary instrumentation runner that understands the tick format.
Implementation is AI-generated and I need to do a lot more testing, but I wanted to open this early for feedback.