Skip to content

Gdb stepping fixes - #89

Merged
techomancer merged 6 commits into
techomancer:mainfrom
kulichbulich:gdb-stepping-fixes
Aug 21, 2026
Merged

Gdb stepping fixes#89
techomancer merged 6 commits into
techomancer:mainfrom
kulichbulich:gdb-stepping-fixes

Conversation

@kulichbulich

Copy link
Copy Markdown
Contributor

PR companion: make the GDB stub usable for source-level debugging of guest code

This file is the accompanying write-up for the pull request(s), not repo
documentation — it holds the long-form rationale that was deliberately kept
out of the source comments (those are trimmed to ≤3 lines each). Don't
commit it; paste the relevant sections into the PR description(s).
Companion: bug_report.md (the raw investigation log, including two issues
that are not fixed here).

TL;DR

Debugging a userspace IRIX binary (an SDL2 game, N32 MIPS, DWARF from a
mips-sgi-irix6n32 cross-gcc) through iris --gdb-port did not work: a
breakpoint was hit correctly, but every subsequent stepi / next / step
left the PC unmoved, and continue re-announced the same breakpoint forever or
produced phantom SIGTRAP … in ?? () stops. No error was reported anywhere —
in GDB, in VS Code, or in the emulator log.

Root cause: one bit in a breakpoint-address mask. Five further defects on
the same paths turned up while chasing it — all six are fixed here (the last one
is a parity fix with no reproduced failure; see §3.6). With them applied, VS Code
stops at a source line, shows correct locals/args, and steps.

 src/gdb_stub.rs  |  9 ++++--
 src/mips_exec.rs | 92 +++++++++++++++++++++++++++++++++++++++++++++++++-------
 2 files changed, 87 insertions(+), 14 deletions(-)

cargo build --release --bin iris --features rex-jit,chd — clean.


1. How this maps onto the contribution policy

README.md § Contribution policy:

We have no problems with LLM generated code. In fact most of IRIS is made with
LLMs. But that doesn't mean we don't do proper software engineering. So lets
keep PRs small and reasonable to review. One issue/fix per PR, preferably in
one commit, since LLM code churn doesn't help with clarity. Lets keep this
bisectable too.

Proposed as one PR of six commits, not six PRs. The reading is that this is
one issue — "the GDB stub can't step guest code" — and the six commits are the
layers of that single bug rather than six unrelated topics. Reviewed in
isolation each one looks speculative: #1 alone doesn't explain the phantom
SIGTRAPs, and #2/#3 alone don't make stepping work. The policy's other two
asks are met directly: every commit compiles on its own with zero warnings
(verified by checking out each in turn), and every commit carries its own root
cause and evidence in its message, so git bisect and git log -p both stay
useful.

The contentious one, #6, is deliberately last so it can be dropped with a
single git rebase --onto HEAD~1 HEAD~1 HEAD without disturbing anything else.

Each row below is still a self-contained change, in the order they are applied
— so if you'd rather have them as separate PRs, any commit can be cherry-picked
onto main on its own without rework (only #6 needs #2 underneath it):

# Change File / fn Lines Independent?
1 Breakpoint address comparison aliases adjacent instructions mips_exec.rs: new bp_match_key(), check_breakpoint() +19 / −6 yes — the actual bug; land first, it stands alone
2 EXEC_RETRY must be retried, not reported as a stop mips_exec.rs: run_debug_loop(), step_one() +39 / −3 yes
3 Stale last_bp_hit reported as a phantom breakpoint mips_exec.rs: run_debug_loop() +5 yes
4 Failed memory read must be an RSP error, not zeros gdb_stub.rs: read_addrs() +6 / −3 yes
5 Don't stop on a PC breakpoint whose instruction can't be fetched mips_exec.rs: step() +8 / −1 yes (nicer with 4)
6 step_one() lacks run_debug_loop()'s "step off the breakpoint" skip mips_exec.rs: step_one() +5 yes — parity fix, no reproduced failure; drop it if you'd rather not carry an unproven change

Only #2 and #6 touch the same function (step_one), in adjacent code, so #6
has to sit on top of #2. Everything else cherry-picks onto main cleanly and
independently — verified, each on its own compiles with zero warnings.

If only one commit can be taken, take #1: it is 25 lines, it is the actual
bug, and it stands entirely on its own.


2. Shared background: why a MIPS GDB client stresses exactly these paths

Every one of these defects is invisible to the monitor console and only shows up
through the RSP stub, for one reason: GDB's MIPS backend always uses software
single-step.
It never sends the s / vCont;s packet. A plain stepi
produces:

[remote] Sending packet: $mffffffff80000180,4#8e     <- read the instruction at PC
[remote] Packet received: 00000000
[remote] Sending packet: $Z0,ffffffff80000184,4#db   <- breakpoint on the computed next PC
[remote] Packet received: OK
[remote] Sending packet: $vCont;c:p1.-1#0f           <- ...and CONTINUE
  [remote] Packet received: T05thread:p01.01;swbreak:;
[remote] Sending packet: $z0,ffffffff80000184,4#fb   <- remove it again

Two consequences worth keeping in mind while reviewing:

That also explains the shape of the symptoms: the failure mode of every one of
these bugs is silence, not an error.


3. The changes in detail

#1check_breakpoint(): the mask cleared bit 2, so adjacent instructions compared equal

This is the root cause. Everything else in this document is secondary.

Before:

// Normalize to physical: strip sign-extension and kseg bits (top 3 of
// low 32), then mask bottom 2 for word alignment.   <-- comment says bottom 2
const PHYS_MASK: u64 = 0x1FFF_FFF8;                  //     constant clears bottom 3
if (bp.addr & PHYS_MASK) == (addr & PHYS_MASK) {

The comment's stated intent ("mask bottom 2 for word alignment") is & !3; the
constant ends in 8, i.e. & !7. One bit, two consequences:

  1. Adjacent-instruction aliasing. 0x1004c758 & 0x1FFF_FFF8 and
    0x1004c75c & 0x1FFF_FFF8 are both 0x0004c758 — every breakpoint also
    fires on its doubleword neighbour. Since GDB single-steps by planting a
    breakpoint at PC+4 and continuing (§2), that breakpoint matched at PC
    itself and execution broke before the instruction ran. PC never advanced.
    Repeat forever.
  2. Upper-bit truncation. Only the low 29 bits survived, so a breakpoint also
    matched any address sharing them — a different segment, or the same virtual
    address in an unrelated process. This is the kseg0/kseg1 "physical
    normalization" being applied to every address rather than to the two
    windows where it is actually correct.

Note it does not fail on every step: if the second attempt happens to retire
the instruction before the aliased match is re-tested, the step completes. That
intermittency is why this looked like several unrelated problems at first.

Fix. Replace the flat mask with an explicit helper:

fn bp_match_key(addr: u64) -> u64 {
    let hi = addr >> 32;
    if hi == 0 || hi == 0xFFFF_FFFF {
        let lo = addr & 0xFFFF_FFFF;
        if (0x8000_0000..0xC000_0000).contains(&lo) {
            return (lo & 0x1FFF_FFFF) & !3;
        }
    }
    addr & !3
}

Word-align, and fold only the unmapped 32-bit compatibility windows —
kseg0 0x8000_0000..0x9FFF_FFFF and kseg1 0xA000_0000..0xBFFF_FFFF, in
either sign-extended or plain 32-bit form — onto the physical address they
alias. That preserves the documented and intended behaviour ("a bp on physical
0x1fbb0010 is hit whether the access came through 0x9fbb0010 or
0xbfbb0010"), which the monitor's device-poking workflow relies on. Every
other address keeps all of its bits, so distinct virtual addresses can no
longer collide.

Blast radius. check_breakpoint() is shared by all breakpoint kinds (Pc,
VirtFetch/PhysFetch, VirtRead/PhysRead, VirtWrite/PhysWrite) and by both the
interpreter (step()) and the jitv2+developer diagnostic hook
jit_dev_trace_bp() — all of them get the fix for free, and all of them were
subject to the same aliasing. Behaviour changes only where two addresses used
to compare equal but shouldn't have; no previously-working match is lost. There
is no other copy of the mask in the tree (grep -rn 1FFF_FFF8 src/ → only this
site). Compiled out entirely under --features lightning (the whole PC-bp
check is #[cfg(not(feature = "lightning"))]).

Reproduction — monitor console only, no GDB involved (this is the cheapest
way to review the fix):

> status
stopped pc=000000001004c758
> bp add 0x1004c75c
Breakpoint 1 added at 000000001004c75c (Pc)
> cont
Running...
PC=000000001004c758: Breakpoint 1 hit          <-- reported at 0x…758, not 0x…75c
Next: 000000001004c758: 8f99973c lw t9, -26820(gp)
> status
stopped pc=000000001004c758                    <-- nothing executed

Verified fixed (same sequence, kernel address this time):

PC        = 0xffffffff8812a124
bp add      0xffffffff8812a128        (PC+4)
cont     -> PC=ffffffff8812a128: Breakpoint 1 hit
status   -> stopped pc=0xffffffff8812a128        (advanced by 4)

#2EXEC_RETRY was never retried, so a debugger made no progress

EXEC_RETRY (bus busy) means the instruction did not retire: nothing
changed architecturally and PC is unmoved, so the caller must attempt the same
PC again. The repo already documents exactly this contract — src/validate.rs
(~lines 204–233) loops until step_status != EXEC_RETRY and caps attempts at
MAX_RETRIES_PER_INSTRUCTION = 100_000 so a permanently-busy device can't spin
forever holding the executor lock.

Neither debugger path honoured it:

  • run_debug_loop() (monitor run/step, and the GDB stub's continue via
    run_blocking()) broke out of the loop and printed
    PC=…: Retry (Bus Busy). For a GDB client that turns every transient GIO
    bus-busy — observed frequently during ordinary desktop/graphics activity,
    logged as MC: GIO Timeout — into a stop with PC unchanged. Because the PC is
    still sitting on the user's breakpoint, GDB re-announces
    Breakpoint N, main (…) at file.c:LINE over and over while the program makes
    no progress. Combined with Fatal Error with NVIDIA GPU #3's stale last_bp_hit, the same condition also
    produced phantom SIGTRAPs at addresses with no breakpoint at all.
  • step_one() (GDB stepi, monitor-independent single step) silently
    ignored
    EXEC_RETRY and reported a completed step, so a single-step became
    a no-op with nothing surfaced.

Fix. Both paths now follow validate.rs's contract:

  • step_one(): retry while status == EXEC_RETRY, bounded by
    MAX_STEP_RETRIES = 100_000 with a spin_loop() hint; if still busy after
    the cap, report a real stop (StopReason::Interrupted) rather than a phantom
    "step completed".
  • run_debug_loop(): on EXEC_RETRY, continue the loop instead of breaking —
    bounded by MAX_RUN_RETRIES = 100_000 per instruction attempt (the
    counter is reset whenever something retires, so a long run can't accumulate
    its way into a false give-up), refunding the count budget so step N still
    retires N architectural instructions. The original Retry (Bus Busy)
    diagnostic is kept for the genuinely-stuck case, now with the attempt count.

Note on the lock. The retry loop in step_one() spins while holding the
executor lock, which is why it is bounded exactly like validate.rs. That
matches the per-device concurrency model in HACKING.md: the device that
answered "busy" runs on its own thread and does not need the executor lock to
make progress, so spinning here cannot deadlock against it.

Verified fixed. An 18-second GDB continue produced no SIGTRAP and
no breakpoint re-announcements, with the monitor console independently
confirming the CPU ran throughout. Before the fix the same operation stopped
almost immediately and repeatedly.


#3 — a stale last_bp_hit was reported to GDB as a phantom breakpoint hit

Symptom, on continue (F5 / -exec continue — Step/Next ruled out):

Program received signal SIGTRAP, Trace/breakpoint trap.
0x00000000103f8fa4 in ?? ()

repeatedly, at slowly-incrementing addresses, with no resolvable symbol, and no
breakpoint anywhere near those addresses.

last_bp_hit was only ever cleared in step_one(). run_debug_loop() never
reset it, and run_blocking() inspects it after the loop to decide what to
report:

let reason = if let Some(exec) = self.cpu.executor.try_lock() {
    if let Some(bp_id) = exec.last_bp_hit {
        // ...
        _ => StopReason::SwBreakpoint,
    } else {
        StopReason::DoneStep
    }
} else { StopReason::DoneStep };

So any loop exit for a reason other than a real breakpoint — EXEC_RETRY
(#2), an exception_mask match, an interrupt — inherited whatever id was left
from an earlier stop (including the initial stopAtConnect pause) and was
misclassified as StopReason::SwBreakpoint. StopReason::to_gdb() maps that to
SwBreak(()), which GDB prints as SIGTRAP/Trace/breakpoint trap; with no
real breakpoint at the reported PC there is no symbol to resolve, hence ?? ().

Fix. exec.last_bp_hit = None; at the start of run_debug_loop()'s task
closure, right after acquiring the executor lock — so a non-breakpoint exit
correctly falls through to StopReason::DoneStep. (The step() change in #5
clears it in the one other place a hit id can be recorded without a stop being
reported.)


#4read_addrs() returned zeros + OK for addresses that don't translate

Before:

if self.cpu.read_mem(mips_sign_extend(start_addr), data).is_err() {
    data.fill(0);
}
Ok(data.len())

For a mapped-space virtual address not currently in the TLB, the stub answered
all zeros with an OK reply. The monitor console, for the very same address,
correctly refused:

> status
stopped pc=000000001004c758
> dis 0x1004c758 4
0x000000001004c758: Could not fetch          (x4)
> translate 0x1004c758
Exception(0x38000008)                        (exc code 2 = TLBL)

but over RSP:

(gdb) x/4xw 0x1004c758
0x1004c758:  0x00000000  0x00000000  0x00000000  0x00000000     <- silent zeros

while the real instructions there (from the binary's own DWARF) are:

0x1004c758 <main+200>:  lw    t9,-26820(gp)
0x1004c75c <main+204>:  jalr  t9              <- a CALL
0x1004c760 <main+208>:  move  a0,zero         <- its delay slot

0x00000000 decodes as a perfectly valid nop. Per §2, GDB then plans its
single-step from that fiction: it computes the "next PC" for a nop, plants its
step breakpoint there, and continues — nothing about which matches the real
control flow. Zeros are also indistinguishable from genuinely zeroed memory
when inspecting data structures, so a user gets plausible-looking garbage
instead of a fault.

Fix. Propagate the failure as TargetError::NonFatal, i.e. an E xx reply.
GDB then prints Cannot access memory at address 0x… and declines the step.
This converts a silent, deeply confusing failure into an obvious one; it does
not by itself make the memory readable (see §5).


#5 — a PC breakpoint was honoured before the instruction was known to be fetchable

step() tests check_breakpoint::<Pc>(pc) before the fetch. On MIPS the
TLB is software-managed, so a userland PC whose page isn't resident right now is
unreadable — and a breakpoint at such an address fires before the
demand-paging fault that would map the page. That is the normal case, not a
corner case: observed with K:U, EXL=0, EPC=0x1004c758, no pending
interrupts — the kernel had just eret'd back into user code whose TLB entries
had been recycled while other processes ran, and the stop was taken on the very
first, not-yet-faulted instruction.

Reporting a stop there strands the debugger on code it cannot read (§2): every
step becomes a no-op and the user sees the debugger stuck on one source line.

Fix. Probe first, and only report the breakpoint if the instruction is
actually fetchable:

if !self.debug_translate(pc).is_exception() {
    return EXEC_BREAKPOINT;
}
self.last_bp_hit = None;

debug_translate() is the non-faulting probe (it does not set CP0 state or
raise), so this doesn't perturb the guest. If the fetch would fault, execution
proceeds normally: the fetch takes the TLB exception, the kernel maps the page
and erets back to this same PC, the breakpoint matches again — now with the
code readable by the debugger. That is also the more accurate architectural
semantics: an instruction whose fetch faults never executed, so a breakpoint on
it has not been reached yet.

The last_bp_hit = None on the not-taken path is required, or the id
check_breakpoint() just recorded would make the next stop get misreported as
this breakpoint (same failure as #3).

Observed effect at the address above: translate 0x1004c758 went from
Exception(0x38000008) (TLBL) to Translated { phys_addr: 0x170ce758 } by the
time the stop was reported, and dis showed real instructions.

Why #4 and #5 are both needed. #4 turns "silently wrong" into "clear
error"; #5 removes most of the occasions on which that error would be hit at
all. Either alone leaves stepping across a call unreliable.


#6step_one() lacked run_debug_loop()'s "step off the breakpoint you're sitting on" skip

Honest status: this is a parity fix with no reproduced failure. Later
measurement showed stepi does advance the PC even while stopped on a
breakpoint, so the code inconsistency is real but its user-visible impact is
unconfirmed. Given §2 (step_one is dead code for a MIPS GDB client) it may
never be reachable from GDB at all. Include or drop it on that basis.

step() checks for a breakpoint match before dispatching, so if the CPU sits
exactly on an active breakpoint's address — as it always does right after that
breakpoint was hit — a step_one() that returns immediately on
EXEC_BREAKPOINT never executes the instruction:

let status = exec.step();
let reason = if status == EXEC_BREAKPOINT {
    drop(exec);
    return StopReason::SwBreakpoint;      // instruction never dispatched
} else if ...

run_debug_loop() already handles this with a one-shot
exec.skip_breakpoints = true before retrying exec.step() on its first
iteration (first_step); step_one() was simply missing the equivalent.

Fix. Mirror it: on an initial EXEC_BREAKPOINT, set
exec.skip_breakpoints = true and call exec.step() again. step() clears the
flag itself after every call (it is a genuine one-shot), so no manual reset is
needed.


4. What is deliberately not in this PR

Two further defects were found and are written up in bug_report.md, but they
are separate concerns and are not touched here (per "one issue/fix per PR"):

  • Detaching a GDB client while the CPU is stopped leaves the CPU stopped.
    The guest desktop stops repainting and looks like the emulator crashed; it has
    to be resumed with start from the monitor console. Detaching while the CPU
    is running does not reproduce it. GDB's own D semantics are "resume the
    target and stop debugging it", and HACKING.md §9 already documents that
    GDB-set breakpoints are removed on disconnect — the run state arguably
    deserves the same treatment.
  • A client that dies mid-continue is never reaped, and the stub then refuses
    all further connections.
    IrisEventLoop::wait_for_stop_reason's poll loop
    appears to see peer EOF as conn.peek() -> Ok(None), indistinguishable from
    "no data yet"; with the CPU running, neither exit is ever taken and the
    single client slot is never released (ss -tlnp | grep 1234 shows a growing
    Recv-Q with nothing accepted). Non-destructive workaround: cpu stop then
    start from the monitor console. Suggested direction: treat a zero-length/EOF
    peek as a disconnect.

Also noted, not changed:

  • A misleading monitor message. While the CPU is running, status reports
    Error: CPU thread holds the executor lock; try 'cpu stop' first. That is the
    normal state for a running CPU, but the wording reads like a fault — it was
    initially misread as a deadlock during this investigation (as was locks
    showing cpu::executor LOCKED). Something like "CPU is running — stop it
    first to inspect state" would remove the ambiguity.
  • Reading guest memory that isn't in the TLB. NAT Failure on external host routing ICMP test #4 makes the failure explicit;
    it does not make the memory readable. The better fix is for the debug
    read/translate path to fall back to walking IRIX's page tables on a TLB miss.
    Investigated and rejected for this PR based on measurement: the PTE page
    itself (0xffffffffff0602c0, computed from CP0 Context) is also not in the
    TLB (Error 0x18000008), so it needs recursive knowledge of IRIX's page-table
    topology rather than a single walk. Worth doing separately if next/step
    over calls into cold code needs to be fully reliable.

5. Checklist against the project's own conventions

From CLAUDE.md and the CI configuration:

  • HACKING.md read before touching CPU code — yes; this is CPU/debug-path
    code (mips_exec.rs), not device code.
  • "Endianness lives only at The Edge" — respected. No .to_be() /
    .to_le() anywhere in these changes; addresses and status codes are treated
    as bit-containers throughout. bp_match_key() is pure integer masking.
  • "Concurrency is per-device" — the only new blocking is the bounded
    EXEC_RETRY spin in step_one()/run_debug_loop() while holding
    cpu::executor. The device that answered "bus busy" runs on its own thread
    and doesn't need that lock to make progress, so this cannot deadlock against
    it; the 100_000 cap matches validate.rs's existing precedent for the same
    situation.
  • rules/ note for a hard-won finding — CLAUDE.md asks for one. Suggested
    path rules/debug/gdb-mips-software-single-step.md; draft in §7 below. It is
    worth its own file because the reason all five bugs presented as silence is
    structural (GDB never sends s for MIPS), and that fact is not currently
    recorded anywhere in the tree. HACKING.md §9 is the other candidate home.
  • CI — .github/workflows/rust.yml: cargo build --verbose +
    cargo test --verbose on PRs to main. Both clean locally:
    cargo test --release385 passed, 0 failed, 11 ignored (plus the
    integration binaries), no new warnings.
  • CI — .github/workflows/cpu-tests.yml: triggers on src/mips_*.rs, so
    these changes run the full bare-metal CPU matrix (R4400 vs R5000, interp vs
    jitv2). Relevant here: check_breakpoint() and step()'s pre-dispatch
    check are on the interpreter's per-instruction path, so a regression would
    show up there as a cross-cell disagreement.
  • Feature flags — everything in check_breakpoint()/step() is inside
    #[cfg(not(feature = "lightning"))], i.e. compiled out of a lightning
    build (which has no GDB stub anyway). Verified against the project's real
    feature set: cargo build --release --bin iris --features rex-jit,chd
    (a bare cargo build --release silently drops CHD support, so the same
    disk image won't attach — cross-check the iris: build features: startup
    line when rebuilding).
  • HACKING.md §9's documented GDB setup is incomplete for this target
    set architecture mips:isa64, set mips abi n64, set mips mask-address off
    are listed, but set endian big is missing. Without it every register and
    memory value GDB shows is byte-swapped garbage (pc read as
    0x4cb60188ffffffff instead of 0xffffffff8801b64c) even though the RSP
    connection is fine — very easy to misdiagnose as a broken stub. A one-line
    doc addition, worth folding into whichever PR lands first.

6. Environment / how this was found

  • iris from a local checkout at 47f1242, built
    cargo build --release --bin iris --features rex-jit,chd.
  • Run as iris --config iris.toml --gdb-port 1234 (iris-gui cannot serve this
    — it has no CLI argument parser; the port has to come from gdb_port in
    iris.toml or the Configuration → Debug tab).
  • Guest: IRIX 6.5, Indy indy_ip24 profile, booted to the 4Dwm desktop.
  • Client: gdb-multiarch (stock Ubuntu gdb has no MIPS target), both driven
    from a script and via VS Code cppdbg attach; set architecture mips:isa64,
    set endian big, set mips abi n64, set mips mask-address off.
  • Target program: OpenTyrian2000 cross-built for IRIX N32 (big-endian MIPS,
    ELF32, -g), statically linked against a cross-built libSDL2.a.
  • Independent verification of core.pc for every claim above came from the
    monitor console on port 8888 (status, dis, translate, bp add, cont,
    step), which is how minor audio fixes for macos #1 was ultimately isolated without GDB in the loop.

Measurements that shaped the diagnosis, including two that ruled out an
earlier hypothesis, are in bug_report.md § Measurements.


7. Draft rules/debug/gdb-mips-software-single-step.md

Ready to drop in as its own file if you want the finding recorded per
CLAUDE.md:

# GDB's MIPS backend always software-single-steps — so `m` and `Z0` must be exact

GDB never sends the RSP `s` / `vCont;s` packet for a MIPS target. `stepi` is
implemented as: read the instruction at PC (`m`), compute the next PC, plant a
breakpoint there (`Z0`), `vCont;c`, then remove it (`z0`).

Consequences for anything touching `gdb_stub.rs` or the breakpoint path:

- `CpuDebug::step_one()` is effectively **dead code for a GDB client**. All
  stepping arrives as breakpoint + continue, i.e. through
  `run_blocking()``run_debug_loop()`. Fix stepping bugs there.
- A wrong breakpoint *match* is indistinguishable from a wrong step. If a bp
  at `PC+4` can match at `PC` (as a mask clearing bit 2 makes it), every step
  breaks before the instruction runs and the PC never moves — silently.
  `check_breakpoint()` must never mask below bit 2; use `bp_match_key()`.
- A memory read that answers zeros instead of an error hands GDB a valid
  `nop`, and it plans the step from fabricated code. Untranslatable addresses
  must return `E xx`.
- `EXEC_RETRY` means the instruction did not retire: retry the same PC
  (bounded — see `validate.rs`). Reporting it as a stop makes GDB re-announce
  the breakpoint the PC is still sitting on, forever.
- A PC breakpoint is tested *before* the fetch, and MIPS TLB refill is
  software-managed, so a breakpoint in userland routinely fires on a page that
  is not resident and that the debugger therefore cannot read. Defer the stop
  until the instruction is fetchable.

Every one of these fails **silently** — no error in GDB, VS Code, or the
emulator log. When "stepping does nothing", reproduce it from the monitor
console instead (`status`, `bp add <pc+4>`, `cont`, `status`); it takes four
commands and removes GDB from the equation entirely.

Also: `set endian big` is mandatory for this target and is missing from
`HACKING.md` §9's example. Without it, every value GDB prints is byte-swapped.

kulichbulich and others added 6 commits August 20, 2026 10:54
check_breakpoint() compared both addresses masked with PHYS_MASK =
0x1FFF_FFF8, which clears bit 2 — the comment above it says "mask bottom 2
for word alignment", i.e. & !3, but the constant is & !7. Two consequences:

1. Two adjacent instructions in the same doubleword compare equal.
   0x1004c758 and 0x1004c75c both mask to 0x0004c758. GDB single-steps MIPS
   by planting a breakpoint at PC+4 and continuing (its MIPS backend always
   uses software single-step — no s/vCont;s packet is ever sent), so that
   breakpoint matched at PC itself and execution broke *before* the
   instruction ran. The PC never advanced: single-stepping a guest program
   silently did nothing, forever, with no error reported anywhere.

2. Only the low 29 bits survived, so a breakpoint also matched any address
   sharing them — a different segment, or the same virtual address in an
   unrelated process.

Replace the flat mask with bp_match_key(): word-align, and fold *only* the
unmapped 32-bit compatibility windows (kseg0 0x8000_0000..0x9FFF_FFFF and
kseg1 0xA000_0000..0xBFFF_FFFF, in either sign-extended or plain form) onto
the physical address they alias. That keeps the documented and intended
behaviour — a bp on physical 0x1fbb0010 is hit through 0x9fbb0010 or
0xbfbb0010 — while every other address keeps all of its bits.

Reproducible from the monitor console alone, no GDB needed:

    > status
    stopped pc=000000001004c758
    > bp add 0x1004c75c
    Breakpoint 1 added at 000000001004c75c (Pc)
    > cont
    PC=000000001004c758: Breakpoint 1 hit    <- wrong address
    > status
    stopped pc=000000001004c758              <- nothing executed

After the fix, with PC at 0xffffffff8812a124 and a breakpoint at
0xffffffff8812a128, `cont` reports the hit at 0xffffffff8812a128 and status
shows the PC advanced by 4.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
EXEC_RETRY (bus busy) means the instruction did *not* retire: nothing changed
architecturally and PC is unmoved, so the caller must attempt the same PC
again. validate.rs's reference pass (~lines 204-233) already documents exactly
that contract, and caps the attempts at MAX_RETRIES_PER_INSTRUCTION = 100_000
so a permanently-busy device can't spin forever holding the executor lock.

Neither debugger path honoured it:

- run_debug_loop() (monitor run/step, and the GDB stub's continue via
  run_blocking()) broke out of the loop and printed "Retry (Bus Busy)". For a
  GDB client that turns every transient GIO bus-busy — frequent during
  ordinary desktop/graphics activity, logged as "MC: GIO Timeout" — into a
  stop with PC unchanged. Because the PC is still sitting on the user's
  breakpoint, GDB re-announces "Breakpoint N, main (...) at file.c:LINE" over
  and over while the program makes no progress.

- step_one() silently ignored EXEC_RETRY and reported a completed step, so a
  single-step became a no-op with nothing surfaced to the client.

Both now retry, bounded:

- step_one(): retry while status == EXEC_RETRY up to MAX_STEP_RETRIES, with a
  spin_loop() hint; if still busy after the cap, report a real stop
  (StopReason::Interrupted) rather than a phantom "step completed".

- run_debug_loop(): continue the loop instead of breaking, bounded by
  MAX_RUN_RETRIES per instruction attempt (the counter is reset whenever
  something retires, so a long run can't accumulate its way into a false
  give-up), refunding the `count` budget so `step N` still retires N
  architectural instructions. The original diagnostic is kept for the
  genuinely-stuck case, now with the attempt count.

The bounded spin in step_one() holds cpu::executor, but the device that
answered "busy" runs on its own thread and does not need that lock to make
progress, so it cannot deadlock against it.

Verified: an 18-second GDB `continue` produced no SIGTRAP and no breakpoint
re-announcements, with the monitor console independently confirming the CPU
ran throughout. Before, the same operation stopped almost immediately and
repeatedly.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A GDB client issuing `continue` regularly got

    Program received signal SIGTRAP, Trace/breakpoint trap.
    0x00000000103f8fa4 in ?? ()

repeatedly, at slowly-incrementing addresses, with no resolvable symbol and no
breakpoint anywhere near them.

last_bp_hit was only ever cleared in step_one(). run_debug_loop() never reset
it, and run_blocking() inspects it after the loop to decide what to report to
the client. So any loop exit for a reason other than a real breakpoint —
EXEC_RETRY, an exception_mask match, an interrupt — inherited whatever id was
left from an earlier stop (including the initial stopAtConnect pause) and was
misclassified as StopReason::SwBreakpoint. StopReason::to_gdb() maps that to
SwBreak(()), which GDB prints as SIGTRAP/Trace-breakpoint-trap; with no real
breakpoint at the reported PC there is no symbol to resolve, hence "?? ()".

Clear it once at the start of the task closure so a non-breakpoint exit falls
through to StopReason::DoneStep.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
read_addrs() answered a failed read with all zeros and an OK reply. The
monitor console, for the very same address, correctly refuses it:

    > status
    stopped pc=000000001004c758
    > dis 0x1004c758 4
    0x000000001004c758: Could not fetch          (x4)
    > translate 0x1004c758
    Exception(0x38000008)                        (exc code 2 = TLBL)

but over RSP:

    (gdb) x/4xw 0x1004c758
    0x1004c758:  0x00000000  0x00000000  0x00000000  0x00000000

The real instructions there are `lw t9,-26820(gp)` / `jalr t9` / `move
a0,zero`. 0x00000000 decodes as a perfectly valid `nop`, and GDB's MIPS
backend always software-single-steps (read the instruction at PC, compute the
next PC, plant a breakpoint there, continue — the s/vCont;s packet is never
sent), so it planned the step from that fiction: wrong next PC, step
breakpoint at an address the real code never reaches, and stepping quietly
stopped working with no diagnostic in GDB, VS Code or the emulator log.

Zeros are also indistinguishable from genuinely zeroed memory, so inspecting
a data structure on a non-resident page yields plausible-looking garbage
rather than a fault.

Propagate the failure as TargetError::NonFatal (an "E xx" reply) so GDB says
"Cannot access memory at address 0x..." and declines the step. This does not
make the memory readable — a debug-path fallback to walking IRIX's page
tables on a TLB miss would, and is a separate change — but it turns a silent
wrong answer into an obvious one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
step() tests check_breakpoint::<Pc>(pc) *before* the fetch. MIPS TLB refill is
software-managed, so a userland PC whose page isn't resident right now is
unreadable — and a breakpoint at such an address therefore fires before the
demand-paging fault that would map the page. That is the normal case, not a
corner case: observed with K:U, EXL=0, EPC=0x1004c758 and no pending
interrupts, i.e. the kernel had just eret'd back into user code whose TLB
entries had been recycled while other processes ran, and the stop was taken on
the very first, not-yet-faulted instruction.

Reporting a stop there strands the debugger on code it cannot read. GDB's MIPS
backend always software-single-steps (read the instruction at PC, compute the
next PC, plant a breakpoint there, continue), so every step becomes a no-op
and the user sees the debugger stuck on one source line with no error
anywhere.

Probe with debug_translate() — the non-faulting path, so the guest isn't
perturbed — and only report the breakpoint if the instruction can actually be
fetched. Otherwise let execution proceed: the fetch takes the TLB exception,
the kernel maps the page and eret's back to this same PC, and the breakpoint
matches again with the code now readable. That is also the more accurate
architectural semantics, since an instruction whose fetch faults never
executed.

The last_bp_hit reset on the not-taken path is required, or the id
check_breakpoint() just recorded would make the *next* stop get misreported as
this breakpoint.

Observed at the address above: translate went from Exception(0x38000008)
(TLBL) to Translated { phys_addr: 0x170ce758 } by the time the stop was
reported, and `dis` showed real instructions instead of "Could not fetch".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…_one

step() checks for a breakpoint match before dispatching the instruction, so
when the CPU sits exactly on an active breakpoint's address — as it always
does right after that breakpoint was hit — step_one() returned
StopReason::SwBreakpoint immediately and the instruction was never executed.
Every subsequent single-step would re-detect the same breakpoint.

run_debug_loop() already handles this with a one-shot skip_breakpoints before
retrying exec.step() on its first iteration; step_one() was simply missing the
equivalent. step() clears the flag itself after every call (it is a genuine
one-shot), so no manual reset is needed.

Note this is a parity fix with no reproduced user-visible failure: measurement
showed stepi does advance the PC even while stopped on a breakpoint, and since
GDB's MIPS backend always software-single-steps (breakpoint + continue, never
the s/vCont;s packet), step_one() may not be reachable from a GDB client at
all. The inconsistency between the two execution paths is real; its impact is
unconfirmed. Drop this commit if you'd rather not carry an unproven change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@kulichbulich

Copy link
Copy Markdown
Contributor Author

Hi,
While porting a game to SGI Irix (my first experience :)), I ran into some debugging issues; this commit improves the debugger's behavior. Thanks for this emulator.
Screenshot from 2026-08-20 01-16-21
Milan

@techomancer

Copy link
Copy Markdown
Owner

yeah the model messed up pc breakpoint test when it adjusted r/w data breakpoints to 64 bit word blocks. silly i didnt catch it. thanks for the changes!

@techomancer
techomancer merged commit 1ac0511 into techomancer:main Aug 21, 2026
2 of 6 checks passed
@techomancer

Copy link
Copy Markdown
Owner

we will need to restore this for data though, this matches mips behavior more closely

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants