Skip to content

Built-in assemblers for all ISAs, ISA registry, and simulator restructuring - #6

Merged
timsherwood merged 18 commits into
mainfrom
claude/mapachespim-simulator-review-seyv8i
Sep 27, 2026
Merged

Built-in assemblers for all ISAs, ISA registry, and simulator restructuring#6
timsherwood merged 18 commits into
mainfrom
claude/mapachespim-simulator-review-seyv8i

Conversation

@timsherwood

@timsherwood timsherwood commented Sep 27, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

This merges two rounds of work into main:

  1. The commits from claude/simulator-code-review-vbujym: bug fixes from a code review, built-in pure-Python assemblers for all four ISAs that match GNU as byte for byte (Keystone removed), examples shipped in the package, and updated CI/release workflows.
  2. A restructuring so each ISA is described in one place, plus bugs found while doing it and while getting CI green.

ISA registry

ISA-specific facts used to be spread across if isa == ... branches in the simulator, disassembler, ELF loader and writer, assembler, directive parser, console, CLIs, and examples. They now live in:

  • mapachespim/isa.py — one ISASpec per ISA: names/aliases, memory layout, registers, syscall registers and instruction, instruction sizes, ELF machine/flags, GNU as dialect, trap descriptions
  • mapachespim/engines.py — Unicorn and Capstone constants per ISA
  • mapachespim/toolchain/targets.py — a Target per ISA that drives its encoder (options, nop padding, ARM64 literal pools, x86 jump relaxation)

docs/dev/architecture.md describes the layout and has a checklist for adding an ISA.

Other restructuring

  • The simulator is split into syscalls.py (table-driven, testable without the emulator), disassembler.py, symbols.py, and debug_info.py (moved out of the console).
  • Simulator.run_until() with StopReason / RunResult replaces three separate run loops (simulator, console run, -e mode).
  • The ELF32 and ELF64 writers share one code path.
  • The loader exposes section headers, so the console no longer opens ELF files itself.
  • Removed duplicates: a second AssemblyResult class, two .isa-directive scanners, four hard-coded ISA lists, repeated console argument checks.

The assembler output was verified byte-identical to the previous code for every example (with and without debug info) before the symbol-table fix below.

Bugs fixed

  • ARM64 registers x29 (fp) and x30 (lr) read and wrote vector registers V0/V1: Unicorn does not number x29/x30 after x28.
  • ARM64 read_int/read_char crashed because x0 was treated as a read-only zero register (arm/guess_game crashed on its first input).
  • 32-bit ARM and RISC-V ELF files were run as 64-bit programs; they are now refused.
  • The assembler's ELF symbol tables listed locals after a global (readelf warns) and put every symbol, including data labels, in .text. The example binaries are rebuilt.
  • A single out-of-range file index in DWARF line info discarded all debug info.
  • Found by this PR's first CI run:
    • RISC-V branch relaxation could lengthen a branch GNU as keeps short: layout used the previous pass's address for every label, while GNU uses the current pass's address for labels already passed. Found by the 50× fuzz job; regression test added at normal scale.
    • Tab completion on Windows offered examples\arm/ for load examples/.
    • Tests in test_isa_specification.py deleted still-open temp files, which Windows refuses.

Breaking API changes

Removed: mapachespim.toolchain.memory_map (use mapachespim.memory_map), the SailSimulator alias, the per-ISA *Config classes, STACK_TOP/STACK_SIZE/STACK_ADDR, and elf_loader.Architecture. Listed in CHANGELOG.md.

Tests

723 tests. New:

  • test_isa_conformance.py — every registered ISA assembles and runs programs exercising every syscall and every register mapping (catches the x29/x30 bug)
  • test_isa_registry.py — the registry, engine bindings, assembler targets, and examples are consistent
  • test_syscalls.py, test_simulator_parts.py — syscall handler, symbol table, disassembler, run_until
  • test_elf_output.py — ELF headers/symbol tables are well formed, bundled binaries match the assembler, loader rejects unsupported files
  • The guess_game tests use the real syscall path via sim.stdin/sim.stdout instead of reimplementing syscalls

CI is green on Linux, Windows, and macOS (Python 3.9, 3.11, 3.13), lint, and the 50× GNU as fuzz job.

🤖 Generated with Claude Code

https://claude.ai/code/session_01D764BuGwyCPv1b2tstwGAS

claude added 18 commits September 27, 2026 00:07
@claude
Fix simulator and console bugs found in code review
742d95d 
Simulator:
- Perform syscalls when single-stepping (step used to skip them, so
  stepping printed nothing and ran past exit)
- read_char read two lines and returned the second; it now buffers one
  line and returns characters in order
- print_int sign-extends 32-bit MIPS registers (-5 printed as 4294967291)
- reset reloads the ELF instead of leaving memory empty
- Fault messages name the access and address ("Read from unmapped
  address 0x50000000") instead of Unicorn's generic error
- Stop mapping address 0 on ARM so null dereferences fault
- Back the RISC-V sign-extended alias region with the same memory, so
  writes through lui-built addresses are visible at 0x8xxxxxxx
- Track exit state and exit code; unknown syscalls are errors
- Add SPIM exit2 (17); stdin/stdout are injectable for tests/grading
- Loading a new ELF starts from a fresh machine and may change ISA
- Move memory_map to the package top level so the simulator and
  assembler share one layout (stack placement now follows it)

Console:
- disasm walks x86-64 by real instruction length and marks the PC
- step shows full x86 instruction bytes and keeps program output on
  its own line
- run/step report exit code, fault reason, and source line
- mem/disasm/delete accept symbol names
- regs decimal is signed; set show-changes works; set completes
  output-spacing; list suggests mapachespim-as -g
- Add --version and --max-steps; -e exits with the program's exit
  code, 1 on error, 124 when the instruction limit is hit

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GPT2SED9nsbBHaDXjUEa41
@claude
Add built-in RISC-V encoder and make Keystone optional
f3036ed 
The Keystone release on PyPI (0.9.2) has no RISC-V support, so RISC-V
programs could not be assembled on any machine that installed
MapacheSPIM from pip; this was the cause of all 25 failing tests.

- Add toolchain/riscv.py: a table-driven RV64IM encoder with Zicsr,
  fence/ecall/ebreak, the standard pseudo-instructions, symbol
  expressions, and %hi/%lo. li now handles full 64-bit constants
  (the old lui+addi expansion was wrong for values like 0x7fffffff
  and anything above 32 bits)
- Output is byte-identical to the committed example binaries,
  including debug info
- Label layout for RISC-V uses the encoder's own sizing, and pass 2
  now errors if any label moved instead of emitting bad offsets
- .equ/.set constants can be used as instruction operands
- Keystone is only installed automatically on x86 machines, where
  PyPI has prebuilt wheels; elsewhere (Apple Silicon, ARM Linux) it
  is an optional extra instead of a source build that fails.
  Assembling ARM64/x86-64/MIPS without it gives an actionable error
- Tests that need Keystone are skipped (not failed) when it is absent
- mapachespim-as --version reports the package version

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GPT2SED9nsbBHaDXjUEa41
@claude
Add built-in MIPS32 encoder so MIPS needs no Keystone
100e172 
MIPS is the ISA SPIM itself simulates, so it should assemble on every
platform, including Apple Silicon where Keystone has no wheels.

- Add toolchain/mips.py: MIPS32 integer instructions plus the common
  SPIM/GNU pseudo-instructions (li, la, move, not, neg, abs, b, beqz,
  blt/bgt/ble/bge and unsigned variants, 3-operand div/rem, mul by a
  constant, loads/stores from labels). Branches and jumps get a
  delay-slot nop, matching GNU as in ".set reorder" mode
- Output is byte-identical to the committed MIPS example binaries
- Remove the old Keystone-based MIPS expansion, including its
  hard-coded branch-offset workarounds and a sizing bug (li with a
  value in 32768..65535 was sized as 1 instruction but emitted 2)
- Move shared operand parsing into toolchain/expr.py
- Sizing passes now use real addresses and ignore range checks on
  not-yet-resolved labels, so layout is exact
- Clearer errors: "add t0, t1, t2" says MIPS registers need a $;
  RISC-V "add a0, a1, 5" is accepted as addi (as in GNU as)
- Only ARM64 and x86-64 still need Keystone

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GPT2SED9nsbBHaDXjUEa41
@claude
Ship examples in the package; load .s files and examples by name
8eb74ab 
After `pip install`, the examples referenced throughout the docs did
not exist, and students had to run mapachespim-as separately before
every load.

- Wheels now include examples/ as mapachespim/examples; the new
  mapachespim.examples module finds them in wheels and checkouts
- load accepts assembly sources (.s/.S/.asm), assembling them with
  debug info (ISA from .isa or `load prog.s riscv64`), with clear
  messages for a missing ISA and for assembly errors
- load resolves example names (riscv/fibonacci) and the documented
  examples/... paths even outside the repository
- New commands: examples (list bundled programs) and reload
  (re-assemble after editing; breakpoints on labels follow them)
- New flags: --copy-examples [DIR], --isa for sources without .isa;
  -e accepts sources and example names
- Tab completion offers example names; help/quickstart updated
- Fix DWARF for big-endian targets: debug info was always written
  little-endian, so `list` never worked for MIPS. MIPS example
  binaries are rebuilt (only .debug_info/.debug_line change)
- Handle BrokenPipeError when -e output is piped to e.g. head
- Friendlier message when Keystone is missing for ARM64/x86-64

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GPT2SED9nsbBHaDXjUEa41
@claude
Fix lint and type errors reported by CI
6441fd1 
- Undefined SectionData annotations, bare excepts, unused imports and
  loop variables, a no-op 'or True' condition
- Type ISA_CONFIG and byte-order literals so mypy passes

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GPT2SED9nsbBHaDXjUEa41
@claude
Apply ruff format
3deabb1 
Formatting only; no functional changes. CI runs ruff format --check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GPT2SED9nsbBHaDXjUEa41
@claude
Update CI, add release workflow, and refresh documentation
5dd75ee 
CI:
- Test on Linux, Windows, and macOS (Apple Silicon, where Keystone is
  not installed) on Python 3.9 and 3.13, from the built wheel, with a
  smoke test run outside the repository
- Run on all pull requests and on demand (workflow_dispatch)

Release:
- New release workflow publishes to PyPI with trusted publishing when
  a GitHub release is published; docs/RELEASING.md has the one-time
  setup and steps

Packaging:
- Require Python 3.9+ (3.8 is end-of-life); fix project URLs; add
  classifiers; absolute README links so the PyPI page renders

Console:
- The first regs after loading marks registers changed since the
  program started; ★ falls back to * where the terminal can't show it,
  and output never crashes on unencodable characters
- `.isa riscv64  # comment` is recognized

Docs:
- README, quick start, console guide, syscall reference, examples
  guide, and Python API docs rewritten for the new install and
  workflow; every transcript was checked against real output
- Add CHANGELOG.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GPT2SED9nsbBHaDXjUEa41
@claude
Test built-in assemblers against GNU as and match it byte for byte
e0dd3fb 
Add differential tests that generate randomized programs covering every
supported RISC-V and MIPS instruction form (all registers, immediates
across their full ranges and edges, forward/backward labels) and require
byte-identical output from GNU binutils; invalid inputs must be rejected
by both. MAPACHESPIM_FUZZ_SCALE scales them up (about 400k instructions
compared at 200x). CI installs binutils and runs a 50x job.

Changes to match GNU as:
- RISC-V: li uses a port of GNU's load_const; call/tail are always
  auipc+jalr; out-of-range branches are relaxed; %hi range checked
- MIPS: la is lui+addiu; compare-branches follow GNU's macro rules
  (including constant folding); abs, mul by constant, loads/stores
  with symbols or large offsets; div/rem traps on divide by zero and
  overflow like SPIM; .set noreorder; jalr/bgezal register checks.
  Two-operand div/divu keep SPIM semantics (documented exception)

Bugs found and fixed:
- .align n meant n bytes; GNU uses 2^n on RISC-V/MIPS/ARM
- MIPS data was little-endian when the ISA came from --isa
- labels in .word/.dword became 0; bad values were silently ignored
- "\xNN" in strings emitted two bytes; octal escapes unsupported
- custom sections were never written to the ELF
- MIPS .half/.word are auto-aligned as GNU does
- break codes were encoded in the wrong field

Also: an expression evaluator with GNU semantics (precedence, octal,
'.', name = expr), data/alignment directives in .text, and clear
"Division by zero" messages for MIPS traps. Example binaries rebuilt;
all 20 produce identical output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GPT2SED9nsbBHaDXjUEa41
@claude
Split source lines like GNU as: comments, ';', and local labels
4b58398 
- Comment stripping ignored quotes, truncating strings containing '#',
  ';' or '//'; it now skips string and character literals
- ';' separates statements (it was treated as a comment, silently
  dropping everything after it)
- On ARM64, '#' starts a comment only at the beginning of a line, since
  it marks immediates; '//' and /* ... */ comments work everywhere
- Several labels on one line, .L local labels, and GNU numeric local
  labels (1: ... b 1b / 1f) are supported
- Defining the same label twice is an error

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GPT2SED9nsbBHaDXjUEa41
@claude
Add built-in ARM64 assembler matching GNU as; fix ARM examples
cdc1f1e 
The Keystone-based ARM64 path accepted non-standard code: '#' comments
(which GNU as reads as immediates) and 'adr' to data more than 1MB away,
silently rewritten into movz/movk. None of the ARM examples assembled
with the real ARM toolchain.

- Add toolchain/arm64.py: A64 integer instructions (arithmetic, logic
  with bitmask immediates, moves, shifts/bitfields, multiply/divide,
  conditional select/compare, loads/stores in all addressing modes,
  pairs, branches, adr/adrp with :lo12:, svc/brk/nop), choosing the
  same encodings as GNU as (movz/movn/orr for mov, sub for add of a
  negative, cmn for cmp of a negative, ldur for unaligned offsets)
- 'ldr x0, =value' uses a literal pool laid out like GNU's (dumped at
  .ltorg/.pool and at the end of .text, deduplicated, 4-byte entries
  then 8-byte entries)
- Differential tests against GNU as for every instruction family (match
  at 100x scale), plus an exhaustive round-trip test of all logical
  immediate encodings
- Convert the ARM examples to standard syntax: '//' comments and
  'ldr xN, =label' for data addresses. Rebuilt; they are identical to
  GNU's output and behave exactly as before
- adr out of range now explains what to use instead
- CI installs AArch64 binutils for the differential tests

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GPT2SED9nsbBHaDXjUEa41
@claude
Add built-in x86-64 assembler matching GNU as; remove Keystone
e53c7f4 
All four ISAs are now assembled by pure-Python encoders, so the Keystone
dependency (no ARM wheels, no RISC-V support) is gone.

x86-64 (toolchain/x86.py):
- AT&T and Intel syntax, detected per instruction or set with
  .att_syntax/.intel_syntax; GNU's choice of encodings (short forms,
  accumulator forms, xchg special cases, truncated immediates).
- GNU's jump relaxation: jumps start short and grow to rel32 only when
  out of range, seeing labels during a pass the way relax_segment does.

Also found by extending the differential tests to ARM64 and x86-64:
- Alignment in code is padded like GNU as on each ISA (x86 multi-byte
  nops with a jump over long runs, ARM64 zeros then nops, RISC-V c.nop);
  ARM64 aligns an instruction that follows data.
- .p2align/.balign max-skip argument (empty arguments were dropped).
- .fill directive; .word is 2 bytes on x86.
- ISA aliases (mips, x64, aarch64) were not recognized by the directive
  parser, so e.g. --isa mips produced little-endian data.

Tests: GNU differential tests for x86-64 and for alignment/data in code on
all ISAs, x86 unit tests; CI installs x86-64 binutils. Docs updated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GPT2SED9nsbBHaDXjUEa41
@claude
Keep ARM64 literal-pool test within GNU's pool size at large fuzz scales
e5ef4eb 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GPT2SED9nsbBHaDXjUEa41
@claude
Ignore coverage output
9c45ff1 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GPT2SED9nsbBHaDXjUEa41
@claude
Describe each ISA once in a registry that every layer uses
887ea99 
ISA-specific facts were spread across if/elif chains in the simulator,
disassembler, ELF loader and writer, assembler, directive parser, console,
CLIs, and examples. They now live in one place:

- isa.py: ISASpec per ISA (names/aliases, memory layout, register file,
  syscall ABI and instruction, instruction sizes, ELF machine/flags, GNU as
  dialect, trap descriptions) and lookup helpers
- engines.py: the Unicorn and Capstone constants for each ISA
- toolchain/targets.py: a Target per ISA that drives its encoder (options,
  code padding, ARM64 literal pools, x86 jump relaxation)

The simulator is split into focused modules: syscalls.py (table-driven
SPIM syscalls with a ProgramIO object), disassembler.py, symbols.py
(bisect-based SymbolTable), and debug_info.py (moved out of the console).
Syscall instructions are detected before execution instead of with a
per-instruction hook. run_until() with StopReason replaces three separate
run loops in the simulator and console. The ELF writer builds 32- and
64-bit files from one code path, and the loader exposes section headers so
the console no longer opens ELF files itself.

Bugs fixed along the way:
- ARM64 registers 29 and 30 (fp, lr) read Unicorn's V0/V1 vector
  registers, since Unicorn does not number x29/x30 after x28
- ARM64 read_int/read_char crashed: x0 was treated as a read-only zero
  register
- Loading a 32-bit ARM or RISC-V ELF ran it as a 64-bit program
- The ELF symbol table listed locals after globals (readelf warns) and put
  every symbol, including data labels, in .text; the example binaries are
  rebuilt with the fixed writer
- DWARF line entries with an out-of-range file index discarded all debug
  info

Tests: a conformance suite that every registered ISA must pass (syscalls,
every register mapping, register API, default memory map), registry
consistency tests, SyscallHandler tests with a fake machine, SymbolTable,
Disassembler, run_until, ELF symbol-table validity, loader checks, and a
check that the bundled binaries match the assembler. The guess_game tests
now use the real syscall path instead of reimplementing it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D764BuGwyCPv1b2tstwGAS
@claude
Share console argument checks; document the architecture
728c522 
- Console: _require_program() and _parse_positive() replace the
  "no program loaded" and positive-count checks repeated across commands;
  disasm's address/count parsing is no longer duplicated per branch
- docs/dev/architecture.md: module map, how programs are run and
  assembled, and a checklist for adding an ISA
- tests/README.md rewritten: it described a Sail/C backend and test counts
  from long ago
- Python API docs, changelog, and README updated for the new modules

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D764BuGwyCPv1b2tstwGAS
@claude
Keep conformance-test ELF files in pytest's tmp_path
67972c7 
They were written with mkstemp and never deleted.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D764BuGwyCPv1b2tstwGAS
@claude
Fix tab completion paths and temp-file tests on Windows
5f1f6f8 
- complete_load: glob joins with backslashes on Windows, so completing
  "examples/" gave "examples\arm/", which no longer extends the typed
  text. Keep forward slashes unless the user typed backslashes. New test
  simulates Windows' glob output.
- test_isa_specification: the TestAssembleFile tests unlinked
  NamedTemporaryFile files while they were still open, which Windows
  refuses. Use pytest's tmp_path instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D764BuGwyCPv1b2tstwGAS
@claude
Size RISC-V branches with GNU's view of label addresses
fa01a57 
The layout passes sized every instruction with the previous pass's label
addresses. GNU's relax_segment gives labels already passed in the current
pass their new address and only labels ahead their old one, so a branch
could be relaxed to bne+jal here but not by GNU. The 50x differential fuzz
run in CI found such a program (test_branches); a model of the two rules
matched GNU on 160 of 160 random branch programs with GNU's rule and 146
with the old one.

Instruction sizing now uses the same _RelaxView the x86 relaxation uses,
with no stretch. New regression test at normal fuzz scale.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D764BuGwyCPv1b2tstwGAS
@timsherwood
timsherwood merged commit 75e2643 into main Sep 27, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@timsherwood @claude