Skip to content

[AUTOMATED] [PROPOSAL] Preserve guarded discarded native pointer reads - #1257

Draft
thotypous wants to merge 52 commits into
Noelo-Lab:mainfrom
thotypous:fix/conditional-volatile-read
Draft

thotypous wants to merge 52 commits into
Noelo-Lab:mainfrom
thotypous:fix/conditional-volatile-read

Conversation

@thotypous

@thotypous thotypous commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Problem

An unused native read through an unqualified pointer disappears along with its guards. Missing type metadata is not proof that the read was nonvolatile.

cat >/tmp/conditional-read.c <<'C'
void conditional_read(unsigned state, unsigned mode, unsigned left,
                      unsigned right, void *base) {
    if (state != 7) return;
    if (mode & 1) {
        if ((left >> 2) == (right >> 2)) return;
    } else if (left != 9 && right != 0) return;
    (void)*(volatile unsigned *)((unsigned char *)base + 24);
}
C
clang -O2 -c /tmp/conditional-read.c -o /tmp/conditional-read.o
decompiler/target/release/kuna decompile /tmp/conditional-read.o conditional_read \
  --assert 'prototype conditional_read void conditional_read(unsigned int state,unsigned int mode,unsigned int left,unsigned int right,void *base)' \
  --assert-strict

Unpatched output has only:

{
  return;
}

Fix

  • Preserve discarded native scalar reads through unresolved external pointers as effects at their original CFG position, retaining access width and guards. Keep frame/fixed/thread-storage cleanup, recovered-switch dispatch retirement and same-instruction deduplication.
  • Keep unresolved native read effects independent and alive through emission, including when their computed result is discarded; do not fuse unresolved effects into wider reads. Ordinary resolved-load optimizations remain available.
  • Emit access-local volatile reads in C and Rust without globally qualifying formal parameters. Preserve permitted unaligned accesses; strict/unknown targets retain natural alignment. Discarded Rust reads use MaybeUninit without materializing an initialized integer.
  • Preserve storage bits and ABI classes through split loads, return audits and emission. Same-width floating-register transfers are not numeric integer-to-float conversions.
  • Reconcile C/Rust pointer arguments against final accepted declarations—not earlier inference snapshots—with exact ABI slot/storage/width matching, nested single-evaluation spans, real type dependencies and thread-safe worker transport. Do not mutate IR, caller type evidence or inferred arity to force a cast.
  • Supply self-contained arithmetic concatenation support. Partial scalar reads/writes copy exactly their bytes without evaluating untouched padding; big-endian carriers select the correct significant bytes. Bound x87 extended transfers to their supported storage representation rather than printing a numeric cast.

Validation and current status

The published checkpoint passes 17 focused groups covering guarded reads, invalid inactive pointers, widths, alignment/aliasing, helper hygiene, initialization validity, floating return ABI, integer-return controls, odd-width accesses, extended-float payloads, wide returns, accepted-declaration reconciliation and worker transport. Full generated ARM C and Rust pass the physical native ABI oracle at optimization levels 0 and 2. Actual printed diff3/dpkg caller statements compile against their genuine current callee contracts; this is focused contract proof, not whole-function equivalence.

Process-excluded CPU collection is complete, including an active-workload cost increase. The final-source 113-input corpus shows no new recovery errors or missing functions; 48/48 selected PPC64 entries pass; the complete 770-binary DecBench audit finished without exceptions, with 194 official compilation gains and 14 official losses whose dispositions are in the status comment. Published-head CI failures were test-consumer contracts; the test-only repairs pass stage parity and the affected consumers before push. Older checkpoint results are not presented as current-head results.

Review status: current source-bound results, CPU measurements excluding process overhead, CI status and remaining limitations are tracked in the review status comment. Corpus/benchmark inputs are analyzed, not executed; executable oracles use authored fixtures. Portable scalar accesses do not establish a universal hardware/bus atomicity guarantee.

@thotypous
thotypous marked this pull request as draft October 27, 2026 09:24
@thotypous

thotypous commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor Author

[AUTOMATED] Head 2af85c0e: CI green 2/2 (run 40780864843: parity gates and cargo workspace suite). Now a draft [PROPOSAL] per docs/improvement-pipeline.md: large/multi-part work needs a human go/no-go. @mahaloz, please review the scope (keep whole or split) and approve or reject. Changes since 33703085 are test/fixture/documentation only; every production file is byte-identical. The results below have the source and workload limits stated with them and do not amount to a merge sign-off.

The final committed production source matches the exercised 7232ec43 checkpoint; subsequent changes correct test spans and an explicit unsigned-byte fixture. Canonical executables were rebuilt from 33703085, and archive/member plus cfg(test) equivalence receipts bind the retained passes to that head. Corpus, metric and CPU comparisons use pinned main 5a1033c9, not a moving main revision; signature introducedness is checked against the previously published e76bf67d.

Issues repaired

  • Preserve guarded unresolved native reads independently through emission, including discarded results. Do not fuse unresolved effects into a wider read whose value can subsequently disappear. Resolved frame/fixed/thread-storage and recovered-switch cleanup remain.
  • Close the four ARM floating-return failures in the two Crazyflie copies: same-width storage casts of integer-assembled floating-register bits are not numeric integer-to-float conversions. The repair follows IR and physical ABI evidence, not function names or addresses.
  • Replace the provisional pointer-cast bridge with the existing accepted-final-declaration convention for C and Rust. Earlier inference snapshots can be stale or belong to rejected feedback attempts. Exact callee entry, argument index, physical storage and width must agree; explicit contracts remain authoritative. No IR/prototype/arity mutation forces the cast.
  • Record whole argument spans, preserve single evaluation and valid nesting, and propagate genuine accepted type dependencies through structure renaming and worker transport. Invalid/ambiguous spans or renames relinquish authority instead of falling back to stale types. Pooled text is shared without duplicating nested expressions and uses thread-safe ownership.
  • Remove the introduced diff3 caller mismatch: read_diff retains its printed char ** parameter, and process_diff no longer casts its argument to the wrong word-pointer type. Restore the dpkg maintscript_pre_exec/maintscript_exec printed call/store contract too.
  • Supply missing concatenation support, including implied/nested joins and signed/odd-width operands. Preserve exact bytes in partial scalar reads/writes, including big-endian carriers, without evaluating uninitialized padding. Preserve supported x87 ten-byte storage transfers rather than printing a numeric cast.
  • Retain the earlier Rust null/helper-name, pointer-offset and boolean-expression repairs. Discarded Rust reads use MaybeUninit without assume_init. Permitted unaligned accesses and strict-target natural alignment remain distinct; no universal hardware/bus atomicity guarantee is claimed.

Tests and actual consumers

  • 17 focused groups / 116 executed tests passed and are source-applicable: 14 passing runtime/batch/wire groups from the production checkpoint plus the three corrected groups (24 protocol tests, two native C/Rust pointer consumers and seven shard-synthesis tests). Retained green groups were not needlessly repeated after test-only corrections.
  • Native/generated consumers cover guarded inactive-invalid-pointer paths, widths, alignment/aliasing, helper hygiene, initializedness, post-fusion effect lifetime, full-width pointer arithmetic, Rust word pairs, C concatenation, array-member accesses, ARM floating returns, emitted bitcasts, wide returns and extended storage.
  • Full generated ARM C and Rust pass the physical native ABI oracle at -O0 and -O2, checking eight bit patterns, forwarding, live floating arithmetic and a GPR-returning control. Final 33703085 regeneration produces byte-identical consumer bodies/headers to the passing production checkpoint, so those executions were not needlessly repeated. No prototype assertions, replacement generated bodies, personality stubs or fake runtime support are used.
  • Two actual printed caller-contract consumers compile: the verbatim diff3/dpkg receiver declarations and call/store statements use genuine current callee prototypes and exported record context. This is focused compile-only contract closure, not a claim that the complete benchmark functions or every exported header compile.
  • Actual parallel-worker C/Rust JSON smoke checks complete without producer errors; wire and synthesis tests pass. Parallel-produced getters also pass actual C/Rust native callers at -O0/-O2 in both main and PR: eight successful executions. Only the generated getters are certified by that additional consumer; its forwarding/arithmetic callers are the real authored native fixture. The fully generated parallel-worker forwarding/arithmetic bodies retain the independent-callee-context limitation present in both main and PR.
  • Source-applicable prior controls include actual product Miri checks for discarded uninitialized reads, strict SPARC alignment/opcode checks, real libc consumers, MSan partial-access controls and 49 native extended-float payload pairs. The remaining 13 long-double XML assertions pass; the two incidental expression-shape assertions were retired rather than re-pinned.
  • Catalog prose/count snapshots and dependent capture/rename machinery were removed, not recopied. 106 tooling checks, the actual merge-safety CLI, layout/counter derivation, catalog validation and JSON schema/uniqueness smoke passed. Semantic removal/duplication/footer guards remain.

CPU — process overhead excluded

Independent persistent workers use locked dependencies, two warmups and seven alternating measured rounds on the same pinned CPU. CLOCK_THREAD_CPUTIME_ID measures the named regions. Startup, loading/analysis/inventory and callee-context preparation, serialization and destruction outside those regions are excluded. These are separate context-prepared workload measurements, not end-to-end CLI throughput; do not subtract one row from another.

Selected measured regions Pinned main PR Change
76-function canonical API + plain C emission 1401.129 ms 1395.917 ms −0.372%
76-function API + recorded render + completed-batch reconciliation 1250.373 ms 1252.073 ms +0.136%
Four-function authored active batch, same recorded regions 5.018 ms 6.132 ms +22.196%

Combined medians are medians of the seven per-round sums. Recorded rendering includes take_pointer_arguments/pooled-text capture, normalization and actual type extraction; completed-batch reconciliation includes its real dependency/definition work.

The 76-function recorded workload has zero accepted callee edges and zero inserted casts; its near-flat result does not price the active feature. The authored workload exercises two captured records/two selected edges, one inserted cast and one added/upgraded type. Its combined CPU increases by about 1.114 ms, and all four output bodies differ between main and PR: this is neither an isolated marginal-cast cost nor a whole-program estimate. The small canonical decrease is not an established general speedup. Every measured round is retained.

Final-source corpus and benchmark audit

  • 113 corpus inputs / 19,989 recovered functions, with zero new recovery errors or missing functions against pinned main; the existing 19 function errors and six discovery failures remain. 48/48 explicitly selected PPC64 entries pass. Recovery coverage is not semantic equivalence for all changed bodies. The corpus contains 107 official Debian inputs and six RV32 inputs from a downstream Debian-derived image, not an official signed Debian archive.
  • 770/770 DecBench binaries complete: 95,197 of 96,103 functions recovered (906 unchanged missing), zero metric/backend exceptions. 232 binaries reuse scores only after exact producer-model/C/context/exported-header identity; 538 were scored fresh. The interrupted run resumed only after per-file hash checks; partial scores were not spliced.
  • Versus pinned main: perfect GED 37,479 → 37,480, type 8,996 → 9,010, byte 5,828 → 5,842; means GED 12.68031 → 12.68030 (lower is better), type 0.31943 → 0.31971, byte 0.33260 → 0.33288. 194 official compilation gains and 14 official compilation losses.
  • Historical 3f1a425b reported 334 losses; at the final source 320 of them now compile officially and the remaining 14 are exactly the losses above.
  • Those 14 losses remain official failures. With genuine exported record definitions and emitted support only (no invented types or helpers), 8 compile: 2 e2fsck, 2 Crazyflie, iproute2 make_sci, 3 U-Boot. The six OpenSSH g_stat bodies are byte-identical to main and fail in both arms because a recovered stat * type collides with the libc stat function; main additionally lacked CONCAT44 support. The two U-Boot image_info full exports fail in both arms with the same callee-arity error.
  • All four ARM floating-return records compile with genuine exported context. All 12 signature-change candidates have exact-address declaration shapes identical to the previously proven native/DWARF and e76bf67d dispositions, on byte-identical native inputs.
  • Generic wide-type-header and unsupported extended-float representation limits remain explicit. Original corpus/benchmark programs are analyzed, never executed.

CI repair

Published-head run 40744108246 failed: consumers treated emitted CONCAT support preambles as signatures, one lacked a genuine record definition, and ten stage assertions counted support-macro text or one spelling of scalar-piece reads. Repairs:

  • Signature extraction finds the actual function after support preambles; all register-pair consumers now compile the generated CONCAT support instead of handwritten replacements. The call-result round trip receives the emitted record definitions.
  • The ten spelling assertions are retired and replaced with native consumers: Go boolean low byte vs the original instruction bytes, by-value snapshots, MSVC by-value words, stack-record fields, and a carry-chain page-cross/16-bit-wrap oracle off/on.
  • The stripped Clang wide-string image is no longer a runtime oracle: its vector bodies fail to compile in main and PR alike (wide-projection/return recovery). Symbolful GCC/Clang and stripped GCC remain covered; no prototypes were invented.

Exercised before push, GCC/Clang -O0/-O2 where native: stage parity 3202/3202, PARITY OK; decompile_all_cli 159/159; native stage consumers 3/3; carry-chain oracle; pointer-argument 3/3; cast-narrow; stack-struct 7/7; wide strings 4/4; test layout, tooling unit tests and shared counters. The remaining CI jobs (full workspace suite, spec/catalog/web gates) were not run before the push; the automatic run decides them.

CI on d8078a0f (run 40777019112): the workspace suite passed; stage parity passed; make test-cli failed 301/306. The earlier published-head run never reached that step. All five failures are output-spelling expectations, not arity or argument loss. Four now receive the PR's accepted-declaration pointer casts (overrec(((long *)(v1))), score1(((int *)(v2))), cmp_inner(((unsigned long *)(a0)),…), sub_40117d(a0,((unsigned long *)(a3)))); each fixes a pointee signedness/width mismatch that strict C rejects. The probes accept either form and keep every arity/absence check. In the fifth, simdlane, the body is byte-identical to main, but its seven used CONCAT helpers are now emitted, exceeding a 9,600-byte bound that no longer separates bug from fix. The SUB161 absence and alpha/digit checks remain the discriminator. The updated expectations pass against actual main and PR output. One visible side effect remains for review: main in the callbacktype width fixture now returns *(int *)__builtin_memcpy(&(int){0},&TOTAL,4) where main printed (int)TOTAL.

The updated probes on c72c5620 then failed only the loader's derived probe_id check (IDs hash cmd+expect). 2af85c0e restamps the four IDs, and every tests/cli probe passes the repository loader.

@thotypous
thotypous marked this pull request as ready for review October 28, 2026 22:55
@thotypous thotypous changed the title [AUTOMATED] Preserve guarded discarded native pointer reads [PROPOSAL] conditional-volatile-read: [AUTOMATED] Preserve guarded discarded native pointer reads Oct 28, 2026
@thotypous
thotypous marked this pull request as draft October 28, 2026 22:56
@thotypous thotypous changed the title [PROPOSAL] conditional-volatile-read: [AUTOMATED] Preserve guarded discarded native pointer reads [AUTOMATED] [PROPOSAL] Preserve guarded discarded native pointer reads Oct 28, 2026
@thotypous

Copy link
Copy Markdown
Contributor Author

The agent entered a labyrinth and this got way more complicated than I initially thought it'd be. Although the results on the corpus seem to be okay, this definitely needs some human review.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant