Skip to content

[AUTOMATED] Add --progress: live progress over a reproducible project export - #1290

Merged
mahaloz merged 2 commits into
Noelo-Lab:mainfrom
Lukas-Dresel:stream-jobs-address
Oct 28, 2026
Merged

mahaloz merged 2 commits into
Noelo-Lab:mainfrom
Lukas-Dresel:stream-jobs-address

Conversation

@Lukas-Dresel

@Lukas-Dresel Lukas-Dresel commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

decompile-project --stream --jobs N is the only way to get live progress (.streaming, index.jsonl), but it also writes the .c as functions finish, in worker completion order, so two runs of the same command differ. A pipeline that stores the export content-addressed has to choose between progress and reproducible output. --progress gives the progress over an ordinary export:

kuna decompile-project /usr/bin/ssh -o plain --jobs 4
kuna decompile-project /usr/bin/ssh -o p4 --progress --jobs 4
kuna decompile-project /usr/bin/ssh -o p2 --progress --jobs 2
diff -rq plain p4; diff -rq p4 p2
# Only in p4: index.jsonl

While it runs, .streaming reports as it does under --stream, from the load on:

{"schema":1,"phase":"decompiling",...,"jobs":4,"seeds":0,"functions_total":1090,"functions_done":173,"functions_failed":0,"seconds_since_last_result":0,"c_bytes":0,"asm":"pending","error":null}
progress feed artifacts written reproducible under --jobs N
neither flag no at the end yes
--progress yes at the end yes
--stream yes as the run goes, decompile order no

The change:

  • --progress runs the non-stream export unchanged (same plan, structure naming, merge and artifacts) and reports on it. .c/.h/.asm/README.md are byte-identical to the export without the flag.
  • index.jsonl is a feed while the run lasts: one line per function's first result, in completion order, with c_offset/c_len null. At the end it is replaced by the address-ordered index of the final .c, then .streaming is removed.
  • The previous export's .c/.h/.asm are removed once the load succeeds, so a stale body never sits next to a new feed.
  • --stream is unchanged and wins when both flags are given. --progress accepts --assert under the non-stream rules.
  • Hooks: jobs::PoolWatch on the pool's first pass, a landed callback on the serial and callee-first loops, ExportWatch around the export.

Tests: a_progress_export_is_the_non_stream_export_plus_its_index (byte-identity at --jobs 1/2/3), a_progress_export_reports_progress_before_it_writes_the_export (reads the feed mid-run during an injected worker stall), stream_wins_over_progress, and a README render test. Datatests and stage tests unchanged.

One limit, inherited from the non-stream export: a function that hits --max-fn-seconds gets a wall-clock-dependent error string (the pass it was in), so near-budget functions can still differ run to run.

…port

`decompile-project --stream --stream-layout address` runs the non-stream
export unchanged (same serial or pooled decompile, plan, structure naming,
merge and artifacts) and reports on it while it runs: `.streaming` on its
clock, a running README, and `index.jsonl` as a per-function feed in
completion order with null `.c` spans. At the end the four artifacts are the
non-stream export's byte for byte, `index.jsonl` is replaced by the
address-ordered index of that `.c`, and `.streaming` is removed, so the
folder does not depend on --jobs N wherever the non-stream export does not.

The default layout (`decompile`) is unchanged. Hooks: jobs::PoolWatch on the
pool's planned first pass, a `landed` callback on the serial and
callee-first loops, and decompile_project::ExportWatch around the export.

Measured on /usr/bin/ssh (1,090 functions): address-layout runs at --jobs 2
and 4 and a non-stream --jobs 4 run leave identical folders apart from
index.jsonl; CPU time of the two --jobs 4 arms is within noise.

[AUTOMATED] Authored by an agent.
@Lukas-Dresel
Lukas-Dresel marked this pull request as draft October 27, 2026 21:22
Progress reporting and incremental artifacts are two separate things;
`--stream` bundles both, and only the incremental `.c` costs
reproducibility. `--progress` is now its own flag: the status file and the
index.jsonl feed over an ordinary export written at the end. With both
flags the export is the streamed one. `--progress` accepts `--assert`
under the non-stream export's rules. `--stream-layout` is removed.

[AUTOMATED] Authored by an agent.
@Lukas-Dresel Lukas-Dresel changed the title [AUTOMATED] Add --stream-layout address: streamed progress over a reproducible export [AUTOMATED] Add --progress: live progress over a reproducible project export Oct 27, 2026
@Lukas-Dresel
Lukas-Dresel marked this pull request as ready for review October 27, 2026 22:01
@mahaloz
mahaloz merged commit 03ef2e6 into Noelo-Lab:main Oct 28, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants