Skip to content

test(suite)!: drive timed designs from shared cases - #1159

Draft
tignear wants to merge 163 commits into
feature/1123from
feature/1123-vtest
Draft

tignear wants to merge 163 commits into
feature/1123from
feature/1123-vtest

Conversation

@tignear

@tignear tignear commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #1158, part of #1123. Shared timed SystemVerilog scripts and independent external adapters stage host inputs before executing procedures, callable continuous drivers or module-input actuals. Declaration initializers retain source order, assignment types, immediate finish and constant-initializer clock baselines. Initializer constant probes reuse packages analyzed from all prepared/preprocessed sources, preserving static clock baselines for cross-file and imported constant functions. Both public signal setters publish host work through the same helper as modify, including eager runtime-event drain settling. Ordered macro definitions flow through source emission, scale validation and top-port metadata using the actual adapter defines.

Continuous-driver temporaries preserve declared destination types and unpacked dimensions. Gated scalar input temporaries also preserve package-qualified formal typedefs, including enum identity and unpacked shape. Built-in packed inputs retain instantiated-port width, and packed instance arrays retain the actual expression's natural width for broadcast/distribution (IEEE1800-2023 23.3.3.5). A nominal shape that cannot be transported produces an emission error instead of silently becoming a vector.

Direct adapter tests cover enum input actuals, unpacked typedef input actuals, continuous enum/unpacked drivers, signed/parameterized instance arrays, staged function values, imports and macro state. The enum input example passes both installed tools before and after the type-preserving change; the unpacked input example genuinely failed before with an array/non-array connection mismatch and passes with Verilator after it. Icarus rejects that original unpacked-function-return shape. Celox's existing callable-actual limitations (#88) are tested through the external APIs without adding backend exclusions.

Validation: library217, suite unit46 plus integration/doctests, all10 live adapter integrations, formatting and affected all-target clippy pass. All128 timing cases run fresh: Icarus124 pass/4 existing ignored; Verilator126 pass/2 existing four-state limitations. New portable zero-trip dependency and three-file package-initializer regressions are registered in the Celox harness; the full timing filter passes520 checks and final stacked SV3156 passes in #1161. Public API setter tests check actual process resumption with/without a concurrent drain and intervening reads/evaluation on all four backends. Retained verification files, proof manifests and exclusions are unchanged. Latest-head CI must finish before merge.

Shared cases could only drive a design through host writes and ticks, so
the SystemVerilog timing controls had no portable cases that run on the
external simulators too.

- Add `(run_until TIME)` and `(run_to_finish)` to the script language and
  `run_until` / `run_to_finish` to the `Backend` contract. A case using
  them is timed (`Design::timed`): the design's own processes, delays and
  clocks run, and only checks may follow `run_to_finish`.
- Generate timed testbenches for Verilator and Icarus: every compiled
  source gets a `1ns/1ps` time scale, the testbench counts picoseconds so
  its settles stay inside one time unit, `run_until` delays to the exact
  time, and the checks after `run_to_finish` run in a `final` block while
  a `forever` delay keeps the simulation alive until the design finishes.
  A simulator's own `$finish` notice is not design output.
- Let a `Simulation` be built on each backend
  (`SimulatorBuilder<Simulation>::build_{native,cranelift,wasm,interpreter}`)
  and read or write four-state values, so Celox runs timed cases on every
  backend through a `Simulation`-backed adapter.
- Add the `timing` SystemVerilog group: 13 cases for `#delay`, `@(event)`,
  `wait`, `always` processes, tasks with timing and four-state edges, and
  refresh the retained verification reports. The Veryl suite's results are
  unchanged under the new time scale.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: df1e561960

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/celox-test-suite/src/script/sv.rs
Comment thread crates/celox-test-suite/src/script/sv.rs Outdated
Comment thread crates/celox-test-suite/src/script/sv.rs Outdated
Comment thread crates/celox-test-suite/src/script/sv.rs
Comment thread crates/celox-test-suite/src/script/sv.rs Outdated
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 9, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-15T19:28:24.542126Z bc01dea New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@codspeed

codspeed Bot commented Oct 9, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

⚠️ 3 benchmarks spent significant time in system calls

System calls cannot be consistently instrumented, so they are not included in the measure, which understates the real cost. Please switch to the Walltime instrument to accurately measure system calls.

Measurement and system calls

✅ 4 untouched benchmarks


Comparing feature/1123-vtest (df1e561) with feature/1123 (4f6e6cf)

Open in CodSpeed

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c81c05793e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/celox-test-suite/src/script/sv.rs Outdated
Comment thread crates/celox-test-suite/src/script/sv.rs
- Settling in the Celox timed adapter also resumes the processes a write
  woke (`Simulation::settle`), as the generated bench's settle step does.
- `run_to_finish` keeps one late event pending instead of a perpetual one;
  when it comes, the run fails as unfinished and skips the final checks.
- Simulator `$finish` notices are filtered only for the staged source files,
  not for every line of their shape.
- The script's top-level bindings are module-scope variables, so the checks
  after `run_to_finish` see them in their `final` block.
- `run_until` rejects a time above `RUN_UNTIL_LIMIT` in every adapter.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5c9b14ced8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/celox-test-suite/src/script/sv.rs Outdated
Comment thread crates/celox/tests/frontends/systemverilog/suite.rs
Comment thread crates/celox-test-suite/src/script/sv.rs Outdated
@tignear
tignear added this pull request to stack #1173 October 10, 2026 10:11

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c98c668625

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/celox/tests/frontends/systemverilog/suite.rs
# Conflicts:
#	crates/celox-test-suite/verification/sv/icarus.json
#	crates/celox-test-suite/verification/sv/verilator.json

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: efa7b07be5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/celox/src/simulation.rs
Comment thread crates/celox/tests/frontends/systemverilog/suite.rs Outdated
Comment thread crates/celox-test-suite/src/script/sv.rs Outdated
# Conflicts:
#	crates/celox/src/simulation.rs

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 641831c37c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/celox-test-suite/src/script/sv.rs

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fba52a3850

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/celox/tests/frontends/systemverilog/suite.rs Outdated
@tignear
tignear added this pull request to the merge queue Oct 10, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue because a pull request earlier in the stack was removed Oct 10, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ed99110802

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/celox-test-suite/src/frontend.rs Outdated
Comment thread crates/celox-test-suite/src/frontend.rs Outdated
Comment thread crates/celox-test-suite/src/script/timed.rs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 756ae2e6d1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/celox-test-suite/src/frontend.rs

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f48ea9d8a8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/celox-test-suite/src/frontend/timed_ff.rs Outdated
Comment thread crates/celox-test-suite/src/frontend/timed_initializers.rs Outdated
Comment thread crates/celox-test-suite/src/frontend/timed_initializers.rs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c5affcfdea

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/celox-test-suite/src/frontend/timed_connections.rs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a51bc365d8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/celox-test-suite/src/frontend/timed_ff.rs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 24c0a54aee

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/celox-test-suite/src/frontend.rs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 2c4f7451a4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/celox-test-suite/src/frontend/timed_connections.rs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e87c219a52

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/celox/src/simulation.rs

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e87c219a52

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread crates/celox/src/simulation.rs
Comment thread crates/celox-test-suite/src/frontend/timed_initializers.rs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bc01dea00b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

/// time: processes waiting for it resume when the simulation next runs.
pub fn set_wide(&mut self, signal: SignalRef, value: num_bigint::BigUint) {
self.simulator.set_wide(signal, value);
self.synchronize_host_write();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Publish direct FF edges through next_event_time

When either new setter drives an always_ff/edge-sensitive input in a design with no waiting processes or queued events, the default non-eager write leaves only pending_host_work; next_event_time() still requires has_waiting_processes(), so it returns None even though step() would dispatch the FF edge at the current time. An event loop driven by next_event_time() therefore stops before applying this write. Fresh evidence beyond the earlier waiter report is this FF-only path, which has no runtime waiter to satisfy that predicate; publish dirty edge work as current-time readiness and cover it with an owning-crate public-API regression.

AGENTS.md reference: AGENTS.md:L51-L53

Useful? React with 👍 / 👎.

@@ -0,0 +1,246 @@
{
"cases": [

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Retain results for every new timed case

The committed timing.vtest contains 128 cases, but this new Icarus report records only 17 of them (and the corresponding Verilator report only 13), while the checked-in matrix and summary were not updated with the remaining timed cases. Consequently the retained verification set and aggregate counts present the new timed coverage as externally validated while omitting most of the changed shared cases, including the later staging regressions; regenerate the focused reports and their indexes after running the changed adapters. This is a coverage-contract change rather than a routine evidence refresh.

AGENTS.md reference: AGENTS.md:L22-L30

Useful? React with 👍 / 👎.

@tignear

tignear commented Oct 16, 2026

Copy link
Copy Markdown
Contributor Author

Converted to draft: superseded by the redesign in #1123 (comment). Kept as a reference implementation; it will be closed when its replacement merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant