Skip to content

Pin four more CodeGraph grammar candidates and expose Dart loader gap #25

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
loong10k merged 1 commit into from
Sep 29, 2026
Merged

Pin four more CodeGraph grammar candidates and expose Dart loader gap #25

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,9 +94,9 @@ cargo build --locked -p codeguard-cli

Version output includes `cli_version`, `target`, and protocol information. Discovery and init preview return observations and unknowns. Dry-run creates neither `AGENTS.md` nor a project data directory. Add Cargo `--offline` only when dependencies are cached.

The workspace requires `std`. The opt-in `wasm-precheck` build feature uses a bounded Rust worker with eighteen pinned grammar candidates: ArkTS, C, C++, C#, Go, Java, JavaScript, Lua, Luau, Objective-C, Python, Rust, Solidity, TypeScript, TSX, Zig, Nix, and Terraform. Java and TypeScript/TSX have narrow single-file fallback paths; `lint python` can provide bounded suspected syntax positions when Ruff is unavailable or no project Ruff configuration is declared. Native Ruff results take priority. It is absent from the published npm `0.1.2` package. No `no_std`, completed project-wide WASM fallback, zero-unsafe, or fastest-runtime claim is made. Runtime OS calls require safety review; a comprehensive security audit is not claimed.
The workspace requires `std`. The opt-in `wasm-precheck` build feature uses a bounded Rust worker with twenty-two pinned grammar candidates: ArkTS, C, C++, C#, Go, Java, JavaScript, Lua, Luau, Objective-C, Python, Rust, Solidity, TypeScript, TSX, Zig, Nix, Terraform, R, Ruby, PHP, and Kotlin. Java and TypeScript/TSX have narrow single-file fallback paths; `lint python` can provide bounded suspected syntax positions when Ruff is unavailable or no project Ruff configuration is declared. Native Ruff results take priority. It is absent from the published npm `0.1.2` package. No `no_std`, completed project-wide WASM fallback, zero-unsafe, or fastest-runtime claim is made. Runtime OS calls require safety review; a comprehensive security audit is not claimed.

The current source-built CLI's `codeguard grammar status --format=json` reports the coverage gap without loading grammars or running lint: the pinned CodeGraph source has 30 vendored WASM files plus two distinct grammars resolved through its dependency; CodeGuard has eighteen unqualified asset candidates (ArkTS, C, C++, C#, Go, Java, JavaScript, Lua, Luau, Objective-C, Python, Rust, Solidity, TypeScript, TSX, Zig, Nix, Terraform) and zero released syntax capabilities. C, C++, C#, Go, JavaScript, Lua, Luau and Rust are byte-pinned CodeGraph assets that load in the Rust worker but have no public lint route or release qualification. ArkTS, Nix and Terraform are also pinned Rust-loadable candidates with narrow valid/invalid fixtures and no public lint route or release qualification; see the [limited acceptance record](tests/acceptance/arkts-nix-terraform-grammar-candidates.md). The two dependency grammars retain their source bytes and use a byte-pinned legacy `dylink` metadata conversion for Rust loading; they have no public lint route or release qualification. Zig uses a byte-pinned import adaptation so the corrected CodeGraph grammar can load in Rust; it has no public `lint zig` route or release qualification yet. [Zig's limited acceptance record](tests/acceptance/zig-grammar-candidate.md) distinguishes this from released syntax support. Python has a source-build-only, unqualified CLI fallback path; its report remains incomplete and cannot approve delivery. The separate [coverage inventory](grammars/codegraph-coverage.json) is not an approved asset manifest. COBOL's 16.4 MB source asset also exceeds the current Rust loader's 8 MiB limit.
The current source-built CLI's `codeguard grammar status --format=json` reports the coverage gap without loading grammars or running lint: the pinned CodeGraph source has 30 vendored WASM files plus two distinct grammars resolved through its dependency; CodeGuard has twenty-two unqualified asset candidates (ArkTS, C, C++, C#, Go, Java, JavaScript, Lua, Luau, Objective-C, Python, Rust, Solidity, TypeScript, TSX, Zig, Nix, Terraform, R, Ruby, PHP, Kotlin) and zero released syntax capabilities. C, C++, C#, Go, JavaScript, Lua, Luau and Rust are byte-pinned CodeGraph assets that load in the Rust worker but have no public lint route or release qualification. ArkTS, Nix and Terraform are also pinned Rust-loadable candidates with narrow valid/invalid fixtures and no public lint route or release qualification; see the [limited acceptance record](tests/acceptance/arkts-nix-terraform-grammar-candidates.md). The two dependency grammars retain their source bytes and use a byte-pinned legacy `dylink` metadata conversion for Rust loading; they have no public lint route or release qualification. Zig uses a byte-pinned import adaptation so the corrected CodeGraph grammar can load in Rust; it has no public `lint zig` route or release qualification yet. [Zig's limited acceptance record](tests/acceptance/zig-grammar-candidate.md) distinguishes this from released syntax support. Python has a source-build-only, unqualified CLI fallback path; its report remains incomplete and cannot approve delivery. R, Ruby, PHP and Kotlin have pinned CodeGraph bytes, licenses, measured Rust ABI and narrow worker samples; PHP covers mixed HTML/PHP, and all four remain unqualified. Dart remains a gap because its source WASM external scanner import cannot be instantiated by the current Rust worker; see the [limited acceptance record](tests/acceptance/r-ruby-php-kotlin-grammar-candidates.md). The separate [coverage inventory](grammars/codegraph-coverage.json) is not an approved asset manifest. COBOL's 16.4 MB source asset also exceeds the current Rust loader's 8 MiB limit.

### npm installation and one-off use

Expand Down
68 changes: 66 additions & 2 deletions crates/codeguard-adapters/src/grammar_asset_manifest.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,10 @@ const LUAU_COMMIT: &str = "a8914d6c1fc5131f8e1c13f769fa704c9f5eb02f";
const ARKTS_COMMIT: &str = "56f7fc288715befe92c734d603f9e6bc3c65d2a8";
const NIX_COMMIT: &str = "3d0173d903e630b6e14d17f1cf79488791379ded";
const TERRAFORM_COMMIT: &str = "fad991865fee927dd1de5e172fb3f08ac674d914";
const R_COMMIT: &str = "e9944e9801595ad484f49be492daf0c4c81547ef";
const RUBY_COMMIT: &str = "71bd32fb7607035768799732addba884a37a6210";
const PHP_COMMIT: &str = "5b5627faaa290d89eb3d01b9bf47c3bb9e797dea";
const KOTLIN_COMMIT: &str = "e1a2d5ad1f61f5740677183cd4125bb071cd2f30";

/// 代码来源、许可和每份候选 grammar 的固定身份。
#[derive(Clone, Debug, Deserialize, Eq, PartialEq)]
Expand Down Expand Up @@ -181,6 +185,22 @@ fn verify_bundled_asset(asset: &GrammarAsset) -> Result<&'static [u8], String> {
include_bytes!("../../../grammars/nix/parser.wasm").as_slice(),
include_bytes!("../../../grammars/nix/LICENSE").as_slice(),
),
"r" => (
include_bytes!("../../../grammars/r/parser.wasm").as_slice(),
include_bytes!("../../../grammars/r/LICENSE").as_slice(),
),
"ruby" => (
include_bytes!("../../../grammars/ruby/parser.wasm").as_slice(),
include_bytes!("../../../grammars/ruby/LICENSE").as_slice(),
),
"php" => (
include_bytes!("../../../grammars/php/parser.wasm").as_slice(),
include_bytes!("../../../grammars/php/LICENSE").as_slice(),
),
"kotlin" => (
include_bytes!("../../../grammars/kotlin/parser.wasm").as_slice(),
include_bytes!("../../../grammars/kotlin/LICENSE").as_slice(),
),
"java" => (
include_bytes!("../../../grammars/java/parser.wasm").as_slice(),
include_bytes!("../../../grammars/java/LICENSE").as_slice(),
Expand Down Expand Up @@ -262,7 +282,7 @@ pub fn parse_grammar_asset_manifest(raw: &[u8]) -> Result<GrammarAssetManifest,
|| manifest.dependency_package_license.as_deref() != Some("LICENSE.tree-sitter-wasms")
|| manifest.dependency_package_license_sha256.as_deref()
!= Some("6b0382b16279f26ff69014300541967a356a666eb0b91b422f6862f6b7dad17e")
|| manifest.assets.len() != 18
|| manifest.assets.len() != 22
{
return Err("grammar 清单版本、来源或资产数量不符".into());
}
Expand All @@ -280,11 +300,15 @@ pub fn parse_grammar_asset_manifest(raw: &[u8]) -> Result<GrammarAssetManifest,
"go",
"java",
"javascript",
"kotlin",
"lua",
"luau",
"nix",
"objc",
"php",
"python",
"r",
"ruby",
"rust",
"solidity",
"terraform",
Expand Down Expand Up @@ -358,6 +382,16 @@ pub fn parse_grammar_asset_manifest(raw: &[u8]) -> Result<GrammarAssetManifest,
"java/LICENSE",
"52ed137b039cd9c46409bc22e89938af911c95b157feae2d040b51e6084369a7",
),
"kotlin" => (
"https://github.com/fwcd/tree-sitter-kotlin",
KOTLIN_COMMIT,
"0.3.8",
"kotlin/parser.wasm",
"c80c88867a589a1a0959bcea89de84b7e9684b3693b2cdb2944812458e62ff48",
4052313,
"kotlin/LICENSE",
"948495f61768f7de26bcc61113d8cd95f50bbc15adb678c28c941c6c8fcd5903",
),
"python" => (
"https://github.com/tree-sitter/tree-sitter-python",
PYTHON_COMMIT,
Expand All @@ -368,6 +402,36 @@ pub fn parse_grammar_asset_manifest(raw: &[u8]) -> Result<GrammarAssetManifest,
"python/LICENSE",
"d724405ce238a22c0d35769c5a36b386ad5958192efe8bbb304fb2896254575f",
),
"php" => (
"https://github.com/tree-sitter/tree-sitter-php",
PHP_COMMIT,
"v0.24.2-full-php",
"php/parser.wasm",
"6545a9a110bc878e26ed329950147e190c83da038bb17e999de646fe6c4d6c82",
1058082,
"php/LICENSE",
"664764fabf40c0b318e06324a5a12379092a2df25b8cbf2415f71bc0bb352346",
),
"r" => (
"https://github.com/r-lib/tree-sitter-r",
R_COMMIT,
"v1.2.0",
"r/parser.wasm",
"2a8f5acd1c53d91e0ec5c01a6830d8ac7f5a7f96f0ac4b3768c016c8e9d07711",
481163,
"r/LICENSE",
"de2e49529f03d573bc3fa229dc83acfe22c63a5ad1766b289563edfd45b72dea",
),
"ruby" => (
"https://github.com/tree-sitter/tree-sitter-ruby",
RUBY_COMMIT,
"v0.23.1",
"ruby/parser.wasm",
"4cb5a4b12870876ca864c1e92fe1f5cd47036b2adc083e9306488af88867dbb4",
2106097,
"ruby/LICENSE",
"ee006f02a3d856df282e409be2a86e24a65bb573a98b9c28343771141351bb6b",
),
"javascript" => (
"https://github.com/tree-sitter/tree-sitter-javascript",
JAVASCRIPT_COMMIT,
Expand Down Expand Up @@ -525,7 +589,7 @@ pub fn parse_grammar_asset_manifest(raw: &[u8]) -> Result<GrammarAssetManifest,
|| asset.abi_version
!= if matches!(
asset.language.as_str(),
"zig" | "c" | "csharp" | "javascript" | "lua" | "rust" | "nix"
"zig" | "c" | "csharp" | "javascript" | "lua" | "rust" | "nix" | "php"
) {
15
} else {
Expand Down
18 changes: 17 additions & 1 deletion crates/codeguard-adapters/tests/grammar_asset_manifest.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ fn published_schema_covers_every_current_candidate_and_field() {
#[test]
fn bundled_candidates_pin_source_bytes_license_and_abi_without_claiming_support() {
let manifest = bundled_grammar_candidates().expect("bundled manifest");
assert_eq!(manifest.assets.len(), 18);
assert_eq!(manifest.assets.len(), 22);
for asset in &manifest.assets {
let (wasm, license) = match asset.language.as_str() {
"arkts" => (
Expand Down Expand Up @@ -106,10 +106,26 @@ fn bundled_candidates_pin_source_bytes_license_and_abi_without_claiming_support(
include_bytes!("../../../grammars/java/parser.wasm").as_slice(),
include_bytes!("../../../grammars/java/LICENSE").as_slice(),
),
"kotlin" => (
include_bytes!("../../../grammars/kotlin/parser.wasm").as_slice(),
include_bytes!("../../../grammars/kotlin/LICENSE").as_slice(),
),
"python" => (
include_bytes!("../../../grammars/python/parser.wasm").as_slice(),
include_bytes!("../../../grammars/python/LICENSE").as_slice(),
),
"php" => (
include_bytes!("../../../grammars/php/parser.wasm").as_slice(),
include_bytes!("../../../grammars/php/LICENSE").as_slice(),
),
"r" => (
include_bytes!("../../../grammars/r/parser.wasm").as_slice(),
include_bytes!("../../../grammars/r/LICENSE").as_slice(),
),
"ruby" => (
include_bytes!("../../../grammars/ruby/parser.wasm").as_slice(),
include_bytes!("../../../grammars/ruby/LICENSE").as_slice(),
),
"typescript" => (
include_bytes!("../../../grammars/typescript/parser.wasm").as_slice(),
include_bytes!("../../../grammars/typescript/LICENSE").as_slice(),
Expand Down
1 change: 1 addition & 0 deletions crates/codeguard-cli/src/grammar_status_command.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,7 @@ fn coverage_report() -> Result<Value, String> {
"released":false,
"gap":if candidate.is_some_and(|item| item.codeguard_runtime_validation == "rust_loader_incompatible") {"rust_loader_incompatible"}
else if asset.bytes.is_some_and(|bytes| bytes > 8 * 1024 * 1024) {"current_loader_size_limit"}
else if asset.language == "dart" && candidate.is_none() {"rust_worker_external_scanner_import"}
else if candidate.is_some() {"language_qualification_and_release_pending"}
else {"asset_provenance_license_and_loader_validation_pending"},
})
Expand Down
9 changes: 8 additions & 1 deletion crates/codeguard-cli/tests/grammar_status_cli.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ fn codegraph_coverage_is_explicit_and_never_claims_parser_or_gate_completion() {
assert_eq!(report["report_type"], "grammar_coverage_inventory");
assert_eq!(report["codegraph_grammar_count"], 32);
assert_eq!(report["codegraph_vendored_count"], 30);
assert_eq!(report["candidate_count"], 18);
assert_eq!(report["candidate_count"], 22);
assert_eq!(report["released_count"], 0);
assert_eq!(report["authority"], "source_inventory_only");
assert_eq!(report["parser_capability"], "unverified");
Expand All @@ -33,11 +33,15 @@ fn codegraph_coverage_is_explicit_and_never_claims_parser_or_gate_completion() {
"go",
"java",
"javascript",
"kotlin",
"lua",
"luau",
"nix",
"objc",
"php",
"python",
"r",
"ruby",
"rust",
"solidity",
"terraform",
Expand All @@ -58,6 +62,9 @@ fn codegraph_coverage_is_explicit_and_never_claims_parser_or_gate_completion() {
.unwrap();
assert_eq!(cobol["gap"], "current_loader_size_limit");
assert!(cobol["source_bytes"].as_u64().unwrap() > 8 * 1024 * 1024);
let dart = assets.iter().find(|row| row["language"] == "dart").unwrap();
assert_eq!(dart["integration_status"], "not_integrated");
assert_eq!(dart["gap"], "rust_worker_external_scanner_import");
let zig = assets.iter().find(|row| row["language"] == "zig").unwrap();
assert_eq!(zig["gap"], "language_qualification_and_release_pending");
for language in ["objc", "solidity"] {
Expand Down
27 changes: 27 additions & 0 deletions crates/codeguard-cli/tests/syntax_worker_candidate.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,6 +194,33 @@ fn arkts_nix_and_terraform_run_in_worker_without_claiming_clean() {
}
}

#[test]
fn r_ruby_php_and_kotlin_run_in_worker_without_claiming_clean() {
for (language, path, source) in [
("r", "analysis.R", b"x <- 1\n".as_slice()),
("ruby", "app.rb", b"def f; 1; end\n".as_slice()),
(
"php",
"index.php",
b"<?php function f() { return 1; }".as_slice(),
),
("kotlin", "Main.kt", b"fun main() { val x = 1 }".as_slice()),
] {
let result = run_syntax_worker_candidate(
env!("CARGO_BIN_EXE_codeguard").as_ref(),
language,
path,
source,
Instant::now() + Duration::from_secs(60),
&AtomicBool::new(false),
)
.expect("isolated grammar candidate");
assert!(result.recoveries.is_empty(), "{language}");
assert_eq!(result.precheck.status, SyntaxPrecheckStatus::Incomplete);
assert!(!result.grammar_qualified);
}
}

#[test]
fn invalid_input_and_cancellation_cannot_yield_observations() {
let exe = env!("CARGO_BIN_EXE_codeguard").as_ref();
Expand Down
66 changes: 66 additions & 0 deletions crates/codeguard-runtime/tests/wasm_grammar_load.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,72 @@ const LUAU: &[u8] = include_bytes!("../../../grammars/luau/parser.wasm");
const ARKTS: &[u8] = include_bytes!("../../../grammars/arkts/parser.wasm");
const NIX: &[u8] = include_bytes!("../../../grammars/nix/parser.wasm");
const TERRAFORM: &[u8] = include_bytes!("../../../grammars/terraform/parser.wasm");
const R: &[u8] = include_bytes!("../../../grammars/r/parser.wasm");
const RUBY: &[u8] = include_bytes!("../../../grammars/ruby/parser.wasm");
const PHP: &[u8] = include_bytes!("../../../grammars/php/parser.wasm");
const KOTLIN: &[u8] = include_bytes!("../../../grammars/kotlin/parser.wasm");

#[test]
fn fourth_batch_grammars_load_offline_and_parse_basic_fixtures() {
for (name, wasm, sha, abi, valid, invalid) in [
(
"r",
R,
"2a8f5acd1c53d91e0ec5c01a6830d8ac7f5a7f96f0ac4b3768c016c8e9d07711",
14,
b"x <- 1\n".as_slice(),
b"x <- \n".as_slice(),
),
(
"ruby",
RUBY,
"4cb5a4b12870876ca864c1e92fe1f5cd47036b2adc083e9306488af88867dbb4",
14,
b"def f; 1; end\n".as_slice(),
b"def f(\n".as_slice(),
),
(
"php",
PHP,
"6545a9a110bc878e26ed329950147e190c83da038bb17e999de646fe6c4d6c82",
15,
b"<?php function f() { return 1; }".as_slice(),
b"<?php function f( {".as_slice(),
),
(
"kotlin",
KOTLIN,
"c80c88867a589a1a0959bcea89de84b7e9684b3693b2cdb2944812458e62ff48",
14,
b"fun main() { val x = 1 }".as_slice(),
b"fun main( {".as_slice(),
),
] {
let mut grammar = WasmGrammar::load(name, wasm, sha, abi)
.unwrap_or_else(|reason| panic!("{name} offline load: {reason}"));
assert!(
!grammar.parse(valid).unwrap().root_node().has_error(),
"{name} valid"
);
assert!(
grammar.parse(invalid).unwrap().root_node().has_error(),
"{name} invalid"
);
}
let mut php = WasmGrammar::load(
"php",
PHP,
"6545a9a110bc878e26ed329950147e190c83da038bb17e999de646fe6c4d6c82",
15,
)
.expect("full PHP grammar");
assert!(
!php.parse(b"<div>Hi</div><?php echo 1; ?>")
.unwrap()
.root_node()
.has_error()
);
}

#[test]
fn third_batch_grammars_load_offline_and_parse_basic_fixtures() {
Expand Down
Loading
Loading