Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,9 +94,9 @@ cargo build --locked -p codeguard-cli

Version output includes `cli_version`, `target`, and protocol information. Discovery and init preview return observations and unknowns. Dry-run creates neither `AGENTS.md` nor a project data directory. Add Cargo `--offline` only when dependencies are cached.

The workspace requires `std`. The opt-in `wasm-precheck` build feature uses a bounded Rust worker with fifteen pinned grammar candidates: C, C++, C#, Go, Java, JavaScript, Lua, Luau, Objective-C, Python, Rust, Solidity, TypeScript, TSX, and Zig. Java and TypeScript/TSX have narrow single-file fallback paths; `lint python` can provide bounded suspected syntax positions when Ruff is unavailable or no project Ruff configuration is declared. Native Ruff results take priority. It is absent from the published npm `0.1.2` package. No `no_std`, completed project-wide WASM fallback, zero-unsafe, or fastest-runtime claim is made. Runtime OS calls require safety review; a comprehensive security audit is not claimed.
The workspace requires `std`. The opt-in `wasm-precheck` build feature uses a bounded Rust worker with eighteen pinned grammar candidates: ArkTS, C, C++, C#, Go, Java, JavaScript, Lua, Luau, Objective-C, Python, Rust, Solidity, TypeScript, TSX, Zig, Nix, and Terraform. Java and TypeScript/TSX have narrow single-file fallback paths; `lint python` can provide bounded suspected syntax positions when Ruff is unavailable or no project Ruff configuration is declared. Native Ruff results take priority. It is absent from the published npm `0.1.2` package. No `no_std`, completed project-wide WASM fallback, zero-unsafe, or fastest-runtime claim is made. Runtime OS calls require safety review; a comprehensive security audit is not claimed.

The current source-built CLI's `codeguard grammar status --format=json` reports the coverage gap without loading grammars or running lint: the pinned CodeGraph source has 30 vendored WASM files plus two distinct grammars resolved through its dependency; CodeGuard has fifteen unqualified asset candidates (C, C++, C#, Go, Java, JavaScript, Lua, Luau, Objective-C, Python, Rust, Solidity, TypeScript, TSX, Zig) and zero released syntax capabilities. C, C++, C#, Go, JavaScript, Lua, Luau and Rust are byte-pinned CodeGraph assets that load in the Rust worker but have no public lint route or release qualification. The two dependency grammars retain their source bytes and use a byte-pinned legacy `dylink` metadata conversion for Rust loading; they have no public lint route or release qualification. Zig uses a byte-pinned import adaptation so the corrected CodeGraph grammar can load in Rust; it has no public `lint zig` route or release qualification yet. [Zig's limited acceptance record](tests/acceptance/zig-grammar-candidate.md) distinguishes this from released syntax support. Python has a source-build-only, unqualified CLI fallback path; its report remains incomplete and cannot approve delivery. The separate [coverage inventory](grammars/codegraph-coverage.json) is not an approved asset manifest. COBOL's 16.4 MB source asset also exceeds the current Rust loader's 8 MiB limit.
The current source-built CLI's `codeguard grammar status --format=json` reports the coverage gap without loading grammars or running lint: the pinned CodeGraph source has 30 vendored WASM files plus two distinct grammars resolved through its dependency; CodeGuard has eighteen unqualified asset candidates (ArkTS, C, C++, C#, Go, Java, JavaScript, Lua, Luau, Objective-C, Python, Rust, Solidity, TypeScript, TSX, Zig, Nix, Terraform) and zero released syntax capabilities. C, C++, C#, Go, JavaScript, Lua, Luau and Rust are byte-pinned CodeGraph assets that load in the Rust worker but have no public lint route or release qualification. ArkTS, Nix and Terraform are also pinned Rust-loadable candidates with narrow valid/invalid fixtures and no public lint route or release qualification; see the [limited acceptance record](tests/acceptance/arkts-nix-terraform-grammar-candidates.md). The two dependency grammars retain their source bytes and use a byte-pinned legacy `dylink` metadata conversion for Rust loading; they have no public lint route or release qualification. Zig uses a byte-pinned import adaptation so the corrected CodeGraph grammar can load in Rust; it has no public `lint zig` route or release qualification yet. [Zig's limited acceptance record](tests/acceptance/zig-grammar-candidate.md) distinguishes this from released syntax support. Python has a source-build-only, unqualified CLI fallback path; its report remains incomplete and cannot approve delivery. The separate [coverage inventory](grammars/codegraph-coverage.json) is not an approved asset manifest. COBOL's 16.4 MB source asset also exceeds the current Rust loader's 8 MiB limit.

### npm installation and one-off use

Expand Down
66 changes: 63 additions & 3 deletions crates/codeguard-adapters/src/grammar_asset_manifest.rs
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,9 @@ const CPP_COMMIT: &str = "f41e1a044c8a84ea9fa8577fdd2eab92ec96de02";
const CSHARP_COMMIT: &str = "cac6d5fb595f5811a076336682d5d595ac1c9e85";
const LUA_COMMIT: &str = "816840c592ab973500ae9750763c707b447e7fef";
const LUAU_COMMIT: &str = "a8914d6c1fc5131f8e1c13f769fa704c9f5eb02f";
const ARKTS_COMMIT: &str = "56f7fc288715befe92c734d603f9e6bc3c65d2a8";
const NIX_COMMIT: &str = "3d0173d903e630b6e14d17f1cf79488791379ded";
const TERRAFORM_COMMIT: &str = "fad991865fee927dd1de5e172fb3f08ac674d914";

/// 代码来源、许可和每份候选 grammar 的固定身份。
#[derive(Clone, Debug, Deserialize, Eq, PartialEq)]
Expand Down Expand Up @@ -62,6 +65,8 @@ pub struct GrammarAsset {
pub grammar_commit: String,
/// CodeGraph 记录的上游版本标签。
pub grammar_version: String,
/// 对预编译 npm 资产固定包完整性;其它来源为空。
pub package_integrity: Option<String>,
/// 相对于 grammars/ 的固定制品路径。
pub path: String,
/// WASM 原始字节摘要。
Expand Down Expand Up @@ -140,6 +145,10 @@ fn verify_dependency_license(manifest: &GrammarAssetManifest) -> Result<(), Stri

fn verify_bundled_asset(asset: &GrammarAsset) -> Result<&'static [u8], String> {
let (wasm, license) = match asset.language.as_str() {
"arkts" => (
include_bytes!("../../../grammars/arkts/parser.wasm").as_slice(),
include_bytes!("../../../grammars/arkts/LICENSE").as_slice(),
),
"c" => (
include_bytes!("../../../grammars/c/parser.wasm").as_slice(),
include_bytes!("../../../grammars/c/LICENSE").as_slice(),
Expand Down Expand Up @@ -168,6 +177,10 @@ fn verify_bundled_asset(asset: &GrammarAsset) -> Result<&'static [u8], String> {
include_bytes!("../../../grammars/luau/parser.wasm").as_slice(),
include_bytes!("../../../grammars/luau/LICENSE").as_slice(),
),
"nix" => (
include_bytes!("../../../grammars/nix/parser.wasm").as_slice(),
include_bytes!("../../../grammars/nix/LICENSE").as_slice(),
),
"java" => (
include_bytes!("../../../grammars/java/parser.wasm").as_slice(),
include_bytes!("../../../grammars/java/LICENSE").as_slice(),
Expand Down Expand Up @@ -200,6 +213,10 @@ fn verify_bundled_asset(asset: &GrammarAsset) -> Result<&'static [u8], String> {
include_bytes!("../../../grammars/rust/parser.wasm").as_slice(),
include_bytes!("../../../grammars/rust/LICENSE").as_slice(),
),
"terraform" => (
include_bytes!("../../../grammars/terraform/parser.wasm").as_slice(),
include_bytes!("../../../grammars/terraform/LICENSE").as_slice(),
),
_ => return Err("候选语言未知".into()),
};
verify_grammar_asset(asset, wasm, license)?;
Expand Down Expand Up @@ -237,15 +254,15 @@ pub fn parse_grammar_asset_manifest(raw: &[u8]) -> Result<GrammarAssetManifest,
|| manifest.codegraph_license != "LICENSE.codegraph"
|| manifest.codegraph_license_sha256 != CODEGRAPH_LICENSE_SHA256
|| manifest.build_tool
!= "tree-sitter-cli 0.25.10 build --wasm; Zig: tree-sitter-cli 0.27.0 generate --abi 15 + Zig 0.16.0 wasm32-wasi"
!= "tree-sitter-cli 0.25.10 build --wasm; Zig: tree-sitter-cli 0.27.0 generate --abi 15 + Zig 0.16.0 wasm32-wasi; ArkTS/Terraform: pinned npm prebuilt WASM"
|| manifest.dependency_package_integrity.as_deref()
!= Some(
"tree-sitter-wasms@0.1.13 sha512-wT+cR6DwaIz80/vho3AvSF0N4txuNx/5bcRKoXouOfClpxh/qqrF4URNLQXbbt8MaAxeksZcZd1j8gcGjc+QxQ==",
)
|| manifest.dependency_package_license.as_deref() != Some("LICENSE.tree-sitter-wasms")
|| manifest.dependency_package_license_sha256.as_deref()
!= Some("6b0382b16279f26ff69014300541967a356a666eb0b91b422f6862f6b7dad17e")
|| manifest.assets.len() != 15
|| manifest.assets.len() != 18
{
return Err("grammar 清单版本、来源或资产数量不符".into());
}
Expand All @@ -257,17 +274,20 @@ pub fn parse_grammar_asset_manifest(raw: &[u8]) -> Result<GrammarAssetManifest,
if languages
!= BTreeSet::from([
"c",
"arkts",
"cpp",
"csharp",
"go",
"java",
"javascript",
"lua",
"luau",
"nix",
"objc",
"python",
"rust",
"solidity",
"terraform",
"typescript",
"tsx",
"zig",
Expand All @@ -278,6 +298,16 @@ pub fn parse_grammar_asset_manifest(raw: &[u8]) -> Result<GrammarAssetManifest,
for asset in &manifest.assets {
let (repo, commit, version, wasm_path, wasm_sha, wasm_bytes, license_path, license_sha) =
match asset.language.as_str() {
"arkts" => (
"https://github.com/harmony-contrib/tree-sitter-arkts",
ARKTS_COMMIT,
"0.2.0",
"arkts/parser.wasm",
"db0812971109457d22b3fe9dcdb1cd8e614fba2a338e220670bd254485753623",
4384161,
"arkts/LICENSE",
"048e4dcb7a71fb725495ebb3e7052d0e76dcb6705b828af8b9b461df10bcd8ca",
),
"c" => (
"https://github.com/tree-sitter/tree-sitter-c",
C_COMMIT,
Expand Down Expand Up @@ -368,6 +398,16 @@ pub fn parse_grammar_asset_manifest(raw: &[u8]) -> Result<GrammarAssetManifest,
"luau/LICENSE",
"099c44248f8cf353123211318680e93465587c005a4b3730ce8cb5334de043d6",
),
"nix" => (
"https://github.com/nix-community/tree-sitter-nix",
NIX_COMMIT,
"3d0173d+codegraph-cli-0.25.10",
"nix/parser.wasm",
"4acffa1c013df751193a21ce429777ca214c7d3a7e3665085b6df00dad8869f0",
80876,
"nix/LICENSE",
"c1b72e50266464ec5118d26200e17bf1f472d142e62ab637e1ffc23986657a78",
),
"rust" => (
"https://github.com/tree-sitter/tree-sitter-rust",
RUST_COMMIT,
Expand Down Expand Up @@ -428,9 +468,29 @@ pub fn parse_grammar_asset_manifest(raw: &[u8]) -> Result<GrammarAssetManifest,
"solidity/LICENSE",
"8844f0cc9b76b9c8a9d0251904eb7536b6bb9976e0ec577e8f27ab96d42523ef",
),
"terraform" => (
"https://github.com/tree-sitter-grammars/tree-sitter-hcl",
TERRAFORM_COMMIT,
"1.2.0",
"terraform/parser.wasm",
"0d9ef3ae926acc0411bfecba9b34df4ea659061917b96455570a45a70824e890",
92484,
"terraform/LICENSE",
"c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4",
),
_ => return Err("grammar 语言未知".into()),
};
if asset.dialect != asset.language
|| asset.package_integrity.as_deref()
!= match asset.language.as_str() {
"arkts" => Some(
"tree-sitter-arkts@0.2.0 sha512-j4KpZ21YdX5koXiuuslML24LoKLOzV6uZ1PMG//c/ynAoQfD7XY2LWnFxJiX0sOLqf2pOEYHdUmKFnXjm4QL4g==",
),
"terraform" => Some(
"@tree-sitter-grammars/tree-sitter-hcl@1.2.0 sha512-2bVnOojkkdMLevp0G4v3ksbNoOQFc/Pt9GAdWX4i3aykVyI+CkktE1hsF/XAeUQFjwgGrVZnEyeCll5oD7Ibfg==",
),
_ => None,
}
|| asset.loader_symbol.as_deref()
!= if asset.language == "csharp" {
Some("c_sharp")
Expand Down Expand Up @@ -465,7 +525,7 @@ pub fn parse_grammar_asset_manifest(raw: &[u8]) -> Result<GrammarAssetManifest,
|| asset.abi_version
!= if matches!(
asset.language.as_str(),
"zig" | "c" | "csharp" | "javascript" | "lua" | "rust"
"zig" | "c" | "csharp" | "javascript" | "lua" | "rust" | "nix"
) {
15
} else {
Expand Down
20 changes: 19 additions & 1 deletion crates/codeguard-adapters/tests/grammar_asset_manifest.rs
Original file line number Diff line number Diff line change
Expand Up @@ -63,9 +63,13 @@ fn published_schema_covers_every_current_candidate_and_field() {
#[test]
fn bundled_candidates_pin_source_bytes_license_and_abi_without_claiming_support() {
let manifest = bundled_grammar_candidates().expect("bundled manifest");
assert_eq!(manifest.assets.len(), 15);
assert_eq!(manifest.assets.len(), 18);
for asset in &manifest.assets {
let (wasm, license) = match asset.language.as_str() {
"arkts" => (
include_bytes!("../../../grammars/arkts/parser.wasm").as_slice(),
include_bytes!("../../../grammars/arkts/LICENSE").as_slice(),
),
"c" => (
include_bytes!("../../../grammars/c/parser.wasm").as_slice(),
include_bytes!("../../../grammars/c/LICENSE").as_slice(),
Expand Down Expand Up @@ -94,6 +98,10 @@ fn bundled_candidates_pin_source_bytes_license_and_abi_without_claiming_support(
include_bytes!("../../../grammars/luau/parser.wasm").as_slice(),
include_bytes!("../../../grammars/luau/LICENSE").as_slice(),
),
"nix" => (
include_bytes!("../../../grammars/nix/parser.wasm").as_slice(),
include_bytes!("../../../grammars/nix/LICENSE").as_slice(),
),
"java" => (
include_bytes!("../../../grammars/java/parser.wasm").as_slice(),
include_bytes!("../../../grammars/java/LICENSE").as_slice(),
Expand Down Expand Up @@ -126,6 +134,10 @@ fn bundled_candidates_pin_source_bytes_license_and_abi_without_claiming_support(
include_bytes!("../../../grammars/rust/parser.wasm").as_slice(),
include_bytes!("../../../grammars/rust/LICENSE").as_slice(),
),
"terraform" => (
include_bytes!("../../../grammars/terraform/parser.wasm").as_slice(),
include_bytes!("../../../grammars/terraform/LICENSE").as_slice(),
),
other => panic!("unexpected language {other}"),
};
verify_grammar_asset(asset, wasm, license).expect("pinned candidate bytes");
Expand Down Expand Up @@ -188,6 +200,12 @@ fn missing_provenance_duplicate_fields_and_unknown_languages_are_rejected() {
1,
);
assert!(parse_grammar_asset_manifest(wrong_symbol.as_bytes()).is_err());
let wrong_npm_integrity = original.replacen(
"tree-sitter-arkts@0.2.0 sha512-j4KpZ21YdX5koXiuuslML24LoKLOzV6uZ1PMG//c/ynAoQfD7XY2LWnFxJiX0sOLqf2pOEYHdUmKFnXjm4QL4g==",
"tree-sitter-arkts@0.2.0 sha512-forged",
1,
);
assert!(parse_grammar_asset_manifest(wrong_npm_integrity.as_bytes()).is_err());
}

#[test]
Expand Down
5 changes: 4 additions & 1 deletion crates/codeguard-cli/tests/grammar_status_cli.rs
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ fn codegraph_coverage_is_explicit_and_never_claims_parser_or_gate_completion() {
assert_eq!(report["report_type"], "grammar_coverage_inventory");
assert_eq!(report["codegraph_grammar_count"], 32);
assert_eq!(report["codegraph_vendored_count"], 30);
assert_eq!(report["candidate_count"], 15);
assert_eq!(report["candidate_count"], 18);
assert_eq!(report["released_count"], 0);
assert_eq!(report["authority"], "source_inventory_only");
assert_eq!(report["parser_capability"], "unverified");
Expand All @@ -26,6 +26,7 @@ fn codegraph_coverage_is_explicit_and_never_claims_parser_or_gate_completion() {
assert!(!asset["gap"].as_str().unwrap().is_empty());
}
for language in [
"arkts",
"c",
"cpp",
"csharp",
Expand All @@ -34,10 +35,12 @@ fn codegraph_coverage_is_explicit_and_never_claims_parser_or_gate_completion() {
"javascript",
"lua",
"luau",
"nix",
"objc",
"python",
"rust",
"solidity",
"terraform",
"typescript",
"tsx",
"zig",
Expand Down
30 changes: 30 additions & 0 deletions crates/codeguard-cli/tests/syntax_worker_candidate.rs
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,36 @@ fn cpp_csharp_lua_and_luau_run_in_worker_without_claiming_clean() {
}
}

#[test]
fn arkts_nix_and_terraform_run_in_worker_without_claiming_clean() {
for (language, path, source) in [
(
"arkts",
"src/Main.ets",
b"@Component struct C { build() { Text('hi') } }".as_slice(),
),
("nix", "flake.nix", b"let x = 1; in x".as_slice()),
(
"terraform",
"main.tf",
b"resource \"x\" \"y\" { foo = \"bar\" }".as_slice(),
),
] {
let result = run_syntax_worker_candidate(
env!("CARGO_BIN_EXE_codeguard").as_ref(),
language,
path,
source,
Instant::now() + Duration::from_secs(60),
&AtomicBool::new(false),
)
.expect("grammar worker candidate");
assert!(result.recoveries.is_empty(), "{language}");
assert_eq!(result.precheck.status, SyntaxPrecheckStatus::Incomplete);
assert!(!result.grammar_qualified);
}
}

#[test]
fn invalid_input_and_cancellation_cannot_yield_observations() {
let exe = env!("CARGO_BIN_EXE_codeguard").as_ref();
Expand Down
44 changes: 44 additions & 0 deletions crates/codeguard-runtime/tests/wasm_grammar_load.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,50 @@ const CPP: &[u8] = include_bytes!("../../../grammars/cpp/parser.wasm");
const CSHARP: &[u8] = include_bytes!("../../../grammars/csharp/parser.wasm");
const LUA: &[u8] = include_bytes!("../../../grammars/lua/parser.wasm");
const LUAU: &[u8] = include_bytes!("../../../grammars/luau/parser.wasm");
const ARKTS: &[u8] = include_bytes!("../../../grammars/arkts/parser.wasm");
const NIX: &[u8] = include_bytes!("../../../grammars/nix/parser.wasm");
const TERRAFORM: &[u8] = include_bytes!("../../../grammars/terraform/parser.wasm");

#[test]
fn third_batch_grammars_load_offline_and_parse_basic_fixtures() {
for (name, wasm, sha, abi, valid, invalid) in [
(
"arkts",
ARKTS,
"db0812971109457d22b3fe9dcdb1cd8e614fba2a338e220670bd254485753623",
14,
b"@Component struct C { build() { Text('hi') } }".as_slice(),
b"@Component struct C { build( {".as_slice(),
),
(
"nix",
NIX,
"4acffa1c013df751193a21ce429777ca214c7d3a7e3665085b6df00dad8869f0",
15,
b"let x = 1; in x".as_slice(),
b"let x = ; in x".as_slice(),
),
(
"terraform",
TERRAFORM,
"0d9ef3ae926acc0411bfecba9b34df4ea659061917b96455570a45a70824e890",
14,
b"resource \"x\" \"y\" { foo = \"bar\" }".as_slice(),
b"resource \"x\" {".as_slice(),
),
] {
let mut grammar = WasmGrammar::load(name, wasm, sha, abi)
.unwrap_or_else(|reason| panic!("{name} offline load: {reason}"));
assert!(
!grammar.parse(valid).unwrap().root_node().has_error(),
"{name} valid"
);
assert!(
grammar.parse(invalid).unwrap().root_node().has_error(),
"{name} invalid"
);
}
}

#[test]
fn second_batch_grammars_load_offline_and_parse_basic_fixtures() {
Expand Down
Loading
Loading