Skip to content

Commit 50eace7

Browse files
loong10kloong10k
committed
接入可重建的 Dart WASM 语法候选
1 parent 3a9af4a commit 50eace7

48 files changed

Lines changed: 206227 additions & 35 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/workflows/ci.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,7 @@ jobs:
3232
- name: Verify Linux WASM worker resource boundary
3333
run: |
3434
cargo test -p codeguard-runtime --features wasm-precheck --test process_contract
35+
cargo test -p codeguard-runtime --features wasm-precheck --test wasm_grammar_load --test dart_upstream_corpus
3536
cargo test -p codeguard-cli --features wasm-precheck --test syntax_worker_candidate --test typescript_syntax_fallback_candidate --test java_syntax_fallback_candidate
3637
3738
- name: Run test suite

‎README.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -94,9 +94,9 @@ cargo build --locked -p codeguard-cli
9494

9595
Version output includes `cli_version`, `target`, and protocol information. Discovery and init preview return observations and unknowns. Dry-run creates neither `AGENTS.md` nor a project data directory. Add Cargo `--offline` only when dependencies are cached.
9696

97-
The workspace requires `std`. The opt-in `wasm-precheck` build feature uses a bounded Rust worker with twenty-two pinned grammar candidates: ArkTS, C, C++, C#, Go, Java, JavaScript, Lua, Luau, Objective-C, Python, Rust, Solidity, TypeScript, TSX, Zig, Nix, Terraform, R, Ruby, PHP, and Kotlin. Java and TypeScript/TSX have narrow single-file fallback paths; `lint python` can provide bounded suspected syntax positions when Ruff is unavailable or no project Ruff configuration is declared. Native Ruff results take priority. It is absent from the published npm `0.1.2` package. No `no_std`, completed project-wide WASM fallback, zero-unsafe, or fastest-runtime claim is made. Runtime OS calls require safety review; a comprehensive security audit is not claimed.
97+
The workspace requires `std`. The opt-in `wasm-precheck` build feature uses a bounded Rust worker with twenty-three pinned grammar candidates: ArkTS, C, C++, C#, Go, Java, JavaScript, Lua, Luau, Objective-C, Python, Rust, Solidity, TypeScript, TSX, Zig, Nix, Terraform, R, Ruby, PHP, Kotlin, and Dart. Java and TypeScript/TSX have narrow single-file fallback paths; `lint python` can provide bounded suspected syntax positions when Ruff is unavailable or no project Ruff configuration is declared. Native Ruff results take priority. It is absent from the published npm `0.1.2` package. No `no_std`, completed project-wide WASM fallback, zero-unsafe, or fastest-runtime claim is made. Runtime OS calls require safety review; a comprehensive security audit is not claimed.
9898

99-
The current source-built CLI's `codeguard grammar status --format=json` reports the coverage gap without loading grammars or running lint: the pinned CodeGraph source has 30 vendored WASM files plus two distinct grammars resolved through its dependency; CodeGuard has twenty-two unqualified asset candidates (ArkTS, C, C++, C#, Go, Java, JavaScript, Lua, Luau, Objective-C, Python, Rust, Solidity, TypeScript, TSX, Zig, Nix, Terraform, R, Ruby, PHP, Kotlin) and zero released syntax capabilities. C, C++, C#, Go, JavaScript, Lua, Luau and Rust are byte-pinned CodeGraph assets that load in the Rust worker but have no public lint route or release qualification. ArkTS, Nix and Terraform are also pinned Rust-loadable candidates with narrow valid/invalid fixtures and no public lint route or release qualification; see the [limited acceptance record](tests/acceptance/arkts-nix-terraform-grammar-candidates.md). The two dependency grammars retain their source bytes and use a byte-pinned legacy `dylink` metadata conversion for Rust loading; they have no public lint route or release qualification. Zig uses a byte-pinned import adaptation so the corrected CodeGraph grammar can load in Rust; it has no public `lint zig` route or release qualification yet. [Zig's limited acceptance record](tests/acceptance/zig-grammar-candidate.md) distinguishes this from released syntax support. Python has a source-build-only, unqualified CLI fallback path; its report remains incomplete and cannot approve delivery. R, Ruby, PHP and Kotlin have pinned CodeGraph bytes, licenses, measured Rust ABI and narrow worker samples; PHP covers mixed HTML/PHP, and all four remain unqualified. Dart remains a gap because its source WASM external scanner import cannot be instantiated by the current Rust worker; see the [limited acceptance record](tests/acceptance/r-ruby-php-kotlin-grammar-candidates.md). The separate [coverage inventory](grammars/codegraph-coverage.json) is not an approved asset manifest. COBOL's 16.4 MB source asset also exceeds the current Rust loader's 8 MiB limit.
99+
The current source-built CLI's `codeguard grammar status --format=json` reports the coverage gap without loading grammars or running lint: the pinned CodeGraph source has 30 vendored WASM files plus two distinct grammars resolved through its dependency; CodeGuard has twenty-three unqualified asset candidates (ArkTS, C, C++, C#, Go, Java, JavaScript, Lua, Luau, Objective-C, Python, Rust, Solidity, TypeScript, TSX, Zig, Nix, Terraform, R, Ruby, PHP, Kotlin, Dart) and zero released syntax capabilities. C, C++, C#, Go, JavaScript, Lua, Luau and Rust are byte-pinned CodeGraph assets that load in the Rust worker but have no public lint route or release qualification. ArkTS, Nix and Terraform are also pinned Rust-loadable candidates with narrow valid/invalid fixtures and no public lint route or release qualification; see the [limited acceptance record](tests/acceptance/arkts-nix-terraform-grammar-candidates.md). The two dependency grammars retain their source bytes and use a byte-pinned legacy `dylink` metadata conversion for Rust loading; they have no public lint route or release qualification. Zig uses a byte-pinned import adaptation so the corrected CodeGraph grammar can load in Rust; it has no public `lint zig` route or release qualification yet. [Zig's limited acceptance record](tests/acceptance/zig-grammar-candidate.md) distinguishes this from released syntax support. Python has a source-build-only, unqualified CLI fallback path; its report remains incomplete and cannot approve delivery. R, Ruby, PHP and Kotlin have pinned CodeGraph bytes, licenses, measured Rust ABI and narrow worker samples; PHP covers mixed HTML/PHP, and all four remain unqualified. The original CodeGraph Dart WASM cannot be instantiated by Rust, but CodeGuard now pins a reproducible Zig rebuild with the real scanner and a byte-pinned WASI import adaptation. Its narrow Rust and worker fixtures pass, while public lint and release remain unqualified; see the [Dart rebuild record](tests/acceptance/dart-grammar-rebuild-candidate.md). The separate [coverage inventory](grammars/codegraph-coverage.json) is not an approved asset manifest. COBOL's 16.4 MB source asset also exceeds the current Rust loader's 8 MiB limit.
100100

101101
### npm installation and one-off use
102102

‎README.zh-CN.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -94,9 +94,9 @@ cargo build --locked -p codeguard-cli
9494

9595
版本输出包含 `cli_version`、`target` 和协议信息。发现与 init 预览返回观察及未知项。Dry-run 不创建 `AGENTS.md` 或项目数据目录。仅在依赖已缓存时为 Cargo 添加 `--offline`。
9696

97-
工作区要求 `std`。可选 `wasm-precheck` 构建特性已用固定的 Java、Python、TypeScript、TSX grammar 候选资产与有界 Rust worker。Java 和 TypeScript/TSX 有局部单文件兜底；`lint python` 在 Ruff 不可用或项目未声明 Ruff 配置时可报告有界疑似语法位置，原生 Ruff 结果优先。公开 npm `0.1.2` 不含该路径。不声称 `no_std`、已完成项目级 WASM 兜底、零 unsafe 或性能最快。运行层系统调用需要安全审查，当前不宣称已完成整体安全审计。
97+
工作区要求 `std`。可选 `wasm-precheck` 构建特性已有二十三份固定 grammar 候选资产与有界 Rust worker。Java 和 TypeScript/TSX 有局部单文件兜底；`lint python` 在 Ruff 不可用或项目未声明 Ruff 配置时可报告有界疑似语法位置，原生 Ruff 结果优先。公开 npm `0.1.2` 不含该路径。不声称 `no_std`、已完成项目级 WASM 兜底、零 unsafe 或性能最快。运行层系统调用需要安全审查，当前不宣称已完成整体安全审计。
9898

99-
当前源码构建的 CLI 可用 `codeguard grammar status --format=json` 只读展示来源覆盖缺口，不加载 grammar 或运行 lint：固定 CodeGraph 来源含 30 份随仓 WASM，另有两种独立 grammar 从其依赖取得；CodeGuard 当前有 C、C++、C#、Go、Java、JavaScript、Lua、Luau、Objective-C、Python、Rust、Solidity、TypeScript、TSX、Zig 十五份未验收资产候选，已发行语法能力为零。C、C++、C#、Go、JavaScript、Lua、Luau、Rust 已作为固定字节的 Rust worker 候选接入，但没有公开 lint 路由或发行验收。Objective-C 与 Solidity 的依赖包原始字节、许可证和固定散列的 `dylink` 元数据转换已入库，可作为 Rust 隔离 worker 的候选观察，但没有公开 lint 路由或发行验收；Zig 也仍是未验收候选。Python 已有仅限源码可选构建的 CLI 候选兜底，结果保持不完整，不能批准交付。[覆盖库存](grammars/codegraph-coverage.json)不是已批准资产清单。COBOL 来源资产约 16.4 MB，还超过当前 Rust 加载器的 8 MiB 限额。
99+
当前源码构建的 CLI 可用 `codeguard grammar status --format=json` 只读展示来源覆盖缺口，不加载 grammar 或运行 lint：固定 CodeGraph 来源含 30 份随仓 WASM，另有两种独立 grammar 从其依赖取得；CodeGuard 当前有二十三份未验收资产候选、零项已发行语法能力。C、C++、C#、Go、JavaScript、Lua、Luau、Rust、ArkTS、Nix、Terraform、R、Ruby、PHP、Kotlin 等已固定字节并由 Rust worker 做窄范围验证，但没有公开 lint 路由或发行验收。Objective-C 与 Solidity 的依赖包原始字节、许可证和固定散列的 `dylink` 元数据转换已入库；Zig 的新版语法也有固定导入适配。CodeGraph 原 Dart WASM 无法由 Rust 直接加载；CodeGuard 现以固定源码和真实外部 scanner 经 Zig 重建，并固定适配前后字节，150 例上游语料错误分类与预期一致，但仍未做 Dart 原生工具对照或发布验收，见[Dart 重建局部验收](tests/acceptance/dart-grammar-rebuild-candidate.md)。Python 已有仅限源码可选构建的 CLI 候选兜底，结果保持不完整，不能批准交付。[覆盖库存](grammars/codegraph-coverage.json)不是已批准资产清单。COBOL 来源资产约 16.4 MB，还超过当前 Rust 加载器的 8 MiB 限额。
100100

101101
### npm 安装与一次性调用
102102

‎crates/codeguard-adapters/examples/adapt_dart_wasm.rs‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
//! 固定 Dart 重建资产的离线发行字节生成器。
2+
3+
use std::path::Path;
4+
5+
use codeguard_adapters::adapt_dart_wasm;
6+
7+
fn main() -> Result<(), Box<dyn std::error::Error>> {
8+
let args = std::env::args().collect::<Vec<_>>();
9+
let [_, source, output] = args.as_slice() else {
10+
return Err("用法：adapt_dart_wasm <source.wasm> <parser.wasm>".into());
11+
};
12+
let source = std::fs::read(Path::new(source))?;
13+
let adapted = adapt_dart_wasm(&source)?;
14+
std::fs::write(Path::new(output), adapted)?;
15+
Ok(())
16+
}

‎crates/codeguard-adapters/src/dart_wasm_compat.rs‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
//! 固定 Dart 重建 WASM 的 Rust WasmStore 导入适配。
2+
3+
use sha2::{Digest, Sha256};
4+
5+
const SOURCE_SHA256: &str = "bbb37cc6aebca30188fab912bb2ae7201604d2b001a64ea6ef2c397674f81f5c";
6+
const ADAPTED_SHA256: &str = "7dad281b3b24924d619cb7059a42b409e7690ebeb8ebbc82882e68167656e012";
7+
const OLD_IMPORT: &[u8] = b"\x10__main_argc_argv";
8+
const NEW_IMPORT: &[u8] = b"\x08args_get";
9+
10+
/// 将固定 Dart 重建 WASM 的同签名 WASI 参数导入改为运行时可提供的名称。
11+
/// 参数为经固定源码构建的 WASM；返回适配字节，来源或结果散列不符时拒绝。
12+
/// 外部 scanner 已和 parser 一起编译；此函数不修改 scanner 或语法表。
13+
pub fn adapt_dart_wasm(source: &[u8]) -> Result<Vec<u8>, String> {
14+
if source.len() != 989331 || format!("{:x}", Sha256::digest(source)) != SOURCE_SHA256 {
15+
return Err("Dart grammar 重建来源字节不符".into());
16+
}
17+
let mut offset = 8;
18+
while offset < source.len() {
19+
let section_start = offset;
20+
let section_id = *source.get(offset).ok_or("WASM section 缺失")?;
21+
offset += 1;
22+
let (section_len, payload_start) = read_leb(source, offset)?;
23+
let payload_end = payload_start
24+
.checked_add(section_len)
25+
.filter(|end| *end <= source.len())
26+
.ok_or("WASM section 越界")?;
27+
if section_id == 2 {
28+
let payload = &source[payload_start..payload_end];
29+
let matches = payload
30+
.windows(OLD_IMPORT.len())
31+
.enumerate()
32+
.filter_map(|(index, bytes)| (bytes == OLD_IMPORT).then_some(index))
33+
.collect::<Vec<_>>();
34+
let [position] = matches.as_slice() else {
35+
return Err("Dart grammar 导入布局不符".into());
36+
};
37+
let mut adapted_payload = Vec::with_capacity(payload.len() - 8);
38+
adapted_payload.extend_from_slice(&payload[..*position]);
39+
adapted_payload.extend_from_slice(NEW_IMPORT);
40+
adapted_payload.extend_from_slice(&payload[position + OLD_IMPORT.len()..]);
41+
let mut adapted = Vec::with_capacity(source.len() - 8);
42+
adapted.extend_from_slice(&source[..section_start + 1]);
43+
write_leb(adapted_payload.len(), &mut adapted);
44+
adapted.extend_from_slice(&adapted_payload);
45+
adapted.extend_from_slice(&source[payload_end..]);
46+
if format!("{:x}", Sha256::digest(&adapted)) != ADAPTED_SHA256 {
47+
return Err("Dart grammar 适配结果不符".into());
48+
}
49+
return Ok(adapted);
50+
}
51+
offset = payload_end;
52+
}
53+
Err("Dart grammar 无导入区段".into())
54+
}
55+
56+
fn read_leb(source: &[u8], mut offset: usize) -> Result<(usize, usize), String> {
57+
let mut result = 0usize;
58+
for shift in (0..35).step_by(7) {
59+
let byte = *source.get(offset).ok_or("WASM section 长度缺失")?;
60+
offset += 1;
61+
result |= usize::from(byte & 0x7f) << shift;
62+
if byte & 0x80 == 0 {
63+
return Ok((result, offset));
64+
}
65+
}
66+
Err("WASM section 长度无效".into())
67+
}
68+
69+
fn write_leb(mut value: usize, output: &mut Vec<u8>) {
70+
loop {
71+
let mut byte = (value & 0x7f) as u8;
72+
value >>= 7;
73+
if value != 0 {
74+
byte |= 0x80;
75+
}
76+
output.push(byte);
77+
if value == 0 {
78+
return;
79+
}
80+
}
81+
}

‎crates/codeguard-adapters/src/grammar_asset_manifest.rs‎

Lines changed: 34 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@ const R_COMMIT: &str = "e9944e9801595ad484f49be492daf0c4c81547ef";
2626
const RUBY_COMMIT: &str = "71bd32fb7607035768799732addba884a37a6210";
2727
const PHP_COMMIT: &str = "5b5627faaa290d89eb3d01b9bf47c3bb9e797dea";
2828
const KOTLIN_COMMIT: &str = "e1a2d5ad1f61f5740677183cd4125bb071cd2f30";
29+
const DART_COMMIT: &str = "d4d8f3e337d8be23be27ffc35a0aef972343cd54";
2930

3031
/// 代码来源、许可和每份候选 grammar 的固定身份。
3132
#[derive(Clone, Debug, Deserialize, Eq, PartialEq)]
@@ -201,6 +202,10 @@ fn verify_bundled_asset(asset: &GrammarAsset) -> Result<&'static [u8], String> {
201202
include_bytes!("../../../grammars/kotlin/parser.wasm").as_slice(),
202203
include_bytes!("../../../grammars/kotlin/LICENSE").as_slice(),
203204
),
205+
"dart" => (
206+
include_bytes!("../../../grammars/dart/parser.wasm").as_slice(),
207+
include_bytes!("../../../grammars/dart/LICENSE").as_slice(),
208+
),
204209
"java" => (
205210
include_bytes!("../../../grammars/java/parser.wasm").as_slice(),
206211
include_bytes!("../../../grammars/java/LICENSE").as_slice(),
@@ -245,6 +250,14 @@ fn verify_bundled_asset(asset: &GrammarAsset) -> Result<&'static [u8], String> {
245250
{
246251
return Err("Zig grammar 来源适配关系不符".into());
247252
}
253+
if asset.language == "dart" {
254+
let source = include_bytes!("../../../grammars/dart/source.wasm");
255+
if asset.source_sha256.as_deref() != Some(digest(source).as_str())
256+
|| crate::adapt_dart_wasm(source)? != wasm
257+
{
258+
return Err("Dart grammar 重建来源适配关系不符".into());
259+
}
260+
}
248261
if matches!(asset.language.as_str(), "objc" | "solidity") {
249262
let source = if asset.language == "objc" {
250263
include_bytes!("../../../grammars/objc/source.wasm").as_slice()
@@ -274,15 +287,15 @@ pub fn parse_grammar_asset_manifest(raw: &[u8]) -> Result<GrammarAssetManifest,
274287
|| manifest.codegraph_license != "LICENSE.codegraph"
275288
|| manifest.codegraph_license_sha256 != CODEGRAPH_LICENSE_SHA256
276289
|| manifest.build_tool
277-
!= "tree-sitter-cli 0.25.10 build --wasm; Zig: tree-sitter-cli 0.27.0 generate --abi 15 + Zig 0.16.0 wasm32-wasi; ArkTS/Terraform: pinned npm prebuilt WASM"
290+
!= "tree-sitter-cli 0.25.10 build --wasm; Zig: tree-sitter-cli 0.27.0 generate --abi 15 + Zig 0.16.0 wasm32-wasi; Dart: pinned parser.c/scanner.c + Zig 0.16.0 wasm32-wasi; ArkTS/Terraform: pinned npm prebuilt WASM"
278291
|| manifest.dependency_package_integrity.as_deref()
279292
!= Some(
280293
"tree-sitter-wasms@0.1.13 sha512-wT+cR6DwaIz80/vho3AvSF0N4txuNx/5bcRKoXouOfClpxh/qqrF4URNLQXbbt8MaAxeksZcZd1j8gcGjc+QxQ==",
281294
)
282295
|| manifest.dependency_package_license.as_deref() != Some("LICENSE.tree-sitter-wasms")
283296
|| manifest.dependency_package_license_sha256.as_deref()
284297
!= Some("6b0382b16279f26ff69014300541967a356a666eb0b91b422f6862f6b7dad17e")
285-
|| manifest.assets.len() != 22
298+
|| manifest.assets.len() != 23
286299
{
287300
return Err("grammar 清单版本、来源或资产数量不符".into());
288301
}
@@ -297,6 +310,7 @@ pub fn parse_grammar_asset_manifest(raw: &[u8]) -> Result<GrammarAssetManifest,
297310
"arkts",
298311
"cpp",
299312
"csharp",
313+
"dart",
300314
"go",
301315
"java",
302316
"javascript",
@@ -362,6 +376,16 @@ pub fn parse_grammar_asset_manifest(raw: &[u8]) -> Result<GrammarAssetManifest,
362376
"csharp/LICENSE",
363377
"778fb7d63b8c1844da315648c02f325c4713a6f4a5d19644fd413421422776d3",
364378
),
379+
"dart" => (
380+
"https://github.com/UserNobody14/tree-sitter-dart",
381+
DART_COMMIT,
382+
"master@d4d8f3e+zig-0.16.0-rebuild",
383+
"dart/parser.wasm",
384+
"7dad281b3b24924d619cb7059a42b409e7690ebeb8ebbc82882e68167656e012",
385+
989323,
386+
"dart/LICENSE",
387+
"d270cb3a4985d75033bd77d875ccebff1d66e32788a3f727891e28d76132dd46",
388+
),
365389
"go" => (
366390
"https://github.com/tree-sitter/tree-sitter-go",
367391
GO_COMMIT,
@@ -576,6 +600,13 @@ pub fn parse_grammar_asset_manifest(raw: &[u8]) -> Result<GrammarAssetManifest,
576600
"160745e470f234cae903a9ba445d19e758d0b02e1197401fc765976c6254d2b6",
577601
)
578602
}
603+
"dart" => {
604+
asset.source_wasm.as_deref() != Some("dart/source.wasm")
605+
|| asset.source_sha256.as_deref()
606+
!= Some(
607+
"bbb37cc6aebca30188fab912bb2ae7201604d2b001a64ea6ef2c397674f81f5c",
608+
)
609+
}
579610
_ => asset.source_wasm.is_some() || asset.source_sha256.is_some(),
580611
})
581612
|| asset.grammar_repository != repo
@@ -589,7 +620,7 @@ pub fn parse_grammar_asset_manifest(raw: &[u8]) -> Result<GrammarAssetManifest,
589620
|| asset.abi_version
590621
!= if matches!(
591622
asset.language.as_str(),
592-
"zig" | "c" | "csharp" | "javascript" | "lua" | "rust" | "nix" | "php"
623+
"zig" | "c" | "csharp" | "javascript" | "lua" | "rust" | "nix" | "php" | "dart"
593624
) {
594625
15
595626
} else {

0 commit comments

Comments
 (0)