[diskann-benchmark] Add type reflection to enable documentation of benchmark inputs.

Cargo.toml

@@ -27,6 +27,7 @@ members = [
     "diskann-record",
     "diskann-tools",
     "diskann-bftree",
+    "diskann-benchmark-runner-derive",
 ]
 
 default-members = [
@@ -66,11 +67,11 @@ diskann-inmem = { path = "diskann-inmem", default-features = false, version = "0
 diskann-disk = { path = "diskann-disk", version = "0.59.0" }
 diskann-label-filter = { path = "diskann-label-filter", version = "0.59.0" }
 # Infra
+diskann-benchmark-runner-derive = { path = "diskann-benchmark-runner-derive", version = "0.59.0" }
 diskann-benchmark-runner = { path = "diskann-benchmark-runner", version = "0.59.0" }
 diskann-benchmark-core = { path = "diskann-benchmark-core", version = "0.59.0" }
 diskann-tools = { path = "diskann-tools", version = "0.59.0" }
 diskann-bftree = { path = "diskann-bftree", version = "0.59.0" }
-diskann-record = { path = "diskann-record", version = "0.59.0" }
 
 # External dependencies (shared versions)
 anyhow = "1.0.98"

diskann-benchmark-runner-derive/Cargo.toml

@@ -0,0 +1,22 @@
+[package]
+name = "diskann-benchmark-runner-derive"
+version.workspace = true
+description.workspace = true
+authors.workspace = true
+license.workspace = true
+edition = "2024"
+
+[lib]
+proc-macro = true
+
+[dependencies]
+syn = { version = "2", features = ["full"] }
+quote = "1"
+proc-macro2 = "1"
+
+[dev-dependencies]
+diskann-benchmark-runner = { workspace = true }
+trybuild = "1.0.120"
+
+[lints]
+workspace = true

diskann-benchmark-runner-derive/SERDE_TODO.md

@@ -0,0 +1,92 @@
+# Serde Support: Resume Notes
+
+The initial Serde attribute parser and code generation are in place. It currently handles
+field and variant names, container-level `rename_all`, and external, internal, and adjacent
+enum representations.
+
+## Correctness
+
+- [X] Use separate Serde-compatible case conversion for fields and variants.
+  - Serde applies different rules in each context.
+  - In particular, Serde converts an enum variant such as `XMLHttpRequest` to
+    `x_m_l_http_request` under `snake_case`, while `heck` produces
+    `xml_http_request`.
+  - For fields, `snake_case` is identity and `kebab-case` replaces underscores.
+  - The `heck` dependency has been removed.
+- [X] Confirm the intended treatment of internally tagged newtype variants.
+  - Newtypes remain supported because Serde accepts newtypes containing struct/map-like
+    values.
+  - Serde compatibility tests cover non-empty and empty struct payloads.
+- [X] Strip the `r#` prefix from raw field and variant identifiers so reflected names match
+  Serde's wire names.
+
+## Attribute validation
+
+- [ ] Reject internally tagged tuple variants at the variant span.
+  - Newtype variants must remain supported because Serde permits newtypes containing
+    struct/map-like values.
+  - Do not classify syntactic newtypes as tuple variants.
+- [ ] Reject `#[reflect(type_name = "...")]` on every generic type, including types whose
+  only generic parameters are lifetimes.
+  - Check `input.generics.params.is_empty()` rather than the generated list of displayed
+    type and const arguments.
+- [ ] Reject adjacent enum representations whose `tag` and `content` names are equal.
+- [ ] Reject fields in internally tagged variants whose effective serialized name conflicts
+  with the internal tag.
+  - Compare names after applying field `rename` and variant-level `rename_all`.
+- [ ] Reject duplicate effective serialized names:
+  - enum variants after container `rename_all` and variant `rename`;
+  - named struct fields after container `rename_all` and field `rename`;
+  - named variant fields after variant-level `rename_all` and field `rename`.
+- [ ] Reject misplaced `reflect` attributes instead of silently ignoring them.
+  - Until field-level features such as `#[reflect(opaque)]` exist, any `reflect` attribute
+    on a field or variant should produce an unsupported/misplaced-attribute diagnostic.
+- [ ] Add deliberate diagnostics for asymmetric Serde naming syntax instead of relying on a
+  lower-level parse error:
+  - `rename(serialize = "...", deserialize = "...")`;
+  - `rename_all(serialize = "...", deserialize = "...")`.
+- [X] Keep unsupported representation-changing attributes rejected until the reflection
+  model explicitly supports them, including `untagged`, `flatten`, `skip*`, `default`,
+  `alias`, `with`, `remote`, `from`, and `try_from`.
+
+## Tests
+
+- [X] Update enum compatibility coverage to expect renamed variants such as `"unit"` rather than
+  `"Unit"`.
+- [X] Verify the generated enum representation, including both `tag` and `content` for an
+  adjacently tagged enum, against serialized JSON.
+- [ ] Add naming tests that compare reflection metadata with `serde_json`, covering:
+  - [X] explicit field and variant `rename`;
+  - [X] struct field `rename_all`;
+  - [X] enum variant `rename_all`;
+  - [X] variant-level `rename_all` for struct-variant fields;
+  - [ ] acronym-heavy variants such as `XMLHttpRequest`;
+  - [X] explicit `rename` taking precedence over `rename_all`.
+- [X] Add representation tests for external, internal, and adjacent tagging.
+- [ ] Add compile-fail tests for duplicate attributes, `content` without `tag`, enum-only
+  attributes on structs, internally tagged tuple variants, unsupported rename rules, and
+  unsupported Serde attributes.
+  - [X] duplicate attributes;
+  - [X] `content` without `tag`;
+  - [X] enum-only attributes on structs;
+  - [ ] internally tagged tuple variants;
+  - [X] unsupported rename rules;
+  - [X] unsupported Serde attributes.
+  - Add fixtures for the remaining validation rules above as their implementations land.
+
+## Cleanup and validation
+
+- [X] Fix the `generate_type_name_body` doctest by returning the final
+  `f.write_str(">")` result instead of discarding it with a semicolon.
+- [X] Run `cargo fmt --all`.
+- [X] Run the targeted derive and runner tests.
+- [ ] Run Clippy with warnings denied once the implementation and tests settle.
+
+## Deferred Serde features
+
+- [ ] Decide how `default` and `alias` should appear in reflection metadata before accepting
+  them.
+- [ ] Continue rejecting asymmetric serialization/deserialization names unless the metadata
+  model represents both.
+- [ ] Continue rejecting `untagged`, `flatten`, `remote`, `with`, `deserialize_with`,
+  `try_from`, and skipped input fields until each has an explicit metadata design.

diskann-benchmark-runner-derive/TODO.md

@@ -0,0 +1,128 @@
+# Benchmark Input Discoverability
+
+The benchmark configuration files are a portable protocol: they support sharing
+configurations across machines, batching benchmarks, validating runs, and analyzing results.
+The missing piece is a human-facing projection of that protocol.
+
+Use dedicated deserialization DTOs as the public configuration boundary. Convert DTOs into
+validated runtime types after deserialization. The derive macro should document and enforce a
+deliberately constrained DTO language rather than attempt general Rust or Serde reflection.
+
+## Initial contract
+
+- [ ] Support named structs.
+- [ ] Support explicitly tagged enums, including unit, tuple, and struct variants where needed.
+- [ ] Require documentation for configuration types, fields, and enum variants.
+- [ ] Support known primitive and domain leaf types.
+- [ ] Support documented DTO composition through selected containers such as `Option<T>` and
+      `Vec<T>`.
+- [ ] Read representation-changing metadata from Serde attributes so Serde remains the source
+      of truth.
+- [ ] Support `rename` and `rename_all`.
+- [ ] Support enum `tag` and `content`.
+- [ ] Decide whether `default` and `alias` should be included in the initial metadata model.
+- [ ] Reject unsupported types and Serde attributes with actionable `syn::Error` diagnostics.
+- [ ] Reject asymmetric serialization and deserialization names unless the metadata model
+      explicitly represents both.
+- [ ] Reject `untagged`, `remote`, `with`, `deserialize_with`, `try_from`, and skipped input
+      fields initially.
+- [ ] Add an explicit escape hatch for intentional opaque or externally implemented leaf
+      types.
+- [ ] Evaluate `flatten` separately; prefer explicit nested DTOs unless flattening provides a
+      clear authoring benefit.
+- [ ] Keep validation and DTO-to-runtime conversion outside the reflection system.
+- [ ] Keep complete examples curated rather than synthesizing arbitrary field values or
+      Cartesian products of enum variants.
+
+## Runtime metadata
+
+- [ ] Replace or refine the prototype `Reflect` API around benchmark configuration
+      documentation rather than general-purpose reflection.
+- [ ] Choose names that communicate the constrained public role, such as `BenchmarkInput` for
+      the derive and `DescribeInput` for the generated runtime trait.
+- [ ] Represent type, field, and variant documentation.
+- [ ] Represent effective serialized names after applying supported Serde rules.
+- [ ] Represent nested DTOs and selected containers.
+- [ ] Represent accepted enum variants and their tagging strategy.
+- [ ] Decide whether metadata should be statically stored or constructed on demand; favor the
+      simpler dynamic model unless measurements justify static storage.
+- [ ] Provide getters or a renderer-facing API for all metadata.
+- [ ] Avoid requiring arbitrary runtime and third-party types to implement the reflection
+      trait.
+
+## Examples
+
+- [ ] Add an API for multiple named, curated examples per registered input.
+- [ ] Include a short description with each example.
+- [ ] Keep examples on the input/DTO API rather than inferring values in the derive macro.
+- [ ] Decide whether the derive accepts an examples function:
+
+  ```rust
+  #[benchmark(examples = Self::examples)]
+  ```
+
+- [ ] Ensure every example serializes and deserializes successfully.
+- [ ] Consider validating examples through the normal DTO-to-runtime conversion path.
+
+## CLI integration
+
+- [ ] Expose input descriptions through the dynamic registry.
+- [ ] Extend `inputs --describe <tag>` or settle on a clearer equivalent command.
+- [ ] Render the input summary, fields, nested objects, enum choices, and important defaults.
+- [ ] Render one or more complete examples.
+- [ ] Consider `skeleton --input <tag>` for generating a directly editable configuration.
+- [ ] Consider emitting a commented JSONC-style template for authoring while retaining strict
+      JSON as the canonical shared representation.
+- [ ] Preserve feature-gated input behavior and diagnostics.
+- [ ] Improve deserialization and validation errors with paths such as
+      `jobs[2].input.search.runs[0].search_l`.
+
+## Macro implementation
+
+- [ ] Replace `todo!` branches with structured compile errors.
+- [ ] Implement named struct generation.
+- [ ] Implement the accepted enum forms.
+- [ ] Generate appropriate generic bounds for nested reflected types.
+- [ ] Extract and normalize literal rustdoc.
+- [ ] Parse the supported Serde subset with `syn::parse_nested_meta`.
+- [ ] Implement and test Serde rename rules used by benchmark DTOs.
+- [ ] Preserve useful source spans in generated diagnostics.
+- [ ] Add explicit diagnostics for every rejected Serde feature.
+- [ ] Add documentation-specific helper attributes only for behavior Serde does not control,
+      such as examples, hiding documentation, or an opaque leaf override.
+- [ ] Do not generate serialization, deserialization, validation, or arbitrary example values.
+
+## Migration
+
+- [ ] Select one simple DTO and one representative complex DTO as the initial vertical slice.
+- [ ] Wire those DTOs through derive, registry, CLI rendering, examples, and validation.
+- [ ] Use the vertical slice to confirm the output format before migrating all inputs.
+- [ ] Convert remaining benchmark-facing inputs to dedicated DTOs where runtime concerns are
+      still mixed into deserialization types.
+- [ ] Add rustdoc to all exposed DTO fields and variants.
+- [ ] Replace custom deserialization patterns where a simpler DTO plus conversion can express
+      the same behavior.
+- [ ] Add explicit overrides only where external or specialized types make them unavoidable.
+- [ ] Migrate remaining registered inputs incrementally rather than requiring an atomic
+      workspace-wide conversion.
+
+## Tests
+
+- [ ] Add unit tests for rustdoc extraction and normalization.
+- [ ] Add tests for each supported Serde naming and tagging rule.
+- [ ] Add compile-fail tests for unsupported item shapes, missing documentation, unsupported
+      Serde attributes, and invalid combinations.
+- [ ] Compare generated names and shapes with actual `serde_json` serialization.
+- [ ] Add CLI golden tests for descriptions, examples, nested DTOs, enums, feature-gated
+      inputs, and error output.
+- [ ] Test that curated examples round-trip.
+- [ ] Run formatting, clippy with warnings denied, and targeted workspace tests.
+
+## Suggested rollout
+
+- [ ] Phase 1: named DTO structs, documentation, known leaves and containers, and CLI rendering.
+- [ ] Phase 2: tagged enums, Serde naming, and curated examples.
+- [ ] Phase 3: migrate representative inputs and refine diagnostics.
+- [ ] Phase 4: add flattening or other Serde behavior only in response to concrete DTO needs.
+- [ ] Phase 5: migrate the remaining benchmark inputs and stabilize the public API.
+