Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions .github/workflows/generate-pages.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,15 @@ on:
branches: [main]
paths:
- "docs/**"
- "kb/communities/**"
- "data/isolates/**"
- "data/text_map/**"
- "conf/text_map.yaml"
- "src/communitymech/text_map_*.py"
- "scripts/stage_text_map.py"
- "scripts/embedding_pipeline.py"
- "scripts/check_graph_receipts.py"
- "src/communitymech/graph_embedding_receipts.py"
- ".github/workflows/generate-pages.yaml"
workflow_dispatch:

Expand All @@ -33,11 +42,24 @@ jobs:
- name: Checkout
uses: actions/checkout@v4

- name: Set up Python for map validation
uses: actions/setup-python@v5
with:
python-version-file: .python-version

- name: Validate and stage the configured semantic text map
run: |
python -m pip install --quiet pyyaml
python scripts/stage_text_map.py

- name: Configure Pages
uses: actions/configure-pages@v5
with:
enablement: true

- name: Verify specialty graph receipts
run: python scripts/check_graph_receipts.py

- name: Upload docs/ artifact
uses: actions/upload-pages-artifact@v3
with:
Expand Down
47 changes: 47 additions & 0 deletions conf/embedding-runtime/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# Optional text-map runtime

This governed Python 3.13 environment is for explicit local embedding and
PaCMAP builds. Normal tests, record validation and Pages rendering do not
install it. The lock pins numerical/model dependencies independently of a
Mech's curation environment; the encoder and map receipts record actual
installed library versions as well.

From a Mech repository root, export current semantic input with its domain
adapter, inspect it, and then explicitly run inference and projection:

```bash
uv run python scripts/text_map_inputs.py --output workspace/text-map-inputs.jsonl
uv run python scripts/embedding_pipeline.py inspect --input workspace/text-map-inputs.jsonl
uv run --locked --project conf/embedding-runtime python scripts/embedding_pipeline.py embed --input workspace/text-map-inputs.jsonl --cache workspace/text-map-vectors.sqlite --profile-output workspace/text-map-profile.json --device cpu
uv run --locked --project conf/embedding-runtime python scripts/embedding_pipeline.py project --input workspace/text-map-inputs.jsonl --cache workspace/text-map-vectors.sqlite --profile workspace/text-map-profile.json --output data/text_map
uv run python scripts/embedding_pipeline.py check --output data/text_map --input workspace/text-map-inputs.jsonl --cache workspace/text-map-vectors.sqlite
```

Run one small explicit `--limit` adapter canary first, with separate output and
cache paths. Verify its vector count, finite coordinates, source receipt,
rendered page and repeat-run cache reuse before a full build. A canary is not a
complete-corpus artifact and cannot satisfy an enabled site's full-input check.

The default display limit is 50,000 records selected deterministically. All
input records must have a verified cache entry; the map reports omitted display
records. No all-pairs similarity matrix is allocated.

Model weights are downloaded only by the explicit `embed` command. A cached
model can be used with `HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1`. There are no
provider API calls. Old unpinned vector caches are not assigned this profile.

The existing site renderer stages a validated bundle at `text-map/` only after
the repository enables it in `conf/text_map.yaml`. Its normal Python environment
runs this verification without importing Torch, NumPy or PaCMAP. The build must
fail if an enabled map is absent, stale or invalid. Preflight checks the exact
BGE model/revision, 1024 dimensions and the fleet's 512-token window. The staging
call passes the approved generation name through `expected_bundle`, rejecting
a different pointer or altered generation before changing site files.
Site deployment remains the
publication boundary. A machine interruption during directory staging may
leave `.text-map-recovery-*` beside the destination; preserve it for recovery.

The runtime has been exercised on macOS Apple Silicon. Platform-specific model
wheels may constrain where heavy builds run; the standard-library verifier is
independent of those wheels. Cross-platform reproducibility is not implied by
the lock or a fixed random seed.
17 changes: 17 additions & 0 deletions conf/embedding-runtime/pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
[project]
name = "mech-embedding-runtime"
version = "1.0.0"
requires-python = ">=3.13,<3.14"
dependencies = [
"sentence-transformers==6.0.0",
"transformers==5.17.0",
"torch==2.14.0",
"pacmap==0.9.1",
"numpy==2.3.5",
"numba==0.63.1",
"scikit-learn==1.8.0",
"PyYAML==6.0.3",
]

[tool.uv]
package = false
Loading
Loading