Skip to content

Persist real face embeddings during actor indexing (unblocks #74) - #151

Open
ik020 wants to merge 1 commit into
grayhatdevelopers:mainfrom
ik020:fix/actor-index-persist-real-embeddings
Open

Persist real face embeddings during actor indexing (unblocks #74)#151
ik020 wants to merge 1 commit into
grayhatdevelopers:mainfrom
ik020:fix/actor-index-persist-real-embeddings

Conversation

@ik020

@ik020 ik020 commented Sep 5, 2026

Copy link
Copy Markdown

Prep work for #74 (find people using a reference image).

The bug

Actor indexing detects a face, aligns it, and computes a normalized
SFace encoding per detection — but that encoding was never actually
stored. _actor_records() wrote a placeholder embedding=[0.0] into
every StorageRecord sent to Chroma, so the actor collection's
vector index has never contained real, searchable face embeddings.
Matching during indexing only ever compared against
ActorIndexState.known_encodings, an in-memory list scoped to a
single indexing run, discarded the moment that run finished.

This silently blocks #74: finding a person by reference image
requires comparing a new embedding against every previously detected
face across the whole repository, and there was nothing durable to
compare against.

The fix

  • Carry the computed encoding through into each detection dict in
    process_actor_samples.
  • _actor_records() now writes that real, normalized encoding as the
    StorageRecord.embedding, instead of [0.0].
  • _actor_cluster_records() (the per-cluster summary rollup) is
    intentionally left unchanged — a summary spans multiple detections
    and has no single face image to encode.

Schema version bump — please read

Because this changes what's actually persisted per detection, any
existing generation with the actor modality enabled has
placeholder vectors, not real ones, and is no longer trustworthy for
face search.

This repo already has a mechanism for exactly this situation:
CompletedGenerationManifest.index_schema_version is typed as
Literal[INDEX_SCHEMA_VERSION], and local_snapshots.py's
validate_generation() calls model_validate_json() against the
manifest on disk, explicitly catching ValidationError and
re-raising it as IndexSchemaError. This PR bumps
INDEX_SCHEMA_VERSION from 7 to 8, so old generations now fail
that validation cleanly with IndexSchemaError instead of silently
being treated as complete, searchable indexes with meaningless
vectors.

User-facing impact: any existing generation with actor enabled
will need to be re-indexed after this lands. This is intentional —
the alternative is a search feature (#74) that runs against garbage
data with no indication anything is wrong.

Testing

  • New test: test_actor_indexing_persists_real_face_embeddings_not_ placeholders — drives process_actor_samples through mocked
    YuNet/SFace calls end-to-end and asserts the StorageRecord sent
    to storage.upsert carries the real normalized encoding, not the
    placeholder.
  • Updated test_actor_records_preserve_stable_detection_metadata,
    which built fake detection dicts without an encoding key; added
    an explicit assertion that the resulting embedding matches the
    real value now that it's used, not [0.0].
  • Full suite run clean: 590 passed, 2 skipped, 0 failed.
  • Specifically re-ran everything touching INDEX_SCHEMA_VERSION or
    actor indexing (test_generation_manifest.py,
    test_local_snapshots.py, test_indexing.py,
    test_actor_results.py, test_cli.py, test_frontend_app.py) —
    all green, including the existing test that deliberately constructs
    an invalid INDEX_SCHEMA_VERSION + 1 manifest to prove the
    rejection path works.

Not included here

This PR does not add reference-image search itself — that's #74's
actual feature, tracked as a separate follow-up PR once this lands,
since it depends on real embeddings existing at all.

 prep)

Actor indexing computed a normalized face encoding per detection but
never stored it: _actor_records() wrote a placeholder embedding=[0.0]
into every StorageRecord, so the actor Chroma collection's vector
index has never contained anything searchable. Matching only ever
happened in-memory, within a single indexing run, via
ActorIndexState.known_encodings, and was discarded once that run
finished.

This blocks issue grayhatdevelopers#74 (find people using a reference image), which
requires comparing a reference image's embedding against previously
indexed faces across the whole repository.

Fix:
- Carry the computed encoding through into each detection dict.
- _actor_records() now writes the real normalized encoding as the
  StorageRecord embedding instead of the placeholder.
- Cluster-summary records (_actor_cluster_records) are left
  unchanged: a summary rolls up multiple detections and has no
  single face image to encode.

Because this changes what's actually stored per detection, any
existing generation with the actor modality enabled has placeholder
vectors and is no longer valid. Bump INDEX_SCHEMA_VERSION 7 -> 8 so
CompletedGenerationManifest's Literal[INDEX_SCHEMA_VERSION] check
rejects old generations with a clear IndexSchemaError instead of
silently treating placeholder-vector indexes as complete and
searchable. Affected generations need to be re-indexed.

Adds a new end-to-end test driving process_actor_samples through
mocked detector/recognizer calls, asserting the stored embedding is
the real normalized encoding rather than [0.0]. Updates the existing
_actor_records test, which built detection dicts without the
now-required encoding key.
@SaadBazaz

Copy link
Copy Markdown
Member

Hey @ik020,
Thanks for the Pull Request.
As a starter, we require all contributors to "star" and "fork" the repo, in order to determine if there really is a human behind the wheel or an autonomous agent.
Please star the repo for our review to proceed.
Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants