Persist real face embeddings during actor indexing (unblocks #74) - #151
Open
ik020 wants to merge 1 commit into
Open
Persist real face embeddings during actor indexing (unblocks #74)#151ik020 wants to merge 1 commit into
ik020 wants to merge 1 commit into
Conversation
prep) Actor indexing computed a normalized face encoding per detection but never stored it: _actor_records() wrote a placeholder embedding=[0.0] into every StorageRecord, so the actor Chroma collection's vector index has never contained anything searchable. Matching only ever happened in-memory, within a single indexing run, via ActorIndexState.known_encodings, and was discarded once that run finished. This blocks issue grayhatdevelopers#74 (find people using a reference image), which requires comparing a reference image's embedding against previously indexed faces across the whole repository. Fix: - Carry the computed encoding through into each detection dict. - _actor_records() now writes the real normalized encoding as the StorageRecord embedding instead of the placeholder. - Cluster-summary records (_actor_cluster_records) are left unchanged: a summary rolls up multiple detections and has no single face image to encode. Because this changes what's actually stored per detection, any existing generation with the actor modality enabled has placeholder vectors and is no longer valid. Bump INDEX_SCHEMA_VERSION 7 -> 8 so CompletedGenerationManifest's Literal[INDEX_SCHEMA_VERSION] check rejects old generations with a clear IndexSchemaError instead of silently treating placeholder-vector indexes as complete and searchable. Affected generations need to be re-indexed. Adds a new end-to-end test driving process_actor_samples through mocked detector/recognizer calls, asserting the stored embedding is the real normalized encoding rather than [0.0]. Updates the existing _actor_records test, which built detection dicts without the now-required encoding key.
Member
|
Hey @ik020, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Prep work for #74 (find people using a reference image).
The bug
Actor indexing detects a face, aligns it, and computes a normalized
SFace encoding per detection — but that encoding was never actually
stored.
_actor_records()wrote a placeholderembedding=[0.0]intoevery
StorageRecordsent to Chroma, so theactorcollection'svector index has never contained real, searchable face embeddings.
Matching during indexing only ever compared against
ActorIndexState.known_encodings, an in-memory list scoped to asingle indexing run, discarded the moment that run finished.
This silently blocks #74: finding a person by reference image
requires comparing a new embedding against every previously detected
face across the whole repository, and there was nothing durable to
compare against.
The fix
encodingthrough into each detection dict inprocess_actor_samples._actor_records()now writes that real, normalized encoding as theStorageRecord.embedding, instead of[0.0]._actor_cluster_records()(the per-cluster summary rollup) isintentionally left unchanged — a summary spans multiple detections
and has no single face image to encode.
Schema version bump — please read
Because this changes what's actually persisted per detection, any
existing generation with the
actormodality enabled hasplaceholder vectors, not real ones, and is no longer trustworthy for
face search.
This repo already has a mechanism for exactly this situation:
CompletedGenerationManifest.index_schema_versionis typed asLiteral[INDEX_SCHEMA_VERSION], andlocal_snapshots.py'svalidate_generation()callsmodel_validate_json()against themanifest on disk, explicitly catching
ValidationErrorandre-raising it as
IndexSchemaError. This PR bumpsINDEX_SCHEMA_VERSIONfrom7to8, so old generations now failthat validation cleanly with
IndexSchemaErrorinstead of silentlybeing treated as complete, searchable indexes with meaningless
vectors.
User-facing impact: any existing generation with
actorenabledwill need to be re-indexed after this lands. This is intentional —
the alternative is a search feature (#74) that runs against garbage
data with no indication anything is wrong.
Testing
test_actor_indexing_persists_real_face_embeddings_not_ placeholders— drivesprocess_actor_samplesthrough mockedYuNet/SFace calls end-to-end and asserts the
StorageRecordsentto
storage.upsertcarries the real normalized encoding, not theplaceholder.
test_actor_records_preserve_stable_detection_metadata,which built fake detection dicts without an
encodingkey; addedan explicit assertion that the resulting embedding matches the
real value now that it's used, not
[0.0].INDEX_SCHEMA_VERSIONoractor indexing (
test_generation_manifest.py,test_local_snapshots.py,test_indexing.py,test_actor_results.py,test_cli.py,test_frontend_app.py) —all green, including the existing test that deliberately constructs
an invalid
INDEX_SCHEMA_VERSION + 1manifest to prove therejection path works.
Not included here
This PR does not add reference-image search itself — that's #74's
actual feature, tracked as a separate follow-up PR once this lands,
since it depends on real embeddings existing at all.