Update to pyenvector 1.6 by minseokpark-CL · Pull Request #14 · CryptoLabInc/langchain-envector

minseokpark-CL · 2026-08-27T06:49:13Z

pyenvector 1.4 → 1.6 에 맞춰 integration 을 갱신

(English below.)

왜 PR 하나인가

main 은 0.1.3 에서 멈춰 있고 pyenvector 버전도 고정돼 있지 않았다. 이 브랜치는 SDK 1.4, 1.5, 1.6 이 나올 때마다 그 버전에 맞춰 작업한 것을 다 담고 있어서 diff 가 크다 (37 commits, 19 files). 커밋 단위로 읽는 것이 가장 편하다 — 각 메시지에 무엇을 왜 바꿨는지 적혀 있다. 아래는 전체 그림.

버전은 0.3.0, 의존성은 pyenvector>=1.6.2,<2.0 (1.6.0/1.6.1 은 서버가 V4 암호문을 거부하는 롤백 #2462 대상).

사용자 입장에서 달라지는 것

새 write path (1.6). update_metadata, update_documents, upsert_documents, delete, 그리고 create_partition / drop_partition / list_partitions. 각 write 가 서버를 기다릴지는 WriteSettings 로 정하고, 기본값은 실제 stack 에 대고 잰 것이다 (insert 는 return 시점에 이미 검색되므로 기다리지 않음, update 와 delete 는 기다림). update 는 그 store 가 기다리지 않고 넣은 insert 의 merge 를 먼저 기다린다 — merge 전 row 를 수정하면 검색에서 사라지기 때문.

Item ID.

add_texts / add_documents 는 서버가 발급한 ID 를 list[str] 로 돌려준다. VectorStore 선언과 Document.id 에 맞춘 것. 예전 int 에 산술을 하던 caller 는 int(...) 이 필요하다.
ids= 는 enVector 가 허용하는 데까지 존중한다: 이미 있는 item ID 면 그 row 를 제자리에서 update (upsert 경로). 살아 있는 row 가 없는 id 나 정수가 아닌 id 는 만들 수 없으므로 서버 발급 ID 로 insert 하고, 존중하지 못한 id 를 UserWarning 으로 알린다. ids 가 없으면 base class 처럼 Document.id 를 쓰므로, 검색 결과를 고쳐서 다시 add 하면 중복이 아니라 overwrite 가 된다. 이전에는 ids 를 아무 말 없이 버렸다.

생성자. from_texts / from_documents 가 다른 LangChain vector store 들처럼 embedding model 을 두 번째 positional argument 로 받는다. 예전 embeddings= keyword 도 그대로 동작한다. 덕분에 상속받은 afrom_texts / afrom_documents 도 쓸 수 있게 됐다 — 이 둘은 from_texts 를 positional 로 부르기 때문에 이전엔 TypeError 였다.

embeddings property 가 None 이 아니라 설정된 model 을 돌려준다 (LangSmith 의 ls_embedding_provider, SelfQueryRetriever 등이 읽는 값).

Relevance score. similarity_search_with_relevance_scores 와 retriever 의 search_type="similarity_score_threshold" 가 동작한다. 점수는 inner product 이므로 (1 + score) / 2 로 [0, 1] 에 놓는다 (unit-norm embedding 이면 cosine). 검색 4종은 search_params (예: {"nprobe": 64}) 를 SDK 에 그대로 넘기고, 모르는 keyword 는 TypeError. 검색 결과 metadata 에 _score / _id 를 더 이상 넣지 않는다 — id 는 Document.id, score 는 with_score tuple.

연결과 키. 한 process 안의 여러 store 가 같은 endpoint 를 가리키면 살아 있는 connection 을 공유한다 — 이전엔 두 번째 init_connect 가 첫 store 의 channel 을 닫았다. IndexSettings.index_encryption 을 "cipher" 로 덮어쓰지 않고 그대로 넘긴다. kms_address 가 있으면 key_path 를 생략해 KMS 관리 키가 동작한다.

기본값. preset / eval mode 기본은 ip3 / mms32 (1.5). secure / kms_secure 는 SDK 기본값을 따른다.

README. Limitations 는 사용자가 달리 해야 하는 것만 한 줄씩 (ID, get_by_ids, unit-norm embedding, delete 직후 검색, client-side filtering, process 당 endpoint 하나, write 대기, batch 크기).

어떻게 검증했나

Unit test 는 tests/conftest.py 의 fake 로 돈다 (서버 불필요):

python -m pytest tests -m "not integration"      # 91 passed

Integration test 는 1.6 stack 에 대고 돌렸다 (ENVECTOR_ADDRESS / ENVECTOR_KEY_PATH / ENVECTOR_KEY_ID):

python -m pytest tests/integration_tests -m integration

LangChain 표준 suite (tests/integration_tests/test_vectorstore.py): 7 passed, 16 skipped, 2 xfailed, XPASS 없음. skip 16개는 has_get_by_ids=False (4) 와 has_async=False (12).
test_write_paths.py (11), test_multi_store.py, test_e2e.py: 전부 통과. 전체 21 passed.
CI (PR Checks: pre-commit ruff/black + unit test) 는 ubuntu-latest 로 옮긴 뒤 통과 (9fd9078).

결과를 읽을 때 알아 둘 것 둘. 표준 test 의 fixture 는 fake embedding 을 unit norm 으로 정규화한다 — DeterministicFakeEmbedding 의 원본 vector 는 encrypted inner product 에서 순위가 틀리고, 그 때문에 통과하는 test 하나가 xfail 뒤에 숨어 있었다. 그리고 caller 가 "1", "2" 를 id 로 넘기는 표준 test 들은 새 index 가 item ID 를 1부터 발급하기 때문에 통과한다. 임의의 id 는 여전히 만들 수 없다 (README).

review 요청 전 audit

브랜치 전체 diff 를 여러 관점(정확성, 파일 간 흐름, 제거된 동작, 관례, 효율, 단순화)에서 한 번 더 훑고, 확인된 결함은 ae8a5cc 에서 고쳤다 — 각 항목은 그 커밋 메시지에 있다. 요지: ids 경로로 insert 전용 kwargs 가 새던 것, execute_until="flush" 로 넣은 batch 가 merge drain 에서 영원히 기다리게 되던 것, drain 이 caller 의 timeout_s 에 덮어쓰이던 것, insert 에 600 초 timeout 을 강제하던 것, drain 이 partition 을 가리지 않던 것(실서버 확인: update/delete 는 partition 안에서만 행을 찾는다), 중복·0 ID 가 SDK 검증 에러로 새던 것, KeyConfig() 빈 값이 SDK 깊은 곳에서 죽던 것, pyenvector/langchain 없을 때의 shim 이 hard dependency 에 대해 dead code 였던 것. 설계 trade-off 로 남긴 것(connection cache 가 SDK private 속성 위에 있는 점, foreign id 에 대한 warning 정책 등)은 코드 변경 없이 두었다.

@euphoria0-0 의 review 는 커밋 다섯으로 반영했다: 버전 하한 1.6.2 (bb46d10), search_params 전달 + 미지의 kwarg 차단 (23f686b), metadata 의 _score/_id 제거 (fcac5dc), connection 재사용 가드 + 단위 테스트와 delete 가 drain 하지 않는 서버 측 근거 (bc0cc27), relevance 구현 + MMR 테스트를 #16 으로 (94a652c).

남아 있는 xfail 과 그 뜻

표준 integration test 2개 — test_deleting_documents, test_deleting_bulk_documents. ID 검사와 delete 는 통과하고, 직후 k=1 로 검색하면 빈 결과가 나온다: delete() 가 return 한 뒤 몇 초 동안 지운 row 가 여전히 scoring 에서 이기고 metadata 단계에서 걸러진다. 사용자는 fetch_k 로 넘어갈 수 있고, 고칠 곳은 서버의 delete 완료 신호다 (envector-msa 에 issue 예정).
unit test 에는 xfail 이 없다. MMR 선행 테스트는 MMR (max_marginal_relevance_search) 지원 — 재임베딩 설계 필요 #16 으로 옮겼다.

이 PR 에 없는 것

MMR — 검색이 벡터를 돌려주지 않아 재임베딩 설계가 먼저 필요하다. MMR (max_marginal_relevance_search) 지원 — 재임베딩 설계 필요 #16.
get_by_ids — item ID 로 row 를 찾는 서버 API 가 필요. 지금의 wire request 는 shard/row 위치만 받는다.
진짜 async — async pyenvector 가 필요. 상속된 a* method 는 thread-pool wrapper 로 동작한다.
caller 가 정한 임의의 ID 로 insert, 따라서 LangChain 의 indexing API — 서버가 insert 시 외부 ID 를 받거나 외부 ID → item_id map 을 유지해야 한다.

먼저 볼 곳

libs/envector/langchain_envector/vectorstore.py — add_texts (ids 규칙), _add_or_update, _drain_pending_inserts, 생성자들. config.py — WriteSettings. client.py — 파일 위쪽의 connection 재사용.

Update the integration for pyenvector 1.4 → 1.6

Why this is one PR

main stopped at 0.1.3 with pyenvector unpinned. This branch carries the 1.4, 1.5 and 1.6 updates that were done against each SDK release as it came out, so the diff is large (37 commits, 19 files). It reads best commit by commit — each message says what changed and why — but the sections below give the whole picture.

Version goes to 0.3.0; the dependency becomes pyenvector>=1.6.2,<2.0 (1.6.0 / 1.6.1 are refused by the server since the V4 ciphertext roll-back, envector-msa #2462).

What changes for a user of this package

New write paths (1.6). update_metadata, update_documents, upsert_documents, delete, and create_partition / drop_partition / list_partitions. WriteSettings controls whether each write waits for the server; the defaults were measured against a live stack (inserts are searchable on return and do not wait; updates and deletes wait). Updates first wait for that store's un-awaited inserts to merge, because mutating an unmerged row drops it from search.

Item IDs.

add_texts / add_documents return the server-issued IDs as list[str], matching VectorStore and Document.id. Callers doing arithmetic on the old ints need int(...).
ids= is honoured as far as enVector allows: an id that is an existing item ID updates that row in place (through upsert); an id that names no live row, or a non-integer id, cannot be created, so the row is inserted with a server-issued ID and a UserWarning names the ids that were not honoured. add_documents uses Document.id when ids is absent, like the base class, so re-adding an edited search hit overwrites instead of duplicating. Previously ids was silently dropped.

Constructors. from_texts / from_documents take the embedding model as the second positional argument like every other LangChain vector store; the old embeddings= keyword still works. This also makes the inherited afrom_texts / afrom_documents usable — they call from_texts positionally and raised TypeError before.

embeddings property now returns the configured model instead of None (LangSmith's ls_embedding_provider, SelfQueryRetriever and friends read it).

Relevance scores. similarity_search_with_relevance_scores and the retriever's search_type="similarity_score_threshold" work. Scores are inner products, so (1 + score) / 2 puts them on [0, 1] (a cosine for unit-norm embeddings). All four search methods pass search_params (e.g. {"nprobe": 64}) through to the SDK and raise TypeError on unknown keywords. Search results no longer carry _score / _id in metadata — the id is Document.id, the score the second element of the with_score tuple.

Connection and keys. Several stores in one process share the live connection when they point at the same endpoint — previously the second init_connect closed the first store's channel. IndexSettings.index_encryption is passed through instead of being overridden with "cipher". KMS-managed keys work by omitting key_path when kms_address is set.

Defaults. Preset / eval mode default to ip3 / mms32 (1.5). secure / kms_secure follow the SDK default.

README. Limitations lists what a user has to do differently, one line each (IDs, get_by_ids, unit-norm embeddings, search right after delete, client-side filtering, one endpoint per process, write waiting, batch size).

How it was verified

Unit tests run against the fakes in tests/conftest.py (no server):

python -m pytest tests -m "not integration"      # 91 passed

Integration tests ran against a 1.6 stack (ENVECTOR_ADDRESS / ENVECTOR_KEY_PATH / ENVECTOR_KEY_ID):

python -m pytest tests/integration_tests -m integration

LangChain's standard suite (tests/integration_tests/test_vectorstore.py): 7 passed, 16 skipped, 2 xfailed, no XPASS. The 16 skips are has_get_by_ids=False (4) and has_async=False (12).
test_write_paths.py (11), test_multi_store.py, test_e2e.py: all passed. 21 passed in total.
CI (PR Checks: pre-commit ruff/black + unit tests) passes since the workflow moved to ubuntu-latest (9fd9078).

Two things in the integration tests are worth knowing when reading results. The standard-test fixture normalises its fake embeddings to unit norm — the raw DeterministicFakeEmbedding vectors rank wrongly under encrypted inner product, which had hidden a passing test behind an xfail. And the standard tests that pass caller-chosen ids "1", "2" pass because a fresh index issues item IDs from 1; arbitrary ids are still not creatable (README).

Pre-review audit

The whole branch diff got one more adversarial pass (correctness, cross-file flow, removed behaviour, conventions, efficiency, simplification); the confirmed defects are fixed in ae8a5cc, each listed in that commit message. In short: insert-only kwargs leaking into the upsert arm of add_texts(ids=...), batches inserted with execute_until="flush" waiting forever in the merge drain, the caller's timeout_s overriding the drain budget, a forced 600 s timeout on inserts, the drain not distinguishing partitions (live-verified: updates and deletes only reach rows of the partition they name), duplicate / zero ids surfacing as SDK validation errors, an empty KeyConfig() dying deep in SDK key setup, and import shims for a missing pyenvector / langchain that were dead code against hard dependencies. Design trade-offs (the connection cache sitting on an SDK private attribute, the warning policy for foreign ids) were left as they are.

@euphoria0-0's review landed as five commits: the 1.6.2 floor (bb46d10), search_params passthrough with unknown-kwarg rejection (23f686b), _score/_id out of metadata (fcac5dc), the connection-reuse guard with unit tests plus the server-side reason delete does not drain (bc0cc27), relevance scores implemented and the MMR tests moved to #16 (94a652c).

The xfails that remain, and what they mean

2 standard integration tests — test_deleting_documents, test_deleting_bulk_documents. They get past the ID checks and the delete, then search k=1 immediately and get nothing: for several seconds after delete() returns, the deleted row still wins the scoring and is dropped at the metadata step. fetch_k covers it for users; the fix belongs to the server's delete completion signal (to be filed against envector-msa).
No unit-test xfails remain. The MMR tests written ahead of implementation moved to MMR (max_marginal_relevance_search) 지원 — 재임베딩 설계 필요 #16.

Not in this PR

MMR — search returns no vectors, so a re-embedding design comes first. MMR (max_marginal_relevance_search) 지원 — 재임베딩 설계 필요 #16.
get_by_ids — needs a server API that addresses rows by item ID; the wire request only takes shard/row positions.
Native async — needs an async pyenvector; the inherited a* methods work as thread-pool wrappers.
Arbitrary caller-chosen IDs on insert, and therefore LangChain's indexing API — needs the server to accept an external ID on insert or keep an external-ID → item_id map.

Where to look first

libs/envector/langchain_envector/vectorstore.py — add_texts (the ids rules), _add_or_update, _drain_pending_inserts, and the constructors. config.py — WriteSettings. client.py — the connection reuse at the top of the file.

🤖 Generated with Claude Code

The write paths are asynchronous server-side, and this branch was guessing about what that means for the next read: delete forced `await_completion=False`, overriding the SDK's own default, while the e2e test kept a `time.sleep(0.2)` after inserting. Neither was based on anything. Measured against a live 1.6 stack instead: - insert does not need a wait. Rows are published by `Index.insert`'s own load step; not waiting never lost a row across batches of 2, 100 and 400, while waiting cost a flat ~14s per call regardless of size. The sleep goes, and `await_insert` stays off. - delete does need the SDK's default. Restore it: the wait returned immediately in measurement, so forcing it off bought nothing and risked a stale read. WriteSettings is where those two decisions live, with the measurement recorded next to each, and add_texts grows an `await_completion` override. The SDK's own tuning knobs are deliberately not mirrored there — they reach `Index.insert` through **kwargs, which a test pins. Separately, route search through `_loaded_index()`. The SDK raises "Index not loaded" from search as well as from the write paths, and only the write paths went through the guard, so a first search against a freshly created index failed. None of this is 1.6-specific: both faults are present against 1.5.

pyenvector 1.6 removes `Index.update_metadata` and replaces it with `Index.update` and `Index.upsert`, which take per-item UpdateItem/UpsertItem objects instead of parallel id and metadata lists. That is the only breaking change on the surface this integration uses: a mechanical signature diff against 1.5.0 shows no parameter removed anywhere else, only additions. - `update_metadata` now builds metadata-only UpdateItems, so the vector — and therefore what the item matches — is left alone. - `update_documents` replaces the vector too, which 1.5 could not do; `update_vectors=False` keeps the metadata-only behaviour. - `upsert_documents` is new: id-less entries insert, id-bearing entries replace in place. A caller-chosen id still cannot create an item. - Both paths return the SDK's `{request_id, not_found_item_ids}` shape rather than 1.5's `{updated, skipped}`, and split calls over the SDK's 10k-per-call cap. - UpdateItem/UpsertItem resolve lazily with structural stand-ins so the unit tests keep running without the SDK installed, as `Document` already does. Mutations wait, and they wait for the inserts under them. `update` returns at swap-commit with the rebuilt rows' visibility still pending, so `await_update` is on. Worse, mutating a row whose insert has not merged drops that row from search with no error at all — 4 of 6 mixed upserts and 3 of 6 plain updates lost it, against 0 of 6 when the insert had been awaited. `add_texts` therefore keeps the request ids of inserts it did not wait for, and any update/upsert drains them first. Inserts stay at ~0.13s; the ~15s merge wait is paid once, and only when rows are actually mutated. Search and delete are unaffected — delete over unmerged rows was correct 8 of 8. TODO.md records it as an upstream defect. tests/integration_tests/test_write_paths.py covers the new write paths against a live server: metadata-only update keeps the vector, a vector update moves what the item matches, missing ids are reported rather than raised, and upsert routes a mixed call correctly.

An adversarial review of the two commits above, and of the code they touched, turned up five separate faults. None of them is a 1.6 regression; the upgrade is just what put eyes on this code. Two are production-fatal and pre-date this branch: - A second Envector store killed the first. `init_connect` delegates to the `Index.init_connect` classmethod, which disconnects and replaces the process-global indexer, so every call on the earlier store then failed with "Cannot invoke RPC on closed channel!" — and "one process, several indexes" is ordinary LangChain usage. Record what the live connection was opened with and adopt it when a new store asks for exactly the same endpoint; anything that differs still connects for real, so a channel is never reused for an endpoint the caller did not ask for. test_multi_store.py fails with the reuse disabled. - `from_texts` dropped every add_texts option including `vectors`, so a store without embeddings could not be seeded through it at all. Three are quieter: - The SDK rewrites the `index_params` dict it is handed, upper-casing the type and filling in IVF defaults — mutating the caller's IndexSettings. Pass a copy. It also ignores `index_type` whenever `index_params` is present, and raises a bare KeyError('index_type') if that dict has no type key; fold the two into one value with a clear error when neither supplies one. - Emptying an index makes a search answer NotFound instead of returning []. Normalise that to [] — but only after the server confirms row_count == 0. It is a race, not a fixed state, so matching the error message alone could report "no matches" for a transient failure over live data, which is indistinguishable from data loss. - The LangChain standard-test overrides had `pass` bodies under xfail, so they reported XPASS without asserting anything. They call super() now. Two of the six then passed for real: the placeholder-results xfails were stale, and the backend filtering that makes them pass is present in 1.5 too. The remaining four are genuine — enVector cannot insert at a caller-chosen id. as_retriever and from_texts had no coverage and now have some. Repro scripts and measurements for the three enVector-side issues live outside this repo in ../lc-envector-audit-minseok/.

A second audit pass, attacking the code the previous commits added. - The drain shared `timeout_s` with the per-call waits, but its cost grows with the number of un-awaited insert batches: the server merges them one at a time, so 20 batches took ~125s against a 600s budget. Enough batches and a perfectly ordinary "ingest a lot, then correct one document" would have timed out. Give it its own `drain_timeout_s` (1h). The failure mode was already safe — it raises, keeps the pending ids, and mutates nothing until a retry drains them — which a run with a deliberately short budget confirmed. - `_try_import_item_types` caught every exception, so a broken pyenvector install would have silently fallen back to the stand-in item types instead of failing. Catch ImportError only. - Filtering is client-side and runs after the server has returned `k` hits, so `similarity_search(k=4, filter=...)` quietly returns fewer than 4 — measured 2 of 4, and the full 4 with `fetch_k=10`. That was buried in a config field comment; say it in the limitations. - Pending inserts are tracked per store instance, so a second store instance mutating another's un-awaited rows has nothing to drain. Documented rather than fixed: sharing item ids across store instances is not a shape this integration otherwise supports. Not fixed here, needs a decision: `similarity_search_with_relevance_scores` and therefore `as_retriever(search_type="similarity_score_threshold")` raise a bare NotImplementedError, because `_select_relevance_score_fn` is unimplemented. enVector returns an inner-product similarity while LangChain's built-in helpers all map a distance, so picking the mapping changes what a user's score_threshold means. max_marginal_relevance_search cannot be supported at all — search returns scores and metadata, never the vectors MMR needs.

init() passed index_encryption="cipher" as a literal, with a comment saying the server vectors are always encrypted. That made IndexSettings.index_encryption dead: a caller who set anything else got "cipher" anyway, and nothing told them. Pass the configured value through instead. The default is still "cipher", so a caller who never set it sees no change. tests/test_client_config.py covers both arms against a stubbed SDK — the configured value reaches init_index_config, and the default is unchanged. No pytest fixtures there, because scripts/run_unit_tests.py calls test functions with no arguments, so the stubbing is undone by hand. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

inkme9

1 correctness, 2 design-conformance, 1 silent-gap

Three findings are inline. One is on a file outside the diff:

silent-gap — scripts/run_unit_tests.py:36: the new tests/test_client_config.py is never run by the no-pytest runner it was written for.

That new file documents itself as shaped for that runner — "No pytest fixtures here: scripts/run_unit_tests.py calls test functions with no arguments, so the stubbing is undone by hand instead" — including the manual sys.modules save/restore in _init_with_fake_sdk. But main() still hardcodes modules = ["tests.test_types", "tests.test_vectorstore"], so none of the eight _resolve_index_params / index_encryption tests execute there. They do run under pytest in CI, so nothing is unverified today; the constraint the file accepted simply buys nothing until "tests.test_client_config" is added to that list.

inkme9 · 2026-09-10T06:39:29Z

+        item_cls = update_item if op == "update" else upsert_item
+        index = self._loaded_index()
+        w = self.config.write
+        self._drain_pending_inserts(index, timeout_s, poll_interval_s)


correctness — a caller-supplied per-call timeout_s silently replaces the drain budget.

_mutate forwards its own timeout_s into _drain_pending_inserts, where line 472 resolves it as w.drain_timeout_s if timeout_s is None else timeout_s. So any explicit timeout_s on the mutation also caps the insert drain. That contradicts the docstring right above it (lines 455-458: "It therefore uses config.write.drain_timeout_s rather than the per-call timeout_s") and the stated reason drain_timeout_s exists at all (config.py:82-86: "Separate and much larger than timeout_s because that wait grows with the number of un-awaited insert batches").

Failure scenario: 20 un-awaited add_texts batches — ~125s to drain, per that same comment — then update_documents(ids, docs, timeout_s=30), from a caller tightening the update wait. wait_for_insert_stage is handed 30s, times out, the except block restores the pending ids and re-raises, and nothing is mutated. Retrying with the same argument fails identically — exactly the "timing it out only forces a retry of the same wait" outcome that drain_timeout_s was introduced to avoid.

Passing None through (or taking a distinct drain_timeout_s= keyword) keeps the two budgets independent as documented. The unit test at tests/test_vectorstore.py:741 covers only the default path, so this edge is untested.

고쳤습니다 — ae8a5cc. drain 은 caller 의 timeout_s 와 무관하게 항상 WriteSettings.drain_timeout_s 를 씁니다. 단위 테스트 test_drain_ignores_the_callers_per_call_timeout 추가.

inkme9 · 2026-09-10T06:39:29Z

 - Index encryption is fixed to `cipher`. Query can be `plain` or `cipher`.
- Metadata is stored as a single JSON string per item: `{id, text, metadata}`.
+- Metadata is stored as a single JSON string per item: `{text, metadata}`.
+- Writes (`insert` / `delete` / `update` / `upsert`) are asynchronous server-side; `WriteSettings` makes them wait for completion by default so a write is visible to the next search.


design-conformance — this line says every write path waits by default; insert does not.

WriteSettings.await_insert is False (config.py:78), and the add_texts docstring states the opposite of this sentence: rows are published by the Index.insert load step, and the extra wait is "durability rather than visibility". This reads like a leftover from the earlier revision of the branch that had await_insert=True — the one the PR description says made every insert ~100x slower for nothing. A reader who trusts this line will not reach for await_insert=True when they actually need the merge to have happened.

The same stale claim was added in tests/integration_tests/test_e2e.py:124-125 — "No sleep needed: add_texts waits for the inserted rows to become searchable by default (config.write.await_insert)" — where the accurate reason is that the Index.insert load step publishes the rows, not that add_texts waits.

고쳤습니다 — f64009d. insert 는 기다리지 않고 update/upsert/delete 만 기다린다고 명시했습니다. test_e2e.py 의 같은 주석은 624bb44 에서 수정.

inkme9 · 2026-09-10T06:39:29Z

@@ -5,7 +5,8 @@ High-level VectorStore adaptor for Envector, using the `pyenvector` SDK. Vectors
 Key points
 - Use high-level `pyenvector.EnvectorClient` and `pyenvector.Index`; avoid low-level `pyenvector.api.Indexer`/gRPC.
 - Index encryption is fixed to `cipher`. Query can be `plain` or `cipher`.


design-conformance — "fixed to cipher" stopped being true in b415d55 (stop overriding the caller index_encryption).

EnvectorClient.init now passes index_encryption=i.index_encryption (client.py:160) instead of the hardcoded "cipher", IndexSettings.index_encryption is a plain caller-settable field, and tests/test_client_config.py:129 asserts that "plain" reaches the SDK. The default is still "cipher", so nothing changes for existing callers — but line 3 of this file, "Vectors are always encrypted on the server; the SDK performs required crypto client-side", is now conditional on that setting as well.

These two lines are what a reader checks before trusting the at-rest guarantee, so they should state the default and the escape hatch rather than assert an invariant the code no longer enforces.

고쳤습니다 — f64009d (7행: IndexSettings.index_encryption 기본값 cipher), 624bb44 (3행: "항상 암호화" → 기본값으로 표현).

add_texts and add_documents silently dropped the `ids` argument. That broke LangChain's add-or-update contract in a way nobody could see: search results come back as Documents carrying their item ID, the base class feeds that id back in as `ids`, and re-adding an edited hit created a duplicate row instead of overwriting — with no warning. LangChain's indexing API went the same way: its hash ids were accepted and forgotten, so the first run looked fine and the re-index later failed inside delete(). enVector issues its own item IDs and has no insert-at-ID (the client-facing RPCs are insert_data / batch_insert_data; insert_data_by_id in the proto is shard-internal), so full compliance is not reachable from this package. This does what is reachable and stops being silent about the rest: - ids that are enVector item IDs (int or numeric str) update in place through upsert_documents; None slots insert - an id that names no live row cannot be recreated under that id, so the row is inserted fresh with a UserWarning and the id actually used is returned - non-integer ids (UUIDs, slugs) are inserted fresh with a UserWarning that names them - add_documents uses Document.id when `ids` is not given, as the base class does, so re-adding a search hit now overwrites The returned list always holds the IDs that are really in the index. README and TODO say what is honoured, what is not, and why the indexing API stays unsupported. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

…_documents Every LangChain vector store, and every LangChain example, spells the constructors as `from_texts(texts, embedding, metadatas=...)`. Ours took the model as a keyword-only `embeddings=` and put `metadatas` in the second slot, so the standard call landed the embedding object in metadatas and died with "object has no len()". Worse, the inherited afrom_texts/afrom_documents call from_texts positionally with three arguments and raised TypeError on every call — two methods that existed but could never be used. This has been the shape since the first commit; it was never a 1.6 regression. from_texts and from_documents now take `embedding` second like the base class. `embeddings=` stays as a keyword alias so existing callers keep working; passing both raises ValueError, and the old call shape (a list of dicts in the second slot) raises a TypeError that says to pass metadatas by keyword. afrom_texts/afrom_documents work through the inherited wrappers without further change. Also adds tests/test_vectorstore_api_parity.py: xfail(strict=True) tests, written ahead of implementation, for the VectorStore methods still missing that can be closed inside this package — relevance scores (_select_relevance_score_fn), the `embeddings` property, and MMR. The constructor tests started there and moved to test_vectorstore.py once this change made them pass. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

TODO.md was a running list of what the 1.6 update deliberately left out and of upstream behaviours the integration works around. That is maintainer working state, not something a user of this package needs, and it does not belong in a public repository — it reads as a roadmap nobody committed to and as bug reports filed in the wrong place. The notes are kept outside the repo; the upstream items go to the tracker of the component they concern. The two README bullets that pointed at it now stand on their own. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The 1.6 work had grown the Limitations bullets into paragraphs that explained server internals, quoted timing measurements and listed roadmap items. This README is the public face of the integration; the section's job is to tell a user what to do differently, one line per point, which is how main has it. Same six facts, said that way. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

VectorStore declares `embeddings` as the way to reach the model a store queries with, and the base implementation returns None. Envector kept the model in `_embeddings` and never overrode the property, so a store built with an embedding model still answered None — no error, just a wrong value. LangSmith tracing (ls_embedding_provider) and retrievers that read the property off the store went without it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Several comments and docstrings added during the 1.6 update read as the investigation that produced them — timings, hit counts, what used to be xfailed and why nobody noticed. That belongs in commit messages and issue threads; in the code it buries the one sentence that matters. Each block now states the behaviour and the reason, without the measurements. The four xfail reasons in the standard integration tests also said add_documents(ids=...) was ignored, which stopped being true once item IDs started updating in place; they now name the remaining gap — arbitrary caller-chosen IDs cannot be created. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The repository is black/ruff-clean on main (.pre-commit-config.yaml pins black 24.10.0 and ruff 0.7.1); the files touched this week had drifted. Line wrapping only. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

The fixture used DeterministicFakeEmbedding(size=32) as is. Its components are raw Gaussian samples, up to about ±3, and enVector scores by inner product under encryption, which assumes unit-norm vectors: with the raw vectors a document ranked below another whose true inner product with the query was negative, on a fresh index with no update involved. That made every standard test that checks result order unreliable, and put test_add_documents_by_id_with_mutation in xfail for a reason that had nothing to do with IDs. Normalising the fake vectors makes server scores match local inner products to three decimals. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

VectorStore declares both methods as returning list[str], Document.id is a str, and every LangChain caller — retrievers, the indexing API, the standard tests — hands those values straight back in. We returned the server's ints, so a round trip through LangChain code compared "1" with 1 and failed; the standard delete tests never got past their first assertion for this reason. Every method here already accepted numeric strings, so nothing else moves. The dicts upsert_documents / update_* return are SDK results and keep the SDK's int IDs. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Scores are inner products computed under encryption, which assumes unit-norm vectors. With components outside [-1, 1] the ranking is wrong — reproduced on a live stack with an unnormalised fake embedding, where a document whose true inner product with the query was negative ranked first. Most embedding models emit unit-norm vectors; the ones that do not need to be normalised by the caller, so the README says so. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Three assertions still treated the IDs add_texts returns as ints — an isinstance check and two `max(ids) + 999_999` constructions of an absent ID — and one compared a string ID against the SDK's not_found_item_ids, which stay ints. Verified against a live 1.6 stack. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

One assertion in test_update_documents_replaces_the_vector still compared the int `_id` a scored hit carries against the string ID add_texts now returns. The previous commit claimed the file was verified live; that one test was failing. Whole file green now. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

test_add_documents_by_id_with_mutation and test_add_documents_with_ids_is_idempotent pass now that ids naming existing items update in place and add_texts returns string IDs: a fresh index issues item IDs from 1, which is exactly what those tests pass in. Their xfail overrides are gone. test_deleting_documents and test_deleting_bulk_documents get past the ID checks and the delete, then search k=1 immediately and get nothing back — the row deleted a moment earlier still wins the scoring and is dropped at the metadata step for several seconds after delete() has returned. That is the server's completion signal, not this package, so they stay xfail with that reason instead of the old "caller ids" one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Measured on a live 1.6 stack: for several seconds after delete() returns, the deleted row still wins its top-k slot and is dropped at the metadata step, so k=1 for that row's nearest query comes back empty. fetch_k covers it. The completion signal is the server's to fix; until then users need the one line. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

An adversarial pass over the whole branch diff before asking for review. Each item below was confirmed against the SDK source or a live 1.6 stack. - add_texts forwarded every leftover keyword to the upsert arm when `ids` named item IDs, but Index.upsert takes none of Index.insert's knobs (execute_until, request_ids, use_row_insert, ...): the call raised TypeError after embedding. Insert knobs now apply only to the rows the call inserts; the upsert arm takes timeout_s / poll_interval_s. - Inserts done with execute_until="flush" were queued for the merge drain, but a flush submits no merge, so the next update would have waited the whole drain budget (an hour) and then failed. Only merging inserts queue. - The caller's per-call timeout_s overrode drain_timeout_s for the drain, contrary to its docstring. The drain keeps its own budget. - add_texts pushed WriteSettings.timeout_s (600 s) into Index.insert, whose own default is a day; a large awaited insert would time out client-side after the server had accepted it. The SDK default stands unless a timeout_s is passed. - The drain waited for every partition's pending inserts and, on failure, re-queued the ones already drained. A live probe shows updates and deletes only reach rows of the partition they name (an update without partition_name reports a tenant_a row as not found), so the drain now waits for that partition alone. The unit test that assumed otherwise now names the partition. - Rows inserted through the upsert arm (None slots in add_texts(ids=...)) were never queued for the drain; their insert_request_id now is. - The SDK rejects repeated and non-positive item IDs with its own message. delete de-duplicates; update/upsert raise a ValueError that names the method; "0" (a chunk-index style id) is treated as a foreign id rather than sent to the SDK. - from_documents dropped Document.id where add_documents and the base class use it; both paths now agree. - _index_is_empty read the low-level Indexer; Index.summary() is the high-level call. The import shims for a missing pyenvector / langchain are gone: both are hard dependencies, and the stand-in dataclasses would have hidden a wire-shape change from the unit tests. The mutation cap is imported from the SDK instead of hand-copied. - KeyConfig() with neither key_path nor kms_address reached the SDK and died in key setup with an AttributeError; the client now says which field is missing. A second key path in one process is refused by the SDK with a message about pyenvector.init(); it is re-raised in this package's terms. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

libs/envector/README.md said index encryption is fixed to cipher (it has been configurable since the passthrough fix) and that WriteSettings makes every write wait by default (inserts do not). README.md's delete example carried a leftover measurement remark, its Limitations bullet on IDs now says numeric ids are taken to be item IDs, and the partitions example says updates and deletes address rows within one partition. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

Two of inkme9's four findings were already fixed today (the drain budget in ae8a5cc, the package README's write-default and encryption lines in f64009d). The other two: - scripts/run_unit_tests.py imported a fixed list of two test modules and called each test function directly, so tests/test_client_config.py never ran there — and neither would the newer files. Making the list longer would not have been enough: calling functions directly ignores xfail markers and counts an `async def` test as passed without running it. The script now delegates to pytest with the same selection the CI uses. CONTRIBUTE.md names the script's real path; test_client_config.py drops the docstring that justified its manual stubbing by that runner. - tests/integration_tests/test_e2e.py said add_texts waits for rows to be searchable by default; it does not. The rows are published by Index.insert's own load step, which is what the comment says now. libs/envector/README.md's first line said vectors are always encrypted; that is the default of IndexSettings.index_encryption, not an invariant. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

minseokpark-CL · 2026-09-16T06:18:51Z

9/10 review 의 네 번째 지적(scripts/run_unit_tests.py 가 module 목록을 하드코딩해 test_client_config.py 가 그 runner 로는 돌지 않던 것) — 624bb44 에서 runner 를 pytest 에 위임하는 wrapper 로 바꿨습니다. 목록을 늘리는 것으로는 부족했습니다: 함수를 직접 부르면 xfail marker 를 무시하고 async def 테스트를 실행 없이 통과로 셉니다.

euphoria0-0

1.4 → 1.6 을 한 PR 로 묶은 판단에 동의합니다.
delete / update / upsert / partition 이 전부 SDK 네이티브 호출로 내려가고 흉내 낸 곳이 없다는 점, 비자명한 결정마다 실측 근거가 붙어 있다는 점 좋았습니다.
단위 테스트는 pyenvector 1.6.3rc1 / langchain-core 1.6.3 / pytest 9 에서도 80 passed, 14 xfailed 그대로 재현됩니다.

🔴 2건, 🟡 1건, 🔵 2건 남깁니다.

하나 확인해 드릴 것이 있습니다.
delete 만 _drain_pending_inserts 를 안 부르는 게 의심스러워 서버를 봤는데, DeleteDataCore 가 미머지 행을 Pending 으로 분류해 InsertShardMapList 의 late-binding 으로 처리하고 있었습니다 — 지금 구현이 맞습니다.
근거를 주석 한 줄로 남겨 두시면 다음 사람이 같은 길을 안 돌 것 같습니다.

@euphoria0-0

1.6.0 and 1.6.1 accept seed-only (V4) ciphertexts that the server has since rejected at its API boundary (envector-msa #2462, a security roll-back), so a client on those versions is refused. The floor also contained a pre-release marker, which under PEP 440 makes pip consider pre-releases for this requirement — a user could be handed an rc. `>=1.6.2,<2.0`, and the README says the same. Raised by @euphoria0-0 in review. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@euphoria0-0

The search methods accepted **kwargs and forwarded them to an internal helper that never handed them to Index.search, so `similarity_search("q", search_params={"nprobe": 64})` was accepted and ignored — on an IVF index the probe count stayed at the SDK default with nothing to show for it. `search_params` is now an explicit argument on all four search methods and reaches the SDK; any other unknown keyword raises TypeError naming the accepted ones. Raised by @euphoria0-0 in review. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@euphoria0-0

Scored search results carried the raw score and the server id inside metadata as `_score` / `_id`; only similarity_search stripped them. Now that add_documents honours Document.id, the read-edit-write round trip is real, and re-adding a hit from similarity_search_with_score would have written both keys into the stored payload and returned them as user metadata on the next search. The same value also came back twice, as a str in Document.id and an int in metadata["_id"]. Document.id (a str) is the one place the id lives; the score is the second element of the with_score tuple and nowhere else. Raised by @euphoria0-0. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@euphoria0-0

…euse _reusable_indexer reads pyenvector's private Index._default_indexer. If a future SDK drops it, the old code would have returned None without a word and every second store would have reconnected — closing the first store's channel, which is the behaviour the reuse exists to prevent. hasattr check now raises with a message naming the attribute. client.py had no unit coverage; test_multi_store.py needs a server. Four tests run against a stub pyenvector: same endpoint reuses, different endpoint reconnects, closed channel reconnects, missing private raises. Also records in delete()'s docstring why it does not drain pending inserts like update does: the server's DeleteDataCore classifies an unmerged row as Pending and late-binds the delete, as @euphoria0-0 confirmed in review. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

@euphoria0-0

_select_relevance_score_fn maps a raw score onto [0, 1] as (1 + score) / 2: enVector scores are inner products, which with unit-norm embeddings are cosines in [-1, 1]. The base class's inner-product helper takes a distance and returns 1 - x, which would have inverted the ranking. Values outside the range are clamped. This opens similarity_search_with_relevance_scores, its async form, and the retriever's search_type="similarity_score_threshold". The five relevance tests written ahead of this lose their xfail and join tests/test_vectorstore.py; the corpus fixtures they use move to conftest. The nine MMR tests, which need a re-embedding design that does not exist yet, are recorded in #16 with that design sketch and removed from the tree, so tests/test_vectorstore_api_parity.py is gone. Suggested by @euphoria0-0. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

minseokpark-CL · 2026-09-17T00:47:09Z

delete 가 drain 을 안 하는 근거를 서버 쪽에서 확인해 주신 것 — DeleteDataCore 가 미머지 행을 Pending 으로 분류해 InsertShardMapList late-binding 으로 처리 — delete docstring 에 그대로 적었습니다 (bc0cc27). review 다섯 항목은 각 thread 에 커밋으로 답했고, 최종 상태에서 단위 91 passed, 실서버(1.6.2) 통합 21 passed · 16 skipped · 2 xfailed 입니다.

scripts/run_unit_tests.py is tracked, but .gitignore listed its name, so tools that honour .gitignore skipped it: `black scripts` passed locally while the pre-commit hook in CI — which is handed file names explicitly — rejected the same file (fixed by suyeong in 9fd9078). A tracked file has no business in .gitignore. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

euphoria0-0 and others added 13 commits May 28, 2026 18:05

update pyenvector 1.4.x

8163c63

update pyenvector 1.5.x

79798bc

change default preset/eval_mode to ip3/mms32

4331ebb

update langchain standard test overrides for 1.5.x capabilities

c9998bb

use SDK default for secure/kms_secure options

3044472

remove cipher query e2e leg (unsupported in pyenvector 1.5.x)

e1076b9

fix KMS-managed key wiring by omitting key_path when kms_address is set

54c084d

ensure index is loaded before delete/update_metadata

bf54cfe

inkme9 reviewed Sep 10, 2026

View reviewed changes

minseokpark-CL and others added 16 commits September 16, 2026 09:54

format with black

839993d

The repository is black/ruff-clean on main (.pre-commit-config.yaml pins black 24.10.0 and ruff 0.7.1); the files touched this week had drifted. Line wrapping only. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

euphoria0-0 reviewed Sep 16, 2026

View reviewed changes

Comment thread pyproject.toml Outdated

Comment thread libs/envector/langchain_envector/vectorstore.py

Comment thread libs/envector/langchain_envector/vectorstore.py

Comment thread libs/envector/langchain_envector/client.py

Comment thread tests/test_vectorstore_api_parity.py Outdated

minseokpark-CL and others added 4 commits September 17, 2026 09:40

minseokpark-CL mentioned this pull request Sep 17, 2026

MMR (max_marginal_relevance_search) 지원 — 재임베딩 설계 필요 #16

Open

euphoria0-0 and others added 3 commits September 17, 2026 16:39

run PR checks on ubuntu-latest instead of unavailable self-hosted runner

6f87d6c

format run_unit_tests.py with black

9fd9078

euphoria0-0 approved these changes Sep 17, 2026

View reviewed changes

minseokpark-CL merged commit 363aac1 into main Sep 18, 2026
1 check passed

Navigation Menu

Update to pyenvector 1.6 - #14

Update to pyenvector 1.6#14
minseokpark-CL merged 38 commits into
mainfrom
feat/update-1.6

minseokpark-CL commented Aug 27, 2026 •

edited

Loading

Uh oh!

inkme9 left a comment

Uh oh!

inkme9 Sep 10, 2026

Uh oh!

minseokpark-CL Sep 16, 2026

Uh oh!

inkme9 Sep 10, 2026

Uh oh!

minseokpark-CL Sep 16, 2026

Uh oh!

inkme9 Sep 10, 2026

Uh oh!

minseokpark-CL Sep 16, 2026

Uh oh!

minseokpark-CL commented Sep 16, 2026

Uh oh!

euphoria0-0 left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

minseokpark-CL commented Sep 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

minseokpark-CL commented Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

pyenvector 1.4 → 1.6 에 맞춰 integration 을 갱신

왜 PR 하나인가

사용자 입장에서 달라지는 것

어떻게 검증했나

review 요청 전 audit

남아 있는 xfail 과 그 뜻

이 PR 에 없는 것

먼저 볼 곳

Update the integration for pyenvector 1.4 → 1.6

Why this is one PR

What changes for a user of this package

How it was verified

Pre-review audit

The xfails that remain, and what they mean

Not in this PR

Where to look first

Uh oh!

inkme9 left a comment

Choose a reason for hiding this comment

Uh oh!

inkme9 Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

minseokpark-CL Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

inkme9 Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

minseokpark-CL Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

inkme9 Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

minseokpark-CL Sep 16, 2026

Choose a reason for hiding this comment

Uh oh!

minseokpark-CL commented Sep 16, 2026

Uh oh!

euphoria0-0 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

minseokpark-CL commented Sep 17, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

minseokpark-CL commented Aug 27, 2026 •

edited

Loading