[danielgreen]

Fixes #102

What was wrong

After a merge or mergeOrUpload on a document that already existed, the document kept its values but stopped matching filters, facets and vector queries on the fields the merge did not send. Only its key still matched.

For example, upload a document, then merge only Title:

• Id eq '1' finds it.
• Id eq '1' and Source eq 'blob-1' does not.

The merge read the old document with IndexSearcher.Doc, which returns stored fields only, and wrote those back. Stored fields carry no indexed terms or doc values.

What changed

Two commits, so each can be checked on its own:

1. Move reading a field's stored value into GetStoredValue. The code in LuceneNetIndexSearcher.ConvertSearchDoc that turns each field's stored Lucene fields back into JSON moves, unchanged, into SearchFieldExtensions.GetStoredValue. ConvertSearchDoc still applies retrievable, $select and the hidden sub-fields of a complex value. No change in behaviour: all existing tests pass on this commit alone. The switch's Edm.ComplexType case goes, as it was unreachable.
2. Keep unsupplied fields indexed after a merge. A merge reads the stored document back as JSON with GetStoredValue, puts the item's fields over it, and indexes the result the same way an upload does.
   • An item's field replaces the stored one whole, as in Azure: "Any field you specify in a merge will replace the existing field in the document. This also applies to collections of primitive and complex types."
   • An explicit null still clears a field (merge with an explicit null does not clear the field #71), because null values are not indexed.
   • GetStoredValue does not check retrievable, so a merge keeps a vector that is not retrievable.

Limit: a field marked retrievable: false has no stored copy (except a vector), so a merge that does not send it loses its value. The same happens before this change.

Why the removed code is no longer needed

Most of the lines removed from UpsertIndexDocumentActionBase.cs are comments. The code removed was a set of repairs, each for one symptom of the bug above. Indexing the whole document again makes each one unnecessary:

Removed	What it did	Why it is no longer needed	Test that still covers it, unchanged
ClearedFields, GetClearedFields	#71: recorded which fields a merge set to null	The item, nulls included, is laid over the stored document; a null field gets no Lucene fields	MergeNullClearsFieldTests, MergeNullClearsFieldIntegrationTests
GetLuceneFieldNames	#71: listed every Lucene name a field writes (raw copy, facet values, geo coordinates, vector copy), so a cleared field left none behind	The document is built from nothing, so no old Lucene field can remain	The same tests
RestoreVectorDocValues	#46: rebuilt a vector's packed copy, which the merge lost	The vector is indexed again from its stored copy, as on upload	VectorFieldTests.Merge_OfAnotherField_KeepsTheVector, VectorSearchTests.MergedDocument_IsStillFoundByVectorSearch
The second MergeDocument overload	A shortcut with no cleared fields	One merge path remains	—

RestoreVectorDocValues already noted the remaining gap: "The same reconstruction drops facet, geo and complex-element doc values too, which is a pre-existing fault worth fixing on its own terms rather than quietly here." This change fixes those too; the new tests check facets, geography points and complex fields after a merge.

The lines removed from LuceneNetIndexSearcher.cs are the ones moved into GetStoredValue in the first commit.

Cost: a merge now analyses the document's text and parses its vectors again, where it used to copy stored fields. A merge of a large document is slower; it does not change the result.

Tests

• MergeKeepsFieldsIndexedTests (new, 20 tests): after a merge of one field, filters on each other field type (string, string collection, Int32, Boolean, date, geography point, complex field, complex collection), a text search, a facet, replacement of the merged field, and two merges in a row. 18 of the 20 fail without the change.
• Every existing test passes unchanged; two comments that described the removed code were reworded.
• All 993 unit tests pass, and the integration suite passes against the container.
• Also checked with a probe over every field type and shape (normalized and hidden strings, Int64 above 2^53, Double, dates with offsets and milliseconds, geography points, every collection type, retrievable and hidden vectors, nested complex fields with hidden sub-fields): after a merge, filters, facets, vector queries, scores, highlights and sorts match an untouched copy of the document, and the stored document reads back the same apart from the merged field. Before this change, 33 of its 49 checks fail.

dotnet test from the repo root currently stops before any test runs, on main too: restoring AzureSearchEmulator.Aspire fails with NU1902, a moderate advisory for MessagePack 2.5.302 (GHSA-qhrr-8q5h-9q3h) that the project treats as an error. I ran the unit and integration test projects directly.