Rebuild a missing term_info document instead of serving nothing - #108
Merged
Conversation
term_info documents are written in bulk by VFB_json_schema_indexer, driven by the `precompute live query results` Jenkins job. A record that reaches the PDB after that job last succeeded has no document at all, and the gap is not hours: build #71 started 2026-09-04, #70 was aborted, and the last success before it was #66 on 2026-06-22. VFB_00107fob ("ME_R on JRC2018Unisex") is the case that surfaced it. Its in_register_with edge, its images and its parent class are all in the PDB, but with no document it was invisible to term info -- and it silently dropped out of the medulla class page too. On a SOLR miss, run the indexer's own query for the node's type, serve the result through the deserialiser get_term_info already uses, and index it so the id is only ever slow once. Nothing here re-implements the schema: the queries, the document shape and the SOLR write are imported from the indexer, which the Dockerfile clones exactly as the Jenkins job does (including copying VFB_json_schema into src/vfb). A term_info query that changes upstream is followed on the next image build. The write is conservative: only when the id genuinely has no document, and never when the SOLR cache is disabled, so a live-data test run cannot write into the shared production collection -- a fixture doing that poisoned JRC2018U term info for five days. The indexer's atomic-update shape means the sibling precomputed fields on a document are untouched regardless. Three things worth knowing: - The miss is now detected on the SOLR result, not on the parse. term_info_parse_object skips its whole body when there are no hits and returns its initialised skeleton, which fails schema validation on Name/Id/Meta and is returned raw -- so a missing term surfaced as a truthy object with no Id rather than as None. - The indexer's top-level package is called `src`, and so is VFBquery's own source directory, which carries an __init__.py and is therefore a regular package. The running process always finds its own first, so the import needs a shim that puts the indexer root ahead of it and restores `src` afterwards. - query_roller.get_version_tag shells out to `git rev-parse` against the process working directory and raises outside a repository, which a container never is. It is pinned to the schema SHA baked into the image, which is also what a rebuilt document now records in its version field. Verified live: get_term_info("VFB_00107fob") returns a complete term info (10 queries, images on JRC2018Unisex) where it previously returned a skeleton, and refuses the write with the cache disabled.
|
This was referenced Sep 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rebuild a missing term_info document instead of serving nothing
term_infodocuments are written in bulk by VFB_json_schema_indexer, driven by theprecompute live query resultsJenkins job. A record that reaches the PDB after that job last succeeded has no document at all — and the gap is not hours:VFB_00107fob("ME_R on JRC2018Unisex") is the case that surfaced it. Itsin_register_withedge, its images and its parent class are all in the PDB, but with no document it was invisible to term info — and it silently dropped out of the medulla class page too.On a SOLR miss we now run the indexer's own query for the node's type, serve the result through the deserialiser
get_term_infoalready uses, and index it so the id is only ever slow once.Nothing here re-implements the schema
The queries, the document shape and the SOLR write are all imported from the indexer, which the Dockerfile clones exactly as the Jenkins job does — including cloning VFB_json_schema and copying it into
src/vfb, becausequery_rollerlives there. A term_info query that changes upstream is followed on the next image build.Importing the indexer rather than just the schema matters:
ClusterTermInfoQueryIndexercarries its own inline Cypher and never callsQueryLibrary, so a schema-only import would have silently missed Clusters. It also bringsgenerate_solr_docand the retry-hardenedsend_solr_docs.Pin a build with
--build-arg INDEXER_REF=<sha> --build-arg JSON_SCHEMA_REF=<sha>; the default tracksmasterthe way the Jenkins job does. Both resolved SHAs are baked into the image, and the schema SHA is what a rebuilt document records in itsversionfield.The write is conservative
{"term_info": {"set": …}}) means the sibling precomputed fields (anat_query,anat_2_ep_query,ep_2_anat_query) are untouched regardless.VFBQUERY_CACHE_ENABLED=false, so a live-data test run cannot write into the shared production collection — a fixture doing exactly that poisoned JRC2018U term info for five days.pip install vfbqueryhas no reason to carry it), every entry point degrades to "no fallback", logs why once, andget_term_infobehaves as before.Three things worth knowing
The miss was not where it looked.
term_info_parse_objectskips its whole body when there are no hits and returns its initialised skeleton, which then fails schema validation on Name/Id/Meta and is returned raw — so a missing term surfaced as a truthy object with noId, not asNone, and the "No valid term info found" branch was effectively dead. The fallback triggers onresults.hits.srccollides. The indexer's top-level package is calledsrc, and so is VFBquery's own source directory — which carries an__init__.py(from vfbquery import *) and is therefore a regular package. The running process always finds its own first, soimport src.indexersfails with ModuleNotFoundError even with the indexer on PYTHONPATH. A context manager puts the indexer root ahead of it and restoressrcafterwards; tested on both the normal and the raising path.get_version_tagshells out to git.query_roller.get_version_tag()runsgit rev-parse --short HEADagainst the process working directory on every query build and raises outside a repository, which a container never is. It is pinned to the schema SHA baked into the image.How to test
24 new unit tests, passing with and without the indexer checkout present (no backend needed). The live example suite is still 15/15.
Follow-up, not in this PR
VFB_00107fobalso has avolume.objthat 404s while itsvolume.nrrdandvolume_man.objserve. Separate defect.