Skip to content

Rebuild a missing term_info document instead of serving nothing - #108

Merged
Robbie1977 merged 1 commit into
mainfrom
term-info-solr-fallback
Sep 4, 2026
Merged

Rebuild a missing term_info document instead of serving nothing#108
Robbie1977 merged 1 commit into
mainfrom
term-info-solr-fallback

Conversation

@Robbie1977

Copy link
Copy Markdown
Contributor

Rebuild a missing term_info document instead of serving nothing

term_info documents are written in bulk by VFB_json_schema_indexer, driven by the precompute live query results Jenkins job. A record that reaches the PDB after that job last succeeded has no document at all — and the gap is not hours:

#71  2026-09-04  running
#70  2026-09-02  ABORTED
#66  2026-06-22  SUCCESS

VFB_00107fob ("ME_R on JRC2018Unisex") is the case that surfaced it. Its in_register_with edge, its images and its parent class are all in the PDB, but with no document it was invisible to term info — and it silently dropped out of the medulla class page too.

On a SOLR miss we now run the indexer's own query for the node's type, serve the result through the deserialiser get_term_info already uses, and index it so the id is only ever slow once.

Nothing here re-implements the schema

The queries, the document shape and the SOLR write are all imported from the indexer, which the Dockerfile clones exactly as the Jenkins job does — including cloning VFB_json_schema and copying it into src/vfb, because query_roller lives there. A term_info query that changes upstream is followed on the next image build.

Importing the indexer rather than just the schema matters: ClusterTermInfoQueryIndexer carries its own inline Cypher and never calls QueryLibrary, so a schema-only import would have silently missed Clusters. It also brings generate_solr_doc and the retry-hardened send_solr_docs.

Pin a build with --build-arg INDEXER_REF=<sha> --build-arg JSON_SCHEMA_REF=<sha>; the default tracks master the way the Jenkins job does. Both resolved SHAs are baked into the image, and the schema SHA is what a rebuilt document records in its version field.

The write is conservative

  • Only when the id genuinely has no document. The indexer's atomic-update shape ({"term_info": {"set": …}}) means the sibling precomputed fields (anat_query, anat_2_ep_query, ep_2_anat_query) are untouched regardless.
  • Never when VFBQUERY_CACHE_ENABLED=false, so a live-data test run cannot write into the shared production collection — a fixture doing exactly that poisoned JRC2018U term info for five days.
  • Without the indexer importable (a plain pip install vfbquery has no reason to carry it), every entry point degrades to "no fallback", logs why once, and get_term_info behaves as before.

Three things worth knowing

The miss was not where it looked. term_info_parse_object skips its whole body when there are no hits and returns its initialised skeleton, which then fails schema validation on Name/Id/Meta and is returned raw — so a missing term surfaced as a truthy object with no Id, not as None, and the "No valid term info found" branch was effectively dead. The fallback triggers on results.hits.

src collides. The indexer's top-level package is called src, and so is VFBquery's own source directory — which carries an __init__.py (from vfbquery import *) and is therefore a regular package. The running process always finds its own first, so import src.indexers fails with ModuleNotFoundError even with the indexer on PYTHONPATH. A context manager puts the indexer root ahead of it and restores src afterwards; tested on both the normal and the raising path.

get_version_tag shells out to git. query_roller.get_version_tag() runs git rev-parse --short HEAD against the process working directory on every query build and raises outside a repository, which a container never is. It is pinned to the schema SHA baked into the image.

How to test

vfbquery.get_term_info("VFB_00107fob")
# before: skeleton, no Id, schema validation error
# after:  Id VFB_00107fob, "ME_R on JRC2018Unisex", Images {VFB_00101567}, 10 queries

24 new unit tests, passing with and without the indexer checkout present (no backend needed). The live example suite is still 15/15.

Follow-up, not in this PR

VFB_00107fob also has a volume.obj that 404s while its volume.nrrd and volume_man.obj serve. Separate defect.

term_info documents are written in bulk by VFB_json_schema_indexer, driven
by the `precompute live query results` Jenkins job. A record that reaches
the PDB after that job last succeeded has no document at all, and the gap
is not hours: build #71 started 2026-09-04, #70 was aborted, and the last
success before it was #66 on 2026-06-22.

VFB_00107fob ("ME_R on JRC2018Unisex") is the case that surfaced it. Its
in_register_with edge, its images and its parent class are all in the PDB,
but with no document it was invisible to term info -- and it silently
dropped out of the medulla class page too.

On a SOLR miss, run the indexer's own query for the node's type, serve the
result through the deserialiser get_term_info already uses, and index it so
the id is only ever slow once. Nothing here re-implements the schema: the
queries, the document shape and the SOLR write are imported from the
indexer, which the Dockerfile clones exactly as the Jenkins job does
(including copying VFB_json_schema into src/vfb). A term_info query that
changes upstream is followed on the next image build.

The write is conservative: only when the id genuinely has no document, and
never when the SOLR cache is disabled, so a live-data test run cannot write
into the shared production collection -- a fixture doing that poisoned
JRC2018U term info for five days. The indexer's atomic-update shape means
the sibling precomputed fields on a document are untouched regardless.

Three things worth knowing:

- The miss is now detected on the SOLR result, not on the parse.
  term_info_parse_object skips its whole body when there are no hits and
  returns its initialised skeleton, which fails schema validation on
  Name/Id/Meta and is returned raw -- so a missing term surfaced as a
  truthy object with no Id rather than as None.

- The indexer's top-level package is called `src`, and so is VFBquery's own
  source directory, which carries an __init__.py and is therefore a regular
  package. The running process always finds its own first, so the import
  needs a shim that puts the indexer root ahead of it and restores `src`
  afterwards.

- query_roller.get_version_tag shells out to `git rev-parse` against the
  process working directory and raises outside a repository, which a
  container never is. It is pinned to the schema SHA baked into the image,
  which is also what a rebuilt document now records in its version field.

Verified live: get_term_info("VFB_00107fob") returns a complete term info
(10 queries, images on JRC2018Unisex) where it previously returned a
skeleton, and refuses the write with the cache disabled.
@Robbie1977
Robbie1977 merged commit 1a05822 into main Sep 4, 2026
3 of 5 checks passed
@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown

⚠️ 167 test(s) skipped — VFB backend was unreachable

The full suite ran, but 167 test(s) were skipped because the VFB backend (Neo4j / SOLR / Owlery) did not answer during this run.

These are not failures and not a problem with this branch — but those queries went unverified, so a green check here is an incomplete run.

To re-run: open this workflow run and click Re-run all jobs once the backend is healthy (re-running the “Run completeness” check itself does nothing — it has no job behind it).

2 failed, 625 passed, 167 skipped, 339 warnings in 949.53s (0:15:49)

Posted automatically. This comment is removed once a run completes with zero skips.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant