Skip to content

Never let a class-connectivity backend failure read as count 0 - #111

Merged
Robbie1977 merged 1 commit into
mainfrom
fix-silent-zero-class-connectivity
Sep 4, 2026
Merged

Never let a class-connectivity backend failure read as count 0#111
Robbie1977 merged 1 commit into
mainfrom
fix-silent-zero-class-connectivity

Conversation

@Robbie1977

Copy link
Copy Markdown
Contributor

What

Gamma Kenyon cell (FBbt_00100247) showed "There is no data to display" for DownstreamClassConnectivity on 2026-09-04 even though the backend holds 3,886 rows. The count: 0 was not only pinned at the v3-cached nginx edge — a cache-busted request to the origin returned the same zero, so the origin had produced it and the Solr result cache had kept it.

_aggregate_class_connectivity had five return [] paths that are infrastructure failures, not "no partners": the Neo4j membership query raising, or returning False (which dict_cursor silently turns into []); per-instance connectivity unobtainable for every instance; partner classes unresolved although the instances have edges; and the partner membership query failing. The callers turned each into {'count': 0, 'rows': []}, @with_solr_cache stored it (count >= 0 is its definition of valid), and the edge kept the 200 for CACHE_STALE_TIME (one month). Three caches, one lie.

Changes

  • vfb_queries: those paths now raise ConnectivityBackendError. @with_solr_cache does not catch it and ha_api answers 5xx, which neither cache stores. A true zero — a class with no connectivity instances, or none with a positive-weight edge to an in-scope partner class — is unchanged and still cacheable.
  • vfb_queries / solr_result_cache: a result computed with some instances' per-instance connectivity missing (counts are underestimates) is tagged partial. The Solr result cache serves it but refuses to store it, so the next call recomputes rather than inheriting wrong counts.
  • ha_api: /run_query now sets X-Accel-Expires per response. Empty results (count == 0) get X-Accel-Expires: 0 + Cache-Control: no-store, so the edge never stores an empty body; a real zero costs one origin hit per visitor, answered from the Solr cache in milliseconds. Partial results get X-Accel-Expires: 600 (VFBQUERY_PARTIAL_EDGE_TTL). Everything else keeps the edge default. owl_cache's nginx honours X-Accel-Expires as-is (only Cache-Control/Expires/Set-Cookie are in proxy_ignore_headers), so no owl_cache change or redeploy is needed. Emptiness is judged on the full stored result, not the page, so a page past the end of a non-empty result is not treated as "no data".

How to test

src/test/test_class_connectivity_no_silent_zero.py — 15 backend-free tests that monkeypatch every Neo4j/Solr seam of _aggregate_class_connectivity and pin: the two true-zero cases still return []; each failure path raises; partial coverage is reported and flagged; the Solr cache treats a flagged result as not cacheable; and the header logic (0 for empty, 600 for partial, none otherwise, page-past-end not empty). test_ha_api_dispatch, test_ha_api_validation and test_solr_cache_failover pass unchanged (36 tests).

Live: get_downstream_class_connectivity('FBbt_00100247') → 3,886 rows, get_upstream_class_connectivity → 4,231, neither flagged partial, both from the Solr cache in under a second.

After release, a cold request that hits an outage should return 5xx (edge MISS, nothing stored) instead of a cached zero:

curl -s -D - -o /dev/null "https://v3-cached.virtualflybrain.org/run_query?id=<id>&query_type=DownstreamClassConnectivity" | grep -iE "^HTTP|x-cache-status|x-accel|cache-control"

Follow-ups

  • The same silent-empty pattern (except … return []) exists in other query functions; this PR fixes the class-connectivity family only. Worth a sweep.
  • get_term_info's count: -1 two-phase placeholder is a separate edge-poisoning route and is not touched here.

_aggregate_class_connectivity returned [] on five infrastructure
failures (Neo4j membership query raising or returning False, per-instance
connectivity unobtainable for every instance, partner classes unresolved,
partner membership query failing). The callers turned that into
{'count': 0}, @with_solr_cache stored it (count >= 0 is "valid"), and
the v3-cached nginx edge pinned the 200 for CACHE_STALE_TIME. One
transient outage became a month of "There is no data to display" for
gamma Kenyon cell downstream connectivity (2026-09-04).

Raise ConnectivityBackendError on those paths instead. The decorator
does not catch it and ha_api answers 5xx, which neither cache stores.
A real zero (no instances, or no positive-weight edges) is unchanged.

A result computed with some instances' connectivity missing is now
tagged 'partial': the Solr result cache refuses to store it and ha_api
sends X-Accel-Expires: 600 so the edge keeps the underestimate only
briefly. Any empty run_query result is sent with X-Accel-Expires: 0
(plus Cache-Control: no-store) so the edge never stores an empty body;
a true zero is served from the Solr cache in milliseconds, so the cost
is negligible. X-Accel-Expires is honoured by owl_cache's nginx as-is
(only Cache-Control/Expires/Set-Cookie are in proxy_ignore_headers), so
no edge change or redeploy is needed.
@github-actions

github-actions Bot commented Sep 4, 2026

Copy link
Copy Markdown

⚠️ 387 test(s) skipped — VFB backend was unreachable

The full suite ran, but 387 test(s) were skipped because the VFB backend (Neo4j / SOLR / Owlery) did not answer during this run.

These are not failures and not a problem with this branch — but those queries went unverified, so a green check here is an incomplete run.

To re-run: open this workflow run and click Re-run all jobs once the backend is healthy (re-running the “Run completeness” check itself does nothing — it has no job behind it).

431 passed, 387 skipped, 320 warnings in 128.71s (0:02:08)

Posted automatically. This comment is removed once a run completes with zero skips.

@Robbie1977
Robbie1977 merged commit 90d462f into main Sep 4, 2026
6 checks passed
@Robbie1977
Robbie1977 deleted the fix-silent-zero-class-connectivity branch September 4, 2026 16:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant