Bound time-series read memory and concurrency under load

[krowvin]

Summary

Prevent large time-series reads from exhausting memory and starving ordinary requests: retain bounded page data, write JSON directly, enforce response/window caps, keep JDBC deadlines request-owned, and admit eight reads with at most two bulk reads/exports. Overload returns 503 with Retry-After; oversize reads return 413. The experimental SQL paging path remains disabled by default.

Mike (@MikeNeilson) raised the large-row/concurrency issues, CSV streaming versus buffered JSON, timeout/resource pressure, and set_session_user_direct logging overhead during this investigation. He may already have overlapping work, including HydrologicEngineeringCenter/cwms-database#230. I left these changes in this testing pass and can merge his changes into this branch while retaining the tests and comparisons.

Related Issue

Related to #1955 (connection pools for different workloads). This provides load-test evidence and admission limits within the existing pool; separate connection pools remain an investigation.

Companion database PR: HydrologicEngineeringCenter/cwms-database#232. Includes the still-open #1902; account for that dependency before merging. Mike's database work: HydrologicEngineeringCenter/cwms-database#230.

Validation

Exhaustive local performance pass: four million stored observations; mixed sizes/formats; 100–1,000 actual concurrent requests; sustained arrivals, retries, stalled consumers, and recovery. Full build: 773 API unit and 79 Oracle integration tests passed (40/1 skipped); OpenAPI/client checks passed, unchanged GUI artifact reused.

“Before” below is the intermediate export-only candidate, not deployed production. “After” combines bulk admission with the companion database logging fix; these single local runs do not isolate each change's HTTP benefit.

Metric	Before	After
Valid completions: 600 arrivals at 20/s	57	457
HTTP 408 in that arrival phase	7	0
Completed within 120 s: 900 retrying requests	249	722
Logical client deadlines in retry phase	651	178

Resource (same fixture; peaks from retry phase)	Before	After
API logical processors / heap limit	2 / 2 GiB	2 / 2 GiB
Pool maximum / peak active / waiters	30 / 8 / 0	30 / 8 / 0
Peak Java heap	454 MiB	492 MiB
Peak working set / private memory	929 / 983 MiB	915 / 975 MiB
Database	Oracle Free 23.5	Oracle Free 23.5

All 675 ordinary retrying requests completed; the remaining 178 deadlines were bulk requests. All 80 recovery probes passed. Staging with live AWS settings and irregular/shared-pool traffic is still required; no production-capacity claim. No 4-GiB process memory limit was imposed locally.

Detailed results and limitations · Swagger responses

Checklist

• AI tools used

[MikeNeilson]

Right so.... interesting idea and I can't say it's AI slop, it's rather easy to follow. However, it is... a lot.

Compare this to say the idea in #1955, which would end up functionally serving the same purpose while involving far less code.

[krowvin]

Closing this as it was submitted just to show some ideas/brainstorm