[9.5](backport #7688) fix: ignore missing .fleet-agents index on enrollment _refresh path

[mergify]

What is the problem this PR solves?

Fleet Server PR #7662 introduced a pre-enrollment _refresh call on .fleet-agents to make prior writes visible before the deduplication msearch in kQueueEnrollSearch. On a fresh Serverless project the index does not yet exist, so _refresh returns HTTP 404. This causes failQueue to be called and every enrollment attempt to fail permanently — a deadlock where the index is never created because enrollment always fails before the first write can land.

Only enrollments that include enrollment_id are affected (they are routed through kQueueEnrollSearch). Horde always sets enrollment_id; Elastic Agent may also set it. Agents that omit enrollment_id bypass this path and can enroll successfully, but they do not unblock the deadlocked agents.

Discovered while investigating why Horde project drones never come online in Serverless staging (elastic/horde#569). ECH drones were unaffected because the Horde k8s smoke test deploys its own older fleet-server binary that predates #7662.

How does this PR solve the problem?

Two fixes:

1. Pass ignore_unavailable=true to the refresh request.
When .fleet-agents does not exist the refresh becomes a no-op (HTTP 200, zero shards) instead of a fatal 404. The first enrollment proceeds to the msearch, the agent document is written, and ES auto-creates .fleet-agents-7 with the correct alias and mappings via its system index descriptor. Subsequent retries with the same enrollment_id then deduplicate correctly.

2. Fix body-close ordering (also introduced in #7662).
refreshResp.Body.Close() was called before refreshResp.String() in the error path, causing every enrollment failure on this path to be reported as:

enroll search refresh failed: <error reading response body: http: read on closed response body>

instead of the actual ES error. Changed to defer refreshResp.Body.Close().

How to test this PR locally

1. Stand up a fresh Serverless Elasticsearch project (no .fleet-agents index).
2. Enroll an agent with enrollment_id set.
3. Confirm enrollment succeeds and .fleet-agents-7 is created with the .fleet-agents alias.
4. Confirm a second enrollment with the same enrollment_id returns ErrEnrollDuplicate.

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

• Fixes feat: batch enrollment FindAgent searches with pre-refresh dedup #7662 (introduced the _refresh call)
• Relates elastic/horde#569 (investigation that uncovered this)

  ---

  This is an automatic backport of pull request fix: ignore missing .fleet-agents index on enrollment _refresh path #7688 done by Mergify.

[github-actions]

TL;DR

This build failed in two places: the benchmark step still lacks first-failure context (same limitation as the prior detective note), and the new E2E failure is in long-running TestStandAloneRunningSuite cases that are timing/external-dependency sensitive (TestOpAMPWithUpstreamCollector, TestAgentGracefulForceUnenroll). Immediate action is to rerun once; if it repeats, capture full per-test failure output (not tail-only logs).

Remediation

• Retry the failed Buildkite jobs once to confirm transience (Run go benchmark for PR branch, E2E Test).
• For repeated E2E failures, prioritize stabilizing testing/e2e/stand_alone_test.go by reducing external build/network variance in TestOpAMPWithUpstreamCollector and widening/decoupling timing windows in TestAgentGracefulForceUnenroll.
• For repeated benchmark failures, inspect the uploaded build/next.out artifact and use the first package-level failing line for root-cause attribution (tail log is insufficient).

Investigation details

Root Cause

The failing PR (#7690) changes only:

• internal/pkg/bulk/opSearch.go
• internal/pkg/bulk/enroll_search_integration_test.go

No E2E test code was modified in this PR, and the E2E failures occurred in tests with known high runtime variance:

• testing/e2e/stand_alone_test.go:650 TestOpAMPWithUpstreamCollector performs git clone + make otelcontribcol under a 5-minute test context (stand_alone_test.go:653, :672-693), which is sensitive to CI CPU/network variability.
• testing/e2e/stand_alone_test.go:750 TestAgentGracefulForceUnenroll runs a long orchestration flow with external download/process startup and multiple long Eventually windows (:751-753, :862-884, :902-940), also sensitive to timing jitter.

Related flaky tracking exists for this suite's OpAMP path:

• #6590 [Flaky Test] TestStandAloneRunningSuite/TestOpAMP signal: killed

Evidence

• Build: https://buildkite.com/elastic/fleet-server/builds/16472
• Job/step: E2E Test
  • --- FAIL: TestStandAloneRunningSuite/TestAgentGracefulForceUnenroll (236.83s)
  • --- FAIL: TestStandAloneRunningSuite/TestOpAMPWithUpstreamCollector (300.32s)
  • FAIL github.com/elastic/fleet-server/testing/e2e 1544.970s
• Job/step: Run go benchmark for PR branch
  • Error: running "go test -bench=Bench -tags=grpcnotrace -run=Bench -count=8 -benchmem ./..." failed with exit code 1
  • log only contains tail output and wrapper failure (no first benchmark/package failure line).

Verification

• Not run in this environment: E2E depends on Docker-based integration flows and this runner does not support Docker-in-Docker.

Follow-up

If reruns keep failing, collect full failing sections (or artifact excerpts) for:

1. the first failing benchmark package from build/next.out; and
2. the first assertion/stacktrace inside each failed E2E test body.
   With those, I can provide a deterministic, line-level fix plan instead of flaky-triage guidance.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.