[8.19](backport #7688) fix: ignore missing .fleet-agents index on enrollment _refresh path

[mergify]

What is the problem this PR solves?

Fleet Server PR #7662 introduced a pre-enrollment _refresh call on .fleet-agents to make prior writes visible before the deduplication msearch in kQueueEnrollSearch. On a fresh Serverless project the index does not yet exist, so _refresh returns HTTP 404. This causes failQueue to be called and every enrollment attempt to fail permanently — a deadlock where the index is never created because enrollment always fails before the first write can land.

Only enrollments that include enrollment_id are affected (they are routed through kQueueEnrollSearch). Horde always sets enrollment_id; Elastic Agent may also set it. Agents that omit enrollment_id bypass this path and can enroll successfully, but they do not unblock the deadlocked agents.

Discovered while investigating why Horde project drones never come online in Serverless staging (elastic/horde#569). ECH drones were unaffected because the Horde k8s smoke test deploys its own older fleet-server binary that predates #7662.

How does this PR solve the problem?

Two fixes:

1. Pass ignore_unavailable=true to the refresh request.
When .fleet-agents does not exist the refresh becomes a no-op (HTTP 200, zero shards) instead of a fatal 404. The first enrollment proceeds to the msearch, the agent document is written, and ES auto-creates .fleet-agents-7 with the correct alias and mappings via its system index descriptor. Subsequent retries with the same enrollment_id then deduplicate correctly.

2. Fix body-close ordering (also introduced in #7662).
refreshResp.Body.Close() was called before refreshResp.String() in the error path, causing every enrollment failure on this path to be reported as:

enroll search refresh failed: <error reading response body: http: read on closed response body>

instead of the actual ES error. Changed to defer refreshResp.Body.Close().

How to test this PR locally

1. Stand up a fresh Serverless Elasticsearch project (no .fleet-agents index).
2. Enroll an agent with enrollment_id set.
3. Confirm enrollment succeeds and .fleet-agents-7 is created with the .fleet-agents alias.
4. Confirm a second enrollment with the same enrollment_id returns ErrEnrollDuplicate.

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc.

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

• Fixes feat: batch enrollment FindAgent searches with pre-refresh dedup #7662 (introduced the _refresh call)
• Relates elastic/horde#569 (investigation that uncovered this)

  ---

  This is an automatic backport of pull request fix: ignore missing .fleet-agents index on enrollment _refresh path #7688 done by Mergify.

[github-actions]

TL;DR

The Buildkite failure is an infrastructure/network fetch error in the benchmark-compare job, not a code regression from this PR. Immediate action is to retry the build; if it recurs, add retry/pinning for the gobenchdata install step.

Remediation

• Re-run Buildkite build 16474 (or retry only go-benchmark-compare) to clear transient proxy.golang.org transport failure.
• Harden .buildkite/scripts/run_benchmark.sh by pinning go.bobheadxi.dev/gobenchdata to a fixed version and wrapping go install with a small retry loop before failing.

Investigation details

Root Cause

The failing step is .buildkite/scripts/run_benchmark.sh compare, and it fails before comparing benchmarks due to a transient Go module proxy download error while installing gobenchdata:

• Script location: .buildkite/scripts/run_benchmark.sh:56 (go install go.bobheadxi.dev/gobenchdata@latest)
• Build log shows transport failure fetching a transitive dependency zip from proxy.golang.org

PR code changes are limited to enrollment refresh/search logic in:

• internal/pkg/bulk/opSearch.go
• internal/pkg/bulk/enroll_search_integration_test.go
  These files are unrelated to the benchmark tool bootstrap path.

Evidence

• Build: https://buildkite.com/elastic/fleet-server/builds/16474
• Job/step: Compare results → .buildkite/scripts/run_benchmark.sh compare
• Key log excerpt:

../../../../go/pkg/mod/github.com/mattn/go-runewidth@v0.0.14/runewidth.go:7:2: github.com/rivo/uniseg@v0.2.0: read "https://proxy.golang.org/github.com/rivo/uniseg/`@v/v0.2.0.zip`": stream error: stream ID 59; INTERNAL_ERROR; received from peer

Verification

• Not run locally; diagnosis is based on the Buildkite job log and repository script path used by the failing step.

Follow-up

• Checked for existing repo issues labeled flaky-test matching this benchmark/proxy failure pattern and found none to reference.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request