[9.4](backport #7662) feat: batch enrollment FindAgent searches with pre-refresh dedup

[mergify]

What is the problem this PR solves?

At 30k+ agent scale, enrollment retries can create duplicate (ghost) agent documents in .fleet-agents. The root cause is ES near-real-time search latency: when an agent retries enrollment before the previous write is visible to search (default refresh interval: 1s on ES, 5s on Serverless), a concurrent retry also finds "no existing agent" and writes a second document with the same enrollment_id.

How does this PR solve the problem?

Adds a new kQueueEnrollSearch bulker queue that batches enrollment FindAgent searches and, before executing them, fires a single POST .fleet-agents/_refresh to make all recent writes visible. Within each batch:

1. Pre-refresh: all unique refreshIndex values in the batch are refreshed so retries see the most recent writes.
2. De-duplication: requests sharing the same enrollment_id (passed via WithDedupeKey) are grouped by FIFO order — the oldest (first) request is canonical and is included in the msearch; duplicates receive HTTP 429 (ErrEnrollDuplicate) and retry after backoff.
3. Error propagation: if the canonical search itself returns an ES error, that error is forwarded to duplicates instead of ErrEnrollDuplicate, so operational failures are not silently hidden.

The queue flushes when N requests accumulate (default: 50) or after M seconds (default: 1s), whichever comes first — capping refresh calls at one per fleet-server instance per flush window. Both thresholds are configurable via inputs[].server.bulk.enroll.flush_interval and inputs[].server.bulk.enroll.flush_threshold_cnt.

How to test this PR locally

Run the integration test added in this PR:

ELASTICSEARCH_HOSTS=localhost:9200 ELASTICSEARCH_SERVICE_TOKEN=<token> \
  go test -tags integration ./internal/pkg/bulk/... -run TestEnrollSearchDedup -v

For manual testing, set a low flush_threshold_cnt (e.g. 2) in config and fire concurrent enrollment requests with the same enrollment_id. Only one agent document should be created.

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer. (Each instance maintains its own bulker queue; the pre-refresh + op_type: create write guarantees in ES ensure correctness across instances.)
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding. (Flush interval and threshold count cap the refresh rate; duplicates shed load via 429.)

Checklist

• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

• Relates elastic/horde#532

  ---

  This is an automatic backport of pull request feat: batch enrollment FindAgent searches with pre-refresh dedup #7662 done by Mergify.

[mergify]

Cherry-pick of 3890bbb has failed:

On branch mergify/bp/9.4/pr-7662
Your branch is up to date with 'origin/9.4'.

You are currently cherry-picking commit 3890bbb.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   changelog/fragments/1787137647-enroll-search-bulker-queue.yaml
	modified:   internal/pkg/api/error.go
	modified:   internal/pkg/api/handleEnroll.go
	new file:   internal/pkg/bulk/enroll_search_integration_test.go
	modified:   internal/pkg/bulk/opSearch.go
	modified:   internal/pkg/bulk/queue.go
	modified:   internal/pkg/config/input.go
	modified:   internal/pkg/dl/agent.go
	modified:   internal/pkg/dl/common.go

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   internal/pkg/bulk/block.go
	both modified:   internal/pkg/bulk/engine.go
	both modified:   internal/pkg/bulk/opt.go
	both modified:   internal/pkg/dl/search.go

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[github-actions]

TL;DR

Buildkite is failing because this backport branch contains unresolved cherry-pick conflict markers (<<<<<<<, =======, >>>>>>>) in Go source files. Remove the markers and complete conflict resolution in the affected files; all 3 failed jobs are downstream of that same compile error.

Remediation

• Resolve merge conflicts in internal/pkg/bulk/block.go, internal/pkg/bulk/engine.go, internal/pkg/bulk/opt.go, and internal/pkg/dl/search.go so no conflict-marker text remains and the intended 9.4-compatible code is preserved.
• Re-run .buildkite/scripts/check_ci.sh and .buildkite/scripts/release_test.sh (x86_64 + x86_64 FIPS) after conflict resolution.

Investigation details

Root Cause

This is a code bug introduced by an incomplete backport/cherry-pick conflict resolution. The compiler is parsing git conflict markers as Go tokens.

Evidence from PR metadata and source:

• Mergify conflict note on this PR reports unmerged paths in:
  • internal/pkg/bulk/block.go
  • internal/pkg/bulk/engine.go
  • internal/pkg/bulk/opt.go
  • internal/pkg/dl/search.go
• PR head file contents still include markers such as:
  • internal/pkg/bulk/block.go (<<<<<<< HEAD / >>>>>>> 3890bbb ...)
  • internal/pkg/bulk/opt.go (<<<<<<< HEAD / >>>>>>> 3890bbb ...)
  • internal/pkg/dl/search.go (<<<<<<< HEAD / >>>>>>> 3890bbb ...)

Evidence

• Build: https://buildkite.com/elastic/fleet-server/builds/16388
• Jobs/steps:
  • Run check-ci
  • Package x86_64
  • Package x86_64 FIPS
• Key log excerpts:

internal/pkg/bulk/block.go:19:1: expected '}', found '<<'
internal/pkg/bulk/engine.go:624:1: expected statement, found '<<'
internal/pkg/bulk/opt.go:28:1: expected '}', found '<<'
internal/pkg/dl/search.go:29:1: expected declaration, found '<<'

and in packaging jobs:

internal/pkg/bulk/block.go:19:1: syntax error: unexpected <<, expected field name or embedded type
internal/pkg/bulk/engine.go:624:1: syntax error: unexpected <<, expected }
internal/pkg/bulk/opt.go:28:1: syntax error: unexpected <<, expected field name or embedded type

Verification

• Not run locally in this workflow; diagnosis is directly supported by compiler errors plus visible unresolved conflict markers in PR-head source.

Follow-up

• After resolving conflicts, if CI still fails, the next likely check is API compatibility around spanLink / dedupe option plumbing in bulk and dl/search paths.

---

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[mergify]

This pull request has not been merged yet. Could you please review and merge it @ycombinator? 🙏