Gleanex

Elixir client for Glean, covering all four of Glean's public APIs. The typed layer is generated from the OpenAPI descriptions Glean publishes at gleanwork/open-api, so it tracks the real API rather than a hand-picked subset.

Install

def deps do
  [
    {:gleanex, "~> 0.2.0"}
  ]
end

Needs Elixir 1.18 or later.

Use it

config = Gleanex.new(domain: "mycompany", token: System.fetch_env!("GLEAN_API_TOKEN"))

{:ok, response} = Gleanex.search(config, "company holidays")
{:ok, answer} = Gleanex.chat(config, "What are the company holidays this year?")

domain is the backend subdomain Glean gave you, usually your email domain without the TLD. Requests go to https://{domain}-be.glean.com.

With GLEAN_INSTANCE and GLEAN_API_TOKEN exported you can leave the config out and each call falls back to the environment.

Inspecting a config does not print its token, so a crash report or a log line cannot leak it. Read config.token when you need the value itself.

Those two shortcuts cover the common case. Everything else is a generated operation:

{:ok, response} =
  Gleanex.Client.Search.search(
    %{query: "holidays", pageSize: 50},
    config: config
  )

{:ok, agents} = Gleanex.Client.Agents.search_agents(%{}, config: config)
{:ok, _} = Gleanex.Admin.Governance.listpolicies(config: config)

The config travels in the trailing options, alongside per-call overrides like :receive_timeout, :retry and :req_options.

The four APIs

Namespace	What it covers	Token
Gleanex.Client	search, chat, agents, documents, collections, pins, insights	client
Gleanex.Indexing	pushing documents, people, groups and permissions into the index	indexing
Gleanex.Platform	agents, skills and the newer search endpoints	client
Gleanex.Admin	governance policies, reports, findings, datasource administration	client, with governance scopes

Client and Indexing tokens are not interchangeable. Build one config per scope, and a mismatched call fails before it leaves your machine:

indexing = Gleanex.new(domain: "mycompany", token: indexing_token, scope: :indexing)

{:ok, _} = Gleanex.Indexing.Documents.indexdocument(%{document: document}, config: indexing)

Those are the only two families, which is why three of the four rows say client.

A token also carries permission scopes, set when you create it, and those decide which endpoints it may call. Gleanex cannot see them, so a token of the right family with the wrong permissions gets through the check above and is refused by Glean with a 403. The Admin API is where that shows up: a client token that searches perfectly well cannot read governance policies unless it was created with the DATA_GOVERNANCE scope, or change visibility overrides without CONTENT_HIDING.

Results

Every operation returns {:ok, result} or {:error, %Gleanex.Error{}}, never both. Match on reason to tell failures apart:

case Gleanex.search(config, "holidays") do
  {:ok, response} ->
    response.results

  {:error, %Gleanex.Error{reason: :rate_limited, retry_after: seconds}} ->
    back_off(seconds)

  {:error, %Gleanex.Error{reason: :problem_detail, problem: problem}} ->
    Logger.error(problem.detail)

  {:error, error} ->
    raise error
end

retry_after is always a number of seconds, whether Glean sent a delay or a date.

Successful responses are decoded into structs. Field names are Glean's own camelCase, matching their documentation, so request maps and response structs agree with each other:

response.trackingToken
response.hasMoreResults

Retries and timeouts

Transient failures are retried by default, honouring Retry-After on rate limits.

What counts as retryable depends on the API. Client, Platform and Admin calls retry the usual transient failures: HTTP 408, 429, 500, 502, 503 and 504, plus timeouts, refused connections and closed sockets. Indexing retries only the failures that say Glean never processed the request — 408, 429 and 503 — because a 500 or a timeout on a write means the response went missing, not that the write did. Retrying those can index the same batch twice.

The split is by API, not by operation, so a non-idempotent write elsewhere (createannouncement is the clearest) is still retried. Set :retry yourself for those.

Change the policy globally or for one call:

config = Gleanex.new(domain: "mycompany", token: token, retry: %Gleanex.Retry{max_retries: 5})

Gleanex.search(config, "holidays", retry: Gleanex.Retry.disabled())
Gleanex.search(config, "holidays", receive_timeout: 60_000)

Setting retry yourself applies that condition to every API, including Indexing. To keep the per-API behaviour and change only how often or how long, leave retry alone and set the other fields:

%Gleanex.Retry{max_retries: 5, delay: fn _count -> 1_000 end}

Connection pooling

Requests go through Req, which shares one automatically started connection pool across everything in the node. That is fine until two very different kinds of traffic share it: a long bulk index run can hold every connection and leave interactive searches waiting behind it.

Give the slow work its own pool. Start a Finch instance in your supervision tree and name it on the config that does the indexing:

children = [
  {Finch, name: MyApp.IndexingPool, pools: %{default: [size: 10]}}
]

indexing =
  Gleanex.new(
    domain: "mycompany",
    token: indexing_token,
    scope: :indexing,
    req_options: [finch: [name: MyApp.IndexingPool]]
  )

Searches keep using the default pool, and the two can no longer starve each other. The same option sets connection limits, and :connect_options sets the connect timeout, separately from the :receive_timeout above.

Paging

Cursor-paginated endpoints become a Stream:

config
|> Gleanex.Pagination.stream(&Gleanex.Client.Search.search/2, %{query: "holidays"})
|> Stream.flat_map(& &1.results)
|> Enum.take(100)

Streaming

Chat and agent runs can be consumed as they arrive:

{:ok, chunks} = Gleanex.Streaming.chat(config, %{messages: messages})

{:ok, events} = Gleanex.Streaming.agent_run(config, %{agentId: "abc", input: %{}})

for event <- events do
  case Gleanex.SSE.json_data(event) do
    {:ok, payload} -> handle(payload)
    {:error, _} -> :ok
  end
end

The response is delivered to the process that made the request, so consume the stream in that same process, and only once. Consuming it elsewhere raises straight away rather than waiting for chunks that cannot arrive.

Bulk indexing

Bulk uploads are paged, and Glean only swaps in the new batch once it has seen the last page. Gleanex.Bulk drives that protocol:

Gleanex.Bulk.upload(
  indexing_config,
  &Gleanex.Indexing.Documents.bulkindexdocuments/2,
  %{datasource: "mydatasource"},
  :documents,
  documents,
  page_size: 500
)

Only the page count comes back, so a long upload is otherwise silent. :each runs after every page Glean accepts, with that page's response and a map describing it:

Gleanex.Bulk.upload(
  indexing_config,
  &Gleanex.Indexing.Documents.bulkindexdocuments/2,
  %{datasource: "mydatasource"},
  :documents,
  documents,
  page_size: 500,
  each: fn _response, page ->
    Logger.info("uploaded page #{page.page}, #{page.records} documents")
  end
)

Pass your own :upload_id if you want to be able to resume. The generated one reaches :each and nothing else, so an upload that fails on its first page takes its ID with it.

Telemetry

Every request emits a [:gleanex, :request] span with :api, :operation, :method, :url and, on stop, :status.

Working on Gleanex

The typed layer is generated and committed, so users need no Java, no Docker and no generator dependency.

mix glean.specs      # download Glean's descriptions into priv/openapi/
mix glean.gen        # regenerate lib/gleanex/{client,indexing,platform,admin}/
mix test --cover     # the suite, at an enforced 100% threshold

Both tasks live in dev/mix/tasks/ and are not part of the package, so they are available in a checkout of this repository and nowhere else.

priv/openapi/.api-version records the exact upstream commit the committed code came from. Regeneration is deterministic: with unchanged descriptions it should leave the working tree clean.

Descriptions are taken from source_specs/ upstream, not final_specs/. The latter has code samples merged in, which inflates the Client API description from under 400 KB to about 19 MB without adding anything a generator can use.

Do not hand-edit anything under lib/gleanex/client, lib/gleanex/indexing, lib/gleanex/platform or lib/gleanex/admin. Naming and rendering are steered from config/config.exs and the plugin in dev/gleanex/generator/processor.ex.

Git hooks

mix deps.get followed by mix git_hoox.install writes two hooks. They run the checks CI does, split by how long they take, plus two CI does not:

• pre-commit — mix format --check-formatted, mix credo --strict, mix deps.unlock --check-unused when the commit touches mix.exs or mix.lock, and mado check . when it touches Markdown.
• pre-push — mix compile --warnings-as-errors and mix test --cover.

Dialyzer is in neither. Its first run builds a PLT that takes minutes, which is too long to sit in front of a push, so CI runs it on a cached PLT instead.

Skip a hook with git commit --no-verify or git push --no-verify. The tasks live in .git_hoox.exs at the repository root; re-run mix git_hoox.install after changing them.

Integration tests

The suite runs against stubs, which prove the library does what Gleanex expects of it, not that this is what Glean expects. A wrong path prefix or a field name that no longer matches the description would pass every stubbed test.

A separate read-only smoke test covers that, against a real deployment. It is excluded unless asked for:

GLEAN_INSTANCE=mycompany GLEAN_API_TOKEN=... mix test --include integration

It only reads, and only through the Client API. The Indexing API writes to a real search index, and a bulk upload replaces the previous batch, so it is left to the stubbed tests rather than pointed at a live deployment.

Releasing

Releases are driven by release-please, run through release-mate with a short-lived GitHub App token.

Every Conventional Commit landed on main is collected into a release pull request that stays open and updates itself. Merging it bumps @version in mix.exs, rewrites CHANGELOG.md, updates the version in the install snippet above, tags the commit vX.Y.Z and cuts the GitHub release. Nothing to run by hand, and no version to remember to bump.

The install snippet is kept in step by the x-release-please-start-version and x-release-please-end comments around it, with README.md listed under extra-files in release-please-config.json. Any version number between those two comments is rewritten on release, so keep unrelated versions out of that block.

Which commits appear in the changelog follows release-please-config.json: feat, fix, perf and revert are listed, everything else is recorded but hidden. bump-minor-pre-major keeps breaking changes inside 0.x rather than jumping to 1.0.0, and initial-version makes the very first release 0.1.0 rather than release-please's default of 1.0.0.

Cutting the GitHub release triggers .github/workflows/publish.yml, which runs mix hex.publish --yes. It needs a HEX_API_KEY secret.

That makes merging the release pull request the point of no return: a Hex version can never be reused or withdrawn, only deprecated.

Licence

BSD 2-Clause. Gleanex is not affiliated with or endorsed by Glean.