[williamgrosset]

Why

Run experiments in the EAS CLI.

How

The implementation is a port from our website's experimentation client. Each command can opt in with ContextOptions.Experimentation. The context returns a lazy client and the first getClientAsync() call builds it: fetches the experiments config, resolves the bucketing units, and evaluates variants. The bucketing unit depends on the scope. User: actor ID, Account: account ID the command passes to getAccountNamespace, Device: existing analytics device ID (same UUID used with the RudderStack anonymousId).

Experiment errors are never shown to the user (fail silently) and fall back to defaults. The config fetch is capped at 3 seconds, so a slow network returns defaults instead of delaying the command. Disabling analytics turns off all experiments. CI runs skip experiments. Overrides with EAS_EXPERIMENT_OVERRIDE forces a variant for QA without exposure logging.

Test Plan

See example experiments in stacked PRs: #4631, #4633

To test locally:

1. Build the CLI.
2. Temporarily wire a command (e.g. eas whoami or see examples in stacked PRs) with the experimentation context and variant logs:

static override contextDefinition = {
  ...this.ContextOptions.MaybeLoggedIn,
  ...this.ContextOptions.Experimentation
}

// in runAsync, extend the existing getContextAsync destructuring:
const {
  maybeLoggedIn: { actor },
  experimentation
} = await this.getContextAsync(AccountView, { nonInteractive: true })
const client = await experimentation.getClientAsync()
const account = actor?.accounts.find(a => a.name === 'expo')

if (account) {
  Log.log(`account: ${client.getAccountNamespace('churn-chatbot', { accountId: account.id }).getParam('variant', 'control')}`)
}
Log.log(`device: ${client.getDeviceNamespace('onboarding').getParam('variant', 'control')}`)
Log.log(`user: ${client.getUserNamespace('cli-verify').getParam('variant', 'control')}`)

3. Run the command with EXPO_DEBUG=1. Then verify the following steps.

4. Bucketing unit is correct for each device, account, and user-level experiment and an Experiment Viewed exposure is logged to our staging data store. Bucketing remains consistent across reruns.

5. Supports overrides.

6. Disabling analytics disables experiments.

7. Bot users (access tokens) are excluded from experiments.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 74.92%. Comparing base (9aac1a0) to head (2ad44d0).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4558      +/-   ##
==========================================
+ Coverage   74.77%   74.92%   +0.16%     
==========================================
  Files        1038     1046       +8     
  Lines       51083    51333     +250     
  Branches    11105    11152      +47     
==========================================
+ Hits        38190    38457     +267     
+ Misses      11903    11887      -16     
+ Partials      990      989       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Subscribed to pull request

File Patterns	Mentions
packages/eas-cli/**	@douglowder

Generated by CodeMention

Warning: The preamble and epilogue options in commentConfiguration are deprecated. Use template instead.

[keith-kurak]

This looks good to me, but wanted to have a more frequent-flyer on eas-cli weigh in. @hSATAC would you be able to take a look?

[keith-kurak]

Looks good! My one comment is that I'm wondering if experimentation should be disabled in CI. It's likely that a lot of the experiments wouldn't be applicable, but I guess it could have the potential to cause inconsistent results, or at least inconsistent logs (which sometimes matters in debugging). But, I guess suppose experiments are enabled on the eas build command, if they're not relevant to CI / EAS Build / Workflows, then we'd spend time querying on those job runs where we wouldn't need to. And with CI, those "users" are often bots or our Github context, so they're problably n/a for experiments. Oh, I see you disabled based on bot token, so practically speaking, that covers most scenarios I'm thinking of- it might be good to also disable on CI=1 for thoroughness, tho

My Claude had a comment about how on project-scoped experiments, the app config is evaluated for any command that opts into one, even if the subcommand or option doesn't need it. Most of the EAS CLI evaluates app config anyway, so I can't honestly think of an example where that would. be a problem. But my ears perked up because app config eval executes custom code from. the user, and there have been scenarios where it fails for weird reasons (like node version stuff) and it tends to be confusing, so it had me thinking if we could end up evaluating app config where we didn't previously need to.

The more I think about this, the less likely it sounds, because you can't even determine project ID without a config eval, and project ID's are needed pretty much everywhere. And the places where we get that weird behavior reported tend to be in CI / EAS.

[keith-kurak]

Take this with a grain of sale, I coudln't prove or disprove it :-), but interesting

[hSATAC]

Looks good overall. I left two small questions inline. I mostly contribute to the worker side, so I'd appreciate @douglowder taking a look at how this fits into the broader eas-cli architecture. I'll leave the approval to Doug.

[hSATAC]

Following up on Keith's point: skipPlugins still evaluates dynamic config, so this adds an extra config evaluation inside the CLI process. The catch handles throws, but not side effects like process.exit() or slow synchronous work.

Could we have commands pass the account they already resolved via getAccountNamespace(name, { accountId })? That would avoid evaluating config just to initialize experiments.

[hSATAC]

Are CLI experiments going to use their own namespaces? This returns the same configs the website uses, but device bucketing uses the analytics UUID here rather than IP. Reading a website experiment would also log an exposure for it, so we'd need to filter those out of the website analysis.

[williamgrosset]

Good Q! CLI experiments will use their own namespaces with a cli- prefix. Exposure logs between website/CLI go to different BQ datasets so we don't cross-contaminate the data. We may want to share an account or user-level experiment between the website and the CLI in the future. Added a comment about not sharing device-level namespaces.

[github-actions]

🤖 AI code review

Decision: Ready for human review

Overall PR risk: Medium. The change is additive. It adds an experimentation client and optional parameters to shared functions like SessionManager.getUserAsync, createGraphqlClient, and UserQuery.currentUserAsync. It also adds a getDeviceId method to the AnalyticsWithOrchestration interface. No current command uses the new Experimentation context yet, so the new config-fetch and actor-lookup logic does not run in production. The touched files are shared by every CLI command, so the blast radius would be wide if a defect appears later. Rollback stays simple because the additions are optional and unused for now. No specialist reviewer reported a correctness, security, or consistency issue in this diff.

No findings.

---

This review is advisory — it never blocks a merge and never auto-approves.

[douglowder]

eas-cli now uses changesets for changelogs (#4580). The required "Check for a changeset" check fails on PRs that edit CHANGELOG.md.

To update this PR:

1. Revert the changes to CHANGELOG.md.
2. Run yarn changeset, select the changed packages and the bump type, and write the summary.
3. Commit the new file in .changeset/.

The release adds the PR link and the author to the entry. If this PR needs no changelog entry, revert the CHANGELOG.md changes and add the "no changelog" label.

[github-actions]

✅ Thank you for adding a changeset!