Reduce GitHub Actions usage (DataFusion is currently the top ASF consumer of GHA minutes)

Is your feature request related to a problem or challenge?

Recently we have been seeing the Merge Queue get stuck:

CI takes hours to start #25088

We enquired with ASF Infrastructure on INFRA-28374 and they told us the issue is that the ASF's GitHub Action queue is full which is why it is taking a long time to start jobs:

We are aware of a long term usage pattern whereby our limited number of
concurrent GitHub Actions jobs running on GitHub hosted runners is at 100%
utilisation most of the day most of the time. Thus, from time to time we
communicate to projects who utilise the most minutes to review their
workflows for efficiency improvements.

At the moment, datafusion is the top consumer of GitHub Action minutes over
the last 7 days, and 3x the 10th in the list of minutes used.

Note the "datafusion" project as measured by ASF Infra includes all DataFusion subprojects (datafusion, datafusion-comet, datafusion-ballista, datafusion-python, etc.), so reducing usage may require a cross-repository effort.

Here is the report of our current usage (login required):

History

This is the second time we have been contacted by ASF Infra about GHA usage.

The first was in May 2026, when ASF Infra identified DataFusion as a top-5 consumer: #22455. After some tweaks, by June 2026, our usage dropped below the policy thresholds and DataFusion fell out of the top 5.

Useful resources

ASF GitHub Actions policy (limits: 250,000 minutes / calendar week, 216,000 minutes / any 5-day window): https://infra.apache.org/github-actions-policy.html
ASF usage report for the datafusion project (login required): https://infra-reports.apache.org/#ghactions&project=datafusion&hours=168&limit=15&group=name
ASF Infra best-practices wiki page: https://cwiki.apache.org/confluence/spaces/INFRA/pages/430408443/GitHub+Actions+Recommended+Practices
ASF Slack channel for sharing workflow optimisations across projects: #project-workflow-optimisations

Related issues

Reduce Github Action Usage #22455 (previous round of this same problem)
[EPIC] A collection of items to improve developer / CI speed #13813
Add Local Scripts to Reproduce Full CI #21048
CI takes hours to start #25088 (CI takes hours to start — likely a symptom of pool saturation)
Move Extended tests to the standard ones? #21241 (discussion of when extended tests should run)
CI: count-distinct byte tests run for hours and hit the 360-minute limit #25011 (example of a runaway CI job: extended tests hitting the 360 minute limit)

CI time breakdown by workflow

Breaking down the report screenshot above, DataFusion is as

Total usage: 336,347 minutes, or 33 full-time runners — over the ASF policy ceiling of 250,000 minutes / 25 full-time runners per calendar week.

Three workflows exceed 5% of total time, and together account for ~87% of all usage:

1. CI — 49.31% (~165,900 min)

Repo: apache/datafusion-comet
Source: https://github.com/apache/datafusion-comet/blob/main/.github/workflows/ci.yml
Comet's top-level CI orchestrator

2. Rust — 27.16% (~91,350 min)

Repo: apache/datafusion
Source: https://github.com/apache/datafusion/blob/main/.github/workflows/rust.yml
The main DataFusion core test suite that runs on every PR push, again in the merge queue, and again on push to main.

3. Datafusion extended tests — 10.85% (~36,500 min)

Repo: apache/datafusion
Source: https://github.com/apache/datafusion/blob/main/.github/workflows/extended.yml
The long-running correctness suite (extended tests, hash-collision tests, sqlite sqllogictests) that runs on every push to main/branch-* and on every push to any PR touching datafusion/physical*, expr*, optimizer, or sql — a large fraction of all PRs.

Everything else (< 2.5% each)

Workflow	Share
Dev	2.38%
Dependencies	2.01%
Ballista Rust	1.73%
Delta Contrib Build Gate	1.47%
Detect breaking changes	0.96%
TPC-H SF10	0.88%
CodeQL	0.76%
TPC-DS SF1	0.47%
PyArrow UDF Tests	0.42%
Docker	0.29%
h2o	0.24%
Security audit	0.22%
(other builds)	0.86%

Observations

Unlike the May 2026 (#22455) where Comet was ~78% of usage, the core datafusion repo now accounts for ~40%+ of usage itself
The "Datafusion extended tests" slice (10.85%) is exactly the one the merge-queue gating idea in Move Extended tests to the standard ones? #21241 would help

@blaginin

FYI @blaginin

Among other things the fact we have a merge queue and the jobs timeout likely exacerbates the problem (as each PR has to get put in the queue a few times rerunning CI each time)

I also filed a ticket in Comet

Reduce CI time (Comet is consuming 50% of DataFusion CI, which is a lot) datafusion-comet#5830

@alamb

Thanks @alamb for tracking this. Comet got lots of contributions last month and that explains

This ticket may also help DF CI #25092, very frequently the PR removed from MQ because of flaky MinIO tests

Is it possible for Apple to chime in for datafusion and datafusion-comet a little?
Are we getting GitHub Actions minutes from GitHub or from Apache? because if this is from Apache, we might be better change to use Hetzner servers (which is what we are doing in my company)

Is it possible for Apple to chime in for datafusion and datafusion-comet a little? Are we getting GitHub Actions minutes from GitHub or from Apache? because if this is from Apache, we might be better change to use Hetzner servers (which is what we are doing in my company)

Organizational part is bumpy atm, we already tried that. Minutes come from GH actions that has been run using ASF shared resources.

Is it possible for Apple to chime in for datafusion and datafusion-comet a little? Are we getting GitHub Actions minutes from GitHub or from Apache? because if this is from Apache, we might be better change to use Hetzner servers (which is what we are doing in my company)

I believe the runners are provided by Github as an in-kind donation to the ASF

We could try and crowdsource additional runner minutes (the way arrow did with ursa computing, etc) but then someone has to manage / pay / coordinate for those resources

I think we shoudl try and reduce runner usage first, and then if we are still having problems figure out a way to get more resources

Could we have someone (e.g. Apple, probably needs to be someone w/ a big AWS account) run runs-on runners in their AWS account and we coordinate GitHub App installation / auth w/ ASF Infra? Then the AWS account (and it's bill) would be long to whoever is funding it and there would be no need to coordinate payment (which I understand can be the hard part).

Another option which works for Apache Spark and tested in Comet as well is running CI on the contributor forks apache/datafusion-comet#4570

It changes the model and also contributor machines are 20-30% slower than ASF but the benefit is ASF resource pool is being used much more considerate. This might be later option, but taking into account DF ecosystem is growing and take more resources we might think of it

I think some of the benefit of runs-on is limited by the fact that a lot of contributors open PRs from forks, so the vars context is empty (or coming from the fork repo) and we still use the slow gh-provided runners.

so when you open pr from the fork, the base repo still runs them on the head repo infra. it can be runs-on in DF's case - but i had to disable it in a bunch of places because of costs...

Could we have someone (e.g. Apple, probably needs to be someone w/ a big AWS account) run runs-on runners in their AWS account and we coordinate GitHub App installation / auth w/ ASF Infra?

that seems super reasonable

Although the recent numbers for the last 7 days are withing the ASF policy boundary (limits: 250,000 minutes / calendar week, 216,000 minutes / any 5-day window). I feel if datafusion continue to see new contributors and the PR volumes keep increasing, we will pass the limit again, that's why I think the fork pr job runs are the way to go forward so we don't have to deal with this again.

I have opened a pr for that: #25591

ci: cache dependencies for feature checks (apache#25231)

ci: share workspace check artifacts with feature checks (apache#25249)

ci: skip queue-verified Rust jobs on pushes to main (apache#25365)

ci: skip queue-verified Dev and Dependencies jobs on pushes to main (a…

chore(ci): Register the substrait check job with runs-on, route spark…

chore: Move more CI jobs to run on `ubuntu-slim` (apache#26571)

chore: run the circular dependency check on the fork for fork PRs (ap…

chore: Skip `datafusion-sqllogictest` in semver check (apache#26577)

chore: check for circular dependencies with `cargo publish` (apache#2…

chore: skip slow test under `force_hash_collisions` (apache#26594)

Rollout plan for merge queue cleanup (#26760)

#26760 adds the Merge queue cleanup workflow. The workflow finds unfinished merge queue runs whose queue branch was removed or now points to a different commit. After the merge, the workflow only reports candidates. It does not cancel runs until a maintainer enables it.

1. Observe for at least one week

Merge ci: cancel obsolete merge queue runs #26760. Do not set MERGE_QUEUE_CLEANUP_ENABLED.
Let the workflow run for at least one week. It runs for each new merge group and every 15 minutes.
Read the "Would cancel merge-group runs" table in each run summary. Each row gives the run ID, SHA, attempt, reason and UTC observation time.
Compare each candidate with the real outcome of its queue entry and workflow run.

2. Check the results before enabling cancellation

Make sure that there are no false candidates. Each candidate must belong to a group that left the queue or that a new group replaced. It must not belong to an active group or to a merged commit.
Make sure that reruns (attempt 2 or higher) never appear as candidates.
Measure the delay between the creation of a replacement group and the scan that finds the old group.
Count how often the merge_group scan finds candidates. If it almost never finds any, remove the merge_group trigger and keep the 15-minute schedule.
Look for failed scans. For example, a scan fails if a run listing reaches the 1,000-result cap.

Dry runs measure detection only. They do not measure cancellation latency or runner shutdown cost.

3. Enable cancellation

Set the repository Actions variable MERGE_QUEUE_CLEANUP_ENABLED to true.
Read the first scan summaries after the change. The table title changes to "Requested merge-group cancellations".
Find out if a cancelled queued job releases a runs-on instance that is already starting. This decides how much of the estimated saving is real.
A scheduled scan can fail when it and a merge_group scan cancel the same run at the same time. Count how often this happens.
Compare the merge queue runner time before and after the change.

4. Stop cancellation

To go back to observation mode, remove the variable or set it to false.
To stop all scans, disable the workflow in GitHub Actions.

@kumarUjjawal

Thanks @kumarUjjawal -- this sounds like a good plan to me 🙏

ci: cancel obsolete merge queue runs (apache#26760)

ci: Skip macOS tests in forks (apache#27017)

Navigation Menu

Repository navigation

Reduce GitHub Actions usage (DataFusion is currently the top ASF consumer of GHA minutes) #25148

Is your feature request related to a problem or challenge?

History

Useful resources

Related issues

CI time breakdown by workflow

1. CI — 49.31% (~165,900 min)

2. Rust — 27.16% (~91,350 min)

3. Datafusion extended tests — 10.85% (~36,500 min)

Everything else (< 2.5% each)

Observations

26 remaining items

Rollout plan for merge queue cleanup (#26760)

1. Observe for at least one week

2. Check the results before enabling cancellation

3. Enable cancellation

4. Stop cancellation

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Reduce GitHub Actions usage (DataFusion is currently the top ASF consumer of GHA minutes) #25148

Description

Is your feature request related to a problem or challenge?

History

Useful resources

Related issues

Activity

alamb commented on Sep 10, 2026

CI time breakdown by workflow

1. CI — 49.31% (~165,900 min)

2. Rust — 27.16% (~91,350 min)

3. Datafusion extended tests — 10.85% (~36,500 min)

Everything else (< 2.5% each)

Observations

alamb commented on Sep 10, 2026

alamb commented on Sep 10, 2026

alamb commented on Sep 10, 2026

comphead commented on Sep 10, 2026

comphead commented on Sep 10, 2026

rluvaton commented on Sep 10, 2026

comphead commented on Sep 10, 2026

alamb commented on Sep 10, 2026

adriangb commented on Sep 10, 2026

comphead commented on Sep 10, 2026

AdamGS commented on Sep 11, 2026

blaginin commented on Sep 11, 2026

blaginin commented on Sep 11, 2026

26 remaining items

kumarUjjawal commented on Sep 22, 2026

kumarUjjawal commented on Oct 27, 2026

Rollout plan for merge queue cleanup (#26760)

1. Observe for at least one week

2. Check the results before enabling cancellation

3. Enable cancellation

4. Stop cancellation

alamb commented on Oct 27, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions