From 287652b9896f81098e5942ad1586fbf65208028a Mon Sep 17 00:00:00 2001 From: Nicholas Chammas Date: Sun, 23 Aug 2026 12:05:23 -0400 Subject: [PATCH 1/4] remove merge script --- .asf.yaml | 7 ++ README.md | 4 - _config.yml | 2 +- committers.md | 26 +---- merge_pr.py | 290 -------------------------------------------------- 5 files changed, 11 insertions(+), 318 deletions(-) delete mode 100755 merge_pr.py diff --git a/.asf.yaml b/.asf.yaml index b251595269f..f50e63e22a1 100644 --- a/.asf.yaml +++ b/.asf.yaml @@ -23,3 +23,10 @@ staging: publish: whoami: asf-site + +github: + enabled_merge_buttons: + squash: true + squash_commit_message: PR_TITLE_AND_DESC + merge: false + rebase: false diff --git a/README.md b/README.md index bef477a201f..4d4106f2c26 100644 --- a/README.md +++ b/README.md @@ -43,7 +43,3 @@ compile phase, use the following syntax: You probably don't need to install that unless you want to regenerate the Pygments CSS file. It requires Python, and can be installed by running `sudo easy_install Pygments`. - -## Merge PR - -To merge pull request, use the `merge_pr.py` script which also squashes the commits. diff --git a/_config.yml b/_config.yml index 98f05068c08..4fdba1f8418 100644 --- a/_config.yml +++ b/_config.yml @@ -4,6 +4,6 @@ kramdown: entity_output: symbol permalink: none destination: site -exclude: ['README.md', 'content', 'LICENSE', 'merge_pr.py', 'Gemfile', 'Gemfile.lock'] +exclude: ['README.md', 'content', 'LICENSE', 'Gemfile', 'Gemfile.lock'] keep_files: ['docs', 'static', 'llms.txt'] url: https://spark.apache.org diff --git a/committers.md b/committers.md index 9ec78d818bc..0d26138e2fc 100644 --- a/committers.md +++ b/committers.md @@ -245,31 +245,11 @@ Once a PR is merged please leave a comment on the PR stating which branch(es) it

Merging website pull requests

-Pull requests to the Spark website (the -apache/spark-website repository) are merged -separately from the main Spark repo. Unlike `apache/spark`, this repository uses `asf-site` as its -default branch, and there is no `master` branch. +Pull requests to the [Spark website repo](https://github.com/apache/spark-website) are merged separately from the main Spark repo. Unlike `apache/spark`, this repository uses `asf-site` as its default branch; there is no `master` branch. -Website PRs are merged using the -merge_pr.py script -in the root of the `spark-website` repository, which squashes the pull request's changes into one -commit, just like `merge_spark_pr.py` does for the main repo. +There is no merge script for the website. Use GitHub's "Squash and merge" button to merge PRs. -To use it, set up remotes in your local clone of `spark-website` the same way as for the main repo: -an `apache` remote (the default `PUSH_REMOTE_NAME`, used for pushing the squashed commit) and an -`apache-github` remote (the default `PR_REMOTE_NAME`, used for pulling the change), both pointing at -`git@github.com:apache/spark-website.git`. Then run the script from the repository root: - -``` -./merge_pr.py -``` - -The script is interactive and walks you through selecting the PR and pushing the merge to the -`asf-site` branch. Before merging, make sure the pull request includes the regenerated HTML under -`site/` in addition to the Markdown/HTML source changes; see the repository's `README.md` for how -the site is built. - -

Policy on backporting bug fixes

+## Policy on backporting bug fixes From `pwendell`: diff --git a/merge_pr.py b/merge_pr.py deleted file mode 100755 index ad982d0078d..00000000000 --- a/merge_pr.py +++ /dev/null @@ -1,290 +0,0 @@ -#!/usr/bin/env python - -# -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -# Utility for creating well-formed pull request merges and pushing them to Apache. -# usage: ./merge_pr.py (see config env vars below) -# -# This utility assumes you already have local a spark-website git folder and that you -# have added remotes corresponding to both (i) the github apache spark-website -# mirror and (ii) the apache git repo. - -import json -import os -import subprocess -import sys -if sys.version < '3': - input = raw_input # noqa - from urllib2 import urlopen - from urllib2 import Request - from urllib2 import HTTPError -else: - from urllib.request import urlopen - from urllib.request import Request - from urllib.error import HTTPError - - -# Remote name which points to the Github site -PR_REMOTE_NAME = os.environ.get("PR_REMOTE_NAME", "apache-github") -# Remote name which points to Apache git -PUSH_REMOTE_NAME = os.environ.get("PUSH_REMOTE_NAME", "apache") -# OAuth key used for issuing requests against the GitHub API. If this is not defined, then requests -# will be unauthenticated. You should only need to configure this if you find yourself regularly -# exceeding your IP's unauthenticated request rate limit. You can create an OAuth key at -# https://github.com/settings/tokens. This script only requires the "public_repo" scope. -GITHUB_OAUTH_KEY = os.environ.get("GITHUB_OAUTH_KEY") - - -GITHUB_BASE = "https://github.com/apache/spark-website/pull" -GITHUB_API_BASE = "https://api.github.com/repos/apache/spark-website" -# Prefix added to temporary branches -BRANCH_PREFIX = "PR_TOOL" - - -def get_json(url): - try: - request = Request(url) - if GITHUB_OAUTH_KEY: - request.add_header('Authorization', 'token %s' % GITHUB_OAUTH_KEY) - return json.load(urlopen(request)) - except HTTPError as e: - if "X-RateLimit-Remaining" in e.headers and e.headers["X-RateLimit-Remaining"] == '0': - print("Exceeded the GitHub API rate limit; see the instructions in " + - "dev/merge_pr.py to configure an OAuth token for making authenticated " + - "GitHub requests.") - else: - print("Unable to fetch URL, exiting: %s" % url) - sys.exit(-1) - - -def fail(msg): - print(msg) - clean_up() - sys.exit(-1) - - -def run_cmd(cmd): - print(cmd) - if isinstance(cmd, list): - return subprocess.check_output(cmd).decode('utf-8') - else: - return subprocess.check_output(cmd.split(" ")).decode('utf-8') - - -def continue_maybe(prompt): - result = input("\n%s (y/n): " % prompt) - if result.lower() != "y": - fail("Okay, exiting") - - -def clean_up(): - if 'original_head' in globals(): - print("Restoring head pointer to %s" % original_head) - run_cmd("git checkout %s" % original_head) - - branches = run_cmd("git branch").replace(" ", "").split("\n") - - for branch in list(filter(lambda x: x.startswith(BRANCH_PREFIX), branches)): - print("Deleting local branch %s" % branch) - run_cmd("git branch -D %s" % branch) - - -# merge the requested PR and return the merge hash -def merge_pr(pr_num, target_ref, title, body, pr_repo_desc): - pr_branch_name = "%s_MERGE_PR_%s" % (BRANCH_PREFIX, pr_num) - target_branch_name = "%s_MERGE_PR_%s_%s" % (BRANCH_PREFIX, pr_num, target_ref.upper()) - run_cmd("git fetch %s pull/%s/head:%s" % (PR_REMOTE_NAME, pr_num, pr_branch_name)) - run_cmd("git fetch %s %s:%s" % (PUSH_REMOTE_NAME, target_ref, target_branch_name)) - run_cmd("git checkout %s" % target_branch_name) - - had_conflicts = False - try: - run_cmd(['git', 'merge', pr_branch_name, '--squash']) - except Exception as e: - msg = "Error merging: %s\nWould you like to manually fix-up this merge?" % e - continue_maybe(msg) - msg = "Okay, please fix any conflicts and 'git add' conflicting files... Finished?" - continue_maybe(msg) - had_conflicts = True - - commit_authors = run_cmd(['git', 'log', 'HEAD..%s' % pr_branch_name, - '--pretty=format:%an <%ae>']).split("\n") - distinct_authors = sorted(set(commit_authors), - key=lambda x: commit_authors.count(x), reverse=True) - primary_author = input( - "Enter primary author in the format of \"name \" [%s]: " % - distinct_authors[0]) - if primary_author == "": - primary_author = distinct_authors[0] - - run_cmd(['git', 'log', 'HEAD..%s' % pr_branch_name, - '--pretty=format:%h [%an] %s']).split("\n\n") - - merge_message_flags = [] - - merge_message_flags += ["-m", title] - if body is not None: - # We remove @ symbols from the body to avoid triggering e-mails - # to people every time someone creates a public fork of Spark. - merge_message_flags += ["-m", body.replace("@", "")] - - authors = "\n".join(["Author: %s" % a for a in distinct_authors]) - - merge_message_flags += ["-m", authors] - - if had_conflicts: - committer_name = run_cmd("git config --get user.name").strip() - committer_email = run_cmd("git config --get user.email").strip() - message = "This patch had conflicts when merged, resolved by\nCommitter: %s <%s>" % ( - committer_name, committer_email) - merge_message_flags += ["-m", message] - - # The string "Closes #%s" string is required for GitHub to correctly close the PR - merge_message_flags += ["-m", "Closes #%s from %s." % (pr_num, pr_repo_desc)] - - run_cmd(['git', 'commit', '--author="%s"' % primary_author] + merge_message_flags) - - continue_maybe("Merge complete (local ref %s). Push to %s?" % ( - target_branch_name, PUSH_REMOTE_NAME)) - - try: - run_cmd('git push %s %s:%s' % (PUSH_REMOTE_NAME, target_branch_name, target_ref)) - except Exception as e: - clean_up() - fail("Exception while pushing: %s" % e) - - merge_hash = run_cmd("git rev-parse %s" % target_branch_name)[:8] - clean_up() - print("Pull request #%s merged!" % pr_num) - print("Merge hash: %s" % merge_hash) - return merge_hash - - -def cherry_pick(pr_num, merge_hash, default_branch): - pick_ref = input("Enter a branch name [%s]: " % default_branch) - if pick_ref == "": - pick_ref = default_branch - - pick_branch_name = "%s_PICK_PR_%s_%s" % (BRANCH_PREFIX, pr_num, pick_ref.upper()) - - run_cmd("git fetch %s %s:%s" % (PUSH_REMOTE_NAME, pick_ref, pick_branch_name)) - run_cmd("git checkout %s" % pick_branch_name) - - try: - run_cmd("git cherry-pick -sx %s" % merge_hash) - except Exception as e: - msg = "Error cherry-picking: %s\nWould you like to manually fix-up this merge?" % e - continue_maybe(msg) - msg = "Okay, please fix any conflicts and finish the cherry-pick. Finished?" - continue_maybe(msg) - - continue_maybe("Pick complete (local ref %s). Push to %s?" % ( - pick_branch_name, PUSH_REMOTE_NAME)) - - try: - run_cmd('git push %s %s:%s' % (PUSH_REMOTE_NAME, pick_branch_name, pick_ref)) - except Exception as e: - clean_up() - fail("Exception while pushing: %s" % e) - - pick_hash = run_cmd("git rev-parse %s" % pick_branch_name)[:8] - clean_up() - - print("Pull request #%s picked into %s!" % (pr_num, pick_ref)) - print("Pick hash: %s" % pick_hash) - return pick_ref - - -def fix_version_from_branch(branch, versions): - # Note: Assumes this is a sorted (newest->oldest) list of un-released versions - if branch == "master": - return versions[0] - else: - branch_ver = branch.replace("branch-", "") - return list(filter(lambda x: x.name.startswith(branch_ver), versions))[-1] - - -def get_current_ref(): - ref = run_cmd("git rev-parse --abbrev-ref HEAD").strip() - if ref == 'HEAD': - # The current ref is a detached HEAD, so grab its SHA. - return run_cmd("git rev-parse HEAD").strip() - else: - return ref - - -def main(): - global original_head - - original_head = get_current_ref() - - pr_num = input("Which pull request would you like to merge? (e.g. 34): ") - pr = get_json("%s/pulls/%s" % (GITHUB_API_BASE, pr_num)) - pr_events = get_json("%s/issues/%s/events" % (GITHUB_API_BASE, pr_num)) - - url = pr["url"] - - title = pr["title"] - - body = pr["body"] - target_ref = pr["base"]["ref"] - user_login = pr["user"]["login"] - base_ref = pr["head"]["ref"] - pr_repo_desc = "%s/%s" % (user_login, base_ref) - - # Merged pull requests don't appear as merged in the GitHub API; - # Instead, they're closed by asfgit. - merge_commits = \ - [e for e in pr_events if e["actor"]["login"] == "asfgit" and e["event"] == "closed"] - - if merge_commits: - merge_hash = merge_commits[0]["commit_id"] - message = get_json("%s/commits/%s" % (GITHUB_API_BASE, merge_hash))["commit"]["message"] - - print("Pull request %s has already been merged, assuming you want to backport" % pr_num) - commit_is_downloaded = run_cmd(['git', 'rev-parse', '--quiet', '--verify', - "%s^{commit}" % merge_hash]).strip() != "" - if not commit_is_downloaded: - fail("Couldn't find any merge commit for #%s, you may need to update HEAD." % pr_num) - - print("Found commit %s:\n%s" % (merge_hash, message)) - sys.exit(0) - - if not bool(pr["mergeable"]): - msg = "Pull request %s is not mergeable in its current form.\n" % pr_num + \ - "Continue? (experts only!)" - continue_maybe(msg) - - print("\n=== Pull Request #%s ===" % pr_num) - print("title\t%s\nsource\t%s\ntarget\t%s\nurl\t%s" % ( - title, pr_repo_desc, target_ref, url)) - continue_maybe("Proceed with merging pull request #%s?" % pr_num) - - merge_pr(pr_num, target_ref, title, body, pr_repo_desc) - - -if __name__ == "__main__": - import doctest - (failure_count, test_count) = doctest.testmod() - if failure_count: - exit(-1) - try: - main() - except: - clean_up() - raise From a01c1bdca57715f9e92e978f508d664d26e17692 Mon Sep 17 00:00:00 2001 From: Nicholas Chammas Date: Sun, 23 Aug 2026 12:19:23 -0400 Subject: [PATCH 2/4] wording tweak --- committers.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/committers.md b/committers.md index 0d26138e2fc..78d679a7089 100644 --- a/committers.md +++ b/committers.md @@ -245,9 +245,9 @@ Once a PR is merged please leave a comment on the PR stating which branch(es) it

Merging website pull requests

-Pull requests to the [Spark website repo](https://github.com/apache/spark-website) are merged separately from the main Spark repo. Unlike `apache/spark`, this repository uses `asf-site` as its default branch; there is no `master` branch. +Pull requests to the [Spark website repo](https://github.com/apache/spark-website) are merged separately from the [main Spark repo](https://github.com/apache/spark). Unlike the main repo, the website uses `asf-site` as its default branch; there is no `master` branch. -There is no merge script for the website. Use GitHub's "Squash and merge" button to merge PRs. +The website repo does not use a merge script. Instead, use GitHub's "Squash and merge" button to merge PRs. ## Policy on backporting bug fixes From a568941f220af4d05603850940ef58ec0bc47a31 Mon Sep 17 00:00:00 2001 From: Nicholas Chammas Date: Sun, 23 Aug 2026 12:20:33 -0400 Subject: [PATCH 3/4] committers html --- site/committers.html | 25 +++---------------------- 1 file changed, 3 insertions(+), 22 deletions(-) diff --git a/site/committers.html b/site/committers.html index bac471eb802..23c79fcf058 100644 --- a/site/committers.html +++ b/site/committers.html @@ -711,30 +711,11 @@

Merge script

Merging website pull requests

-

Pull requests to the Spark website (the -apache/spark-website repository) are merged -separately from the main Spark repo. Unlike apache/spark, this repository uses asf-site as its -default branch, and there is no master branch.

- -

Website PRs are merged using the -merge_pr.py script -in the root of the spark-website repository, which squashes the pull request’s changes into one -commit, just like merge_spark_pr.py does for the main repo.

- -

To use it, set up remotes in your local clone of spark-website the same way as for the main repo: -an apache remote (the default PUSH_REMOTE_NAME, used for pushing the squashed commit) and an -apache-github remote (the default PR_REMOTE_NAME, used for pulling the change), both pointing at -git@github.com:apache/spark-website.git. Then run the script from the repository root:

- -
./merge_pr.py
-
+

Pull requests to the Spark website repo are merged separately from the main Spark repo. Unlike the main repo, the website uses asf-site as its default branch; there is no master branch.

-

The script is interactive and walks you through selecting the PR and pushing the merge to the -asf-site branch. Before merging, make sure the pull request includes the regenerated HTML under -site/ in addition to the Markdown/HTML source changes; see the repository’s README.md for how -the site is built.

+

The website repo does not use a merge script. Instead, use GitHub’s “Squash and merge” button to merge PRs.

-

Policy on backporting bug fixes

+

Policy on backporting bug fixes

From pwendell:

From d3ce43b45104eab170b0c383ed5b74f365ec6c28 Mon Sep 17 00:00:00 2001 From: Nicholas Chammas Date: Tue, 25 Aug 2026 09:00:26 -0400 Subject: [PATCH 4/4] revert heading change --- committers.md | 2 +- site/committers.html | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/committers.md b/committers.md index 78d679a7089..3c95a561a05 100644 --- a/committers.md +++ b/committers.md @@ -249,7 +249,7 @@ Pull requests to the [Spark website repo](https://github.com/apache/spark-websit The website repo does not use a merge script. Instead, use GitHub's "Squash and merge" button to merge PRs. -## Policy on backporting bug fixes +

Policy on backporting bug fixes

From `pwendell`: diff --git a/site/committers.html b/site/committers.html index 23c79fcf058..2476fe4cf58 100644 --- a/site/committers.html +++ b/site/committers.html @@ -715,7 +715,7 @@

Merging website pull requests

The website repo does not use a merge script. Instead, use GitHub’s “Squash and merge” button to merge PRs.

-

Policy on backporting bug fixes

+

Policy on backporting bug fixes

From pwendell: