Automatically generate and commit site HTML

.github/actions/build-html/action.yml

@@ -0,0 +1,22 @@
+name: Build HTML
+
+description: Set up Ruby and run the Jekyll build.
+
+inputs:
+  ruby-version:
+    description: Ruby version to use.
+    required: false
+    default: "3.4"
+
+runs:
+  using: composite
+  steps:
+  - name: Set up Ruby and Bundler
+    uses: ruby/setup-ruby@v1
+    with:
+      ruby-version: ${{ inputs.ruby-version }}
+      # This will use the version of Bundler specified in `Gemfile.lock`.
+      bundler-cache: true
+  - name: Run documentation build
+    shell: bash
+    run: bundle exec jekyll build

.github/pull_request_template.md

@@ -0,0 +1,7 @@
+<!--
+Include your source changes but not the Jekyll-generated HTML. A GitHub workflow will automatically generate and push the HTML under `site/` in a follow-up commit if necessary.
+
+If you are working through a release, follow the instructions in [that guide](../release-process.md).
+
+For more details review the main [README](../README.md).
+-->

.github/workflows/html-build.yml

@@ -0,0 +1,16 @@
+name: Build HTML
+
+on:
+  pull_request:
+    branches:
+    - asf-site
+
+jobs:
+  build:
+    name: Build HTML
+    runs-on: ubuntu-24.04
+    steps:
+    - name: Checkout Spark Website repository
+      uses: actions/checkout@v7
+    - name: Build HTML
+      uses: ./.github/actions/build-html

.github/workflows/html-push.yml

@@ -0,0 +1,42 @@
+name: Build and Push HTML
+
+on:
+  push:
+    branches:
+    - asf-site
+
+jobs:
+  commit:
+    name: Build and commit HTML to `asf-site`
+    # This condition is important. We don't want to trigger this job if the last
+    # commit was created _by_ this job!
+    if: >-
+      !(
+        contains(github.event.head_commit.message, '[html]') &&
+        github.event.head_commit.author.name == 'github-actions[bot]'
+      )
+    # Not technically necessary, but helps avoid spurious failures if multiple
+    # commits are pushed in rapid succession.
+    concurrency:
+      group: html-push-${{ github.ref }}
+      cancel-in-progress: true
+    runs-on: ubuntu-24.04
+    permissions:
+      contents: write
+    steps:
+    - name: Checkout Spark Website repository
+      uses: actions/checkout@v7
+    - name: Build HTML
+      uses: ./.github/actions/build-html
+    - name: Commit and push generated HTML
+      run: |
+        git config user.name "github-actions[bot]"
+        git config user.email "github-actions[bot]@users.noreply.github.com"
+        git add -f site/
+        if git diff --cached --quiet; then
+          echo "No changes to commit."
+        else
+          COMMIT_TITLE=$(git log -1 --pretty=%s)
+          git commit -m "[html] $COMMIT_TITLE"
+          git push
+        fi

.gitignore

@@ -4,4 +4,3 @@ target/
 .jekyll-cache/
 .jekyll-metadata
 .local_ruby_bundle
-site/python

Gemfile.lock

@@ -79,4 +79,4 @@ RUBY VERSION
    ruby 3.2.3p157
 
 BUNDLED WITH
-   2.4.19
+   2.4.22

README.md

@@ -1,10 +1,26 @@
-## Generating the website HTML
+# Apache Spark Main Website
 
-In this directory you will find text files formatted using Markdown, with an `.md` suffix.
+This repository captures the main Apache Spark website located at https://spark.apache.org. The programming docs under https://spark.apache.org/docs/ are [in the main Spark repo][main], not here. They are built separately for each release of Spark and then copied to the website under the `docs/` directory.
 
-Building the site requires [Ruby 3](https://www.ruby-lang.org), [Jekyll](http://jekyllrb.com/docs), and
-[Rouge](https://github.com/rouge-ruby/rouge). The most reliable way to ensure a compatible environment
-is to use the official Docker build image from the Apache Spark repository.
+[main]: https://github.com/apache/spark/tree/master/docs#readme
+
+## Contributing
+
+To contribute changes, build and test the site locally, then submit a pull request with your changes. Unless you are working through the [release guide](./release-process.md), you only need to commit changes to source files (typically Markdown). A [GitHub Actions workflow](.github/workflows/html-push.yml) will generate the corresponding HTML under `site/` and push it for you.
+
+## Building the site locally
+
+Building the site requires [Ruby 3](https://www.ruby-lang.org), [Jekyll](http://jekyllrb.com/docs), and [Rouge](https://github.com/rouge-ruby/rouge).
+
+```
+gem install bundler -v 2.4.22
+bundle install
+bundle exec jekyll serve
+```
+
+### Building the site with Docker
+
+The most reliable way to ensure a compatible environment is to use the official Docker build image from the Apache Spark repository.
 
 If you haven't already, clone the [Apache Spark](https://github.com/apache/spark) repository. Navigate to
 the Spark root directory and run the following command to create the builder image:
@@ -21,25 +37,12 @@ the Markdown files in the Docker container.
 .dev/build-docs.sh
 ```
 
-## Docs sub-dir
-
-The docs are not generated as part of the website. They are built separately for each release 
-of Spark from the Spark source repository and then copied to the website under the docs 
-directory. See the instructions for building those in the readme in the Spark 
-project's `/docs` directory.
-
-## Rouge and Pygments
-
-We also use [Rouge](https://github.com/rouge-ruby/rouge) for syntax highlighting in documentation Markdown pages.
-Its HTML output is compatible with CSS files designed for [Pygments](https://pygments.org/).
+## Deploying to production
 
-To mark a block of code in your Markdown to be syntax highlighted by `jekyll` during the 
-compile phase, use the following syntax:
+The website is deployed automatically by [ASF Infra][infra]. The deployment configuration is tracked by [.asf.yaml](./.asf.yaml) and is [documented here][asf-docs].
 
-    {% highlight scala %}
-    // Your Scala code goes here, you can replace Scala with many other
-    // supported languages too.
-    {% endhighlight %}
+One deployment detail that appears to be critical is the presence of the [`content`](./content/) symlink to `site/`. Even though ASF Infra is [aware of Jekyll][jek], we perhaps do not have the exact setup required for them to automatically use our [Jekyll config](./_config.yml) to understand where the site content lives. Without the `content` symlink, the website will just show a plain directory listing of the files in this repo.
 
-You probably don't need to install that unless you want to regenerate the Pygments CSS file.
-It requires Python, and can be installed by running `sudo easy_install Pygments`.
+[infra]: https://infra.apache.org
+[asf-docs]: https://github.com/apache/infrastructure-asfyaml/tree/main#readme
+[jek]: https://github.com/apache/infrastructure-asfyaml/tree/76d241ccef02e5397e10c173ebf04c07525311ea#jekyll_cms

release-process.md

@@ -224,6 +224,8 @@ Note that `dev/create-release/do-release-docker.sh` script (`finalize` step ) au
 - [Create and upload Spark Docker Images](#create-and-upload-spark-docker-images)
 - [Create an announcement](#create-an-announcement) 
 
+The `finalize` step pushes to the `asf-site` branch of spark-website. After that push, you may see a follow-up `[html]` commit from GitHub Actions. This is normal.
+
 Please manually verify the result after each step.
 
 <p align="right"><a href="#top">Return to top</a></p>
@@ -337,7 +339,7 @@ The website repository is located at
 It's recommended to not remove the generated docs of the latest RC, so that we can copy it to
 spark-website directly, otherwise you need to re-build the docs.
 
-```
+```sh
 # Build the latest docs
 $ git checkout v1.1.1
 $ cd docs
@@ -352,23 +354,22 @@ $ cp -R _site spark-website/site/docs/1.1.1
 $ cd spark-website/site/docs
 $ rm latest
 $ ln -s 1.1.1 latest
+$ cd ../..
+$ git add site/docs/1.1.1 site/docs/latest
+$ git commit -m "Add docs for Spark 1.1.1"
 ```
 
 <h4 id="update-the-rest-of-the-spark-website">Update the rest of the Spark website</h4>
 
-Next, update the rest of the Spark website. See how the previous releases are documented
-(all the HTML file changes are generated by `jekyll`). In particular:
+Next, update the rest of the Spark website. See how the previous releases are documented. In particular:
 
 * update `documentation.md` to add link to the docs for the new release
 * add the new release to `js/downloads.js` (attention to the order of releases)
 * update `downloads.md` to use the latest release in the linking example
 * add the new release to `site/static/versions.json` (attention to the order of releases) [for `spark version drop down` of the `PySpark` docs]
 * check `security.md` for anything to update
 
-```
-$ git add 1.1.1
-$ git commit -m "Add docs for Spark 1.1.1"
-```
+Commit these source files normally. You do not need to run `jekyll build` or commit anything else under `site/`. After the change is merged to `asf-site`, GitHub Actions generates the HTML under `site/` and pushes it for you.
 
 Then, create the release notes. Go to the
 <a href="https://issues.apache.org/jira/projects/SPARK?selectedItem=com.atlassian.jira.jira-projects-plugin:release-page">release page in JIRA</a>,
@@ -377,13 +378,7 @@ pick the release version from the list, then click on "Release Notes". Copy this
 `spark-2.1.2`. Create a new release post under `releases/_posts` to include this short URL. The date of the post should
 be the date you create it.
 
-Then run `bundle exec jekyll build` to update the `site` directory.
-
-Considering the Pull Request will be large, please separate the commits of code changes and generated `site` directory for an easier review.
-
-After merging the change into the `asf-site` branch, you may need to create a follow-up empty
-commit to force synchronization between ASF's git and the web site, and also the GitHub mirror.
-For some reason synchronization seems to not be reliable for this repository.
+After merging the change into the `asf-site` branch, confirm the GitHub Actions run finished and the generated HTML is on the site. You may need to create a follow-up empty commit to force synchronization between ASF's git and the website, and also the GitHub mirror. For some reason synchronization seems to not be reliable for this repository.
 
 On a related note, make sure the version is marked as released on JIRA. Go find the release page as above, eg.,
 [`https://issues.apache.org/jira/projects/SPARK/versions/12340295`](https://issues.apache.org/jira/projects/SPARK/versions/12340295),
@@ -447,8 +442,7 @@ The apache/spark-docker provides Dockerfiles and GitHub Action for Spark Docker
 <h3 id="create-an-announcement">Create an announcement</h3>
 
 Once everything is working (website docs, website changes) create an announcement on the website
-and then send an e-mail to the mailing list with a subject that looks something like `[ANNOUNCE] ...`. To create an announcement, create a post under
-`news/_posts` and then run `bundle exec jekyll build`.
+and then send an e-mail to the mailing list with a subject that looks something like `[ANNOUNCE] ...`. To create an announcement, create a post under `news/_posts`. GitHub Actions will automatically generate the corresponding HTML when you push to `asf-site`.
 
 Enjoy an adult beverage of your choice, and congratulations on making a Spark release.
 

sitemap.xml

@@ -151,9 +151,18 @@ sitemap: false
   <changefreq>weekly</changefreq>
 </url>
 {% endfor %}
-{% for page in site.pages %}{% if page.sitemap != false %}<url>
+{%- comment -%}
+Explicitly sort `site.pages` so that the order is consistent and we don't get spurious git diffs.
+`site.posts` doesn't have this issue because it's already sorted.
+See: https://jekyllrb.com/docs/variables/#site-variables
+{%- endcomment -%}
+{%- assign sorted_pages = site.pages | sort: "url" -%}
+{%- for page in sorted_pages -%}
+{%- if page.sitemap != false -%}
+<url>
   <loc>{{ site.url }}{{ page.url }}</loc>
   <changefreq>weekly</changefreq>
-</url>{% endif %}
-{% endfor %}
+</url>
+{% endif %}
+{%- endfor -%}
 </urlset>

third-party-projects.md

@@ -100,6 +100,6 @@ transforming, and analyzing genomic data using Apache Spark
 
 ## Adding new projects
 
-To add a project, open a pull request against the [spark-website](https://github.com/apache/spark-website)  repository. Add an entry to  [this markdown file](https://github.com/apache/spark-website/blob/asf-site/third-party-projects.md),  then run `jekyll build` to generate the HTML too. Include both in your pull request. See the README in this repo for more information.
+To add a project, open a pull request against the [spark-website](https://github.com/apache/spark-website)  repository. Add an entry to [this markdown file](https://github.com/apache/spark-website/blob/asf-site/third-party-projects.md) and follow the [README instructions](https://github.com/apache/spark-website#readme) to submit a pull request with your changes.
 
 Note that all project and product names should follow [trademark guidelines](/trademarks.html).