Skip to content

fix(cd): repair TiDBx ARM64 CentOS 7 repository URLs - #1114

Merged
ti-chi-bot[bot] merged 1 commit into
PingCAP-QE:mainfrom
dillon-zheng:fix/tidbx-centos7-arm64-vault
Oct 16, 2026
Merged

ti-chi-bot[bot] merged 1 commit into
PingCAP-QE:mainfrom
dillon-zheng:fix/tidbx-centos7-arm64-vault

Conversation

@dillon-zheng

Copy link
Copy Markdown
Contributor

Problem

The CentOS 7 publish run failed reproducibly in TiDBx's ARM64 non-root yum install sudo step:

https://github.com/PingCAP-QE/artifacts/actions/runs/38860905596/job/116842768156

The pinned TiKV parent image supplies old CentOS ARM64 Vault repository URLs. vault.centos.org/altarch/7/... returns HTTP 403 for the base, updates, extras, SCL rh, and SCL sclo metadata. Direct HTTP, HTTPS, and explicit official 7.9.2009 URL probes also reproduce 403; this is not a Go 1.27 compilation failure or a transient runner timeout. A blind rerun would reuse the broken configuration.

Minimal fix

Only dockerfiles/cd/builders/tidbx/centos7/Dockerfile changes (5 added lines):

  • Before the shared builder's first yum operation, repair ARM64 only CentOS altarch/$releasever and altarch/7 sources to the HTTPS Aliyun CentOS 7.9.2009 archive.
  • Clear old metadata in the existing yum cache mount, then retain the original update/OpenSSL installation.
  • Both root and descendant non-root stages inherit the fix across the existing Go profiles.

The literal $releasever form matters: it is present in the real parent image's CentOS-Base.repo; replacing only a literal /7/ would leave the base/update/extra sources broken. The narrow BRE preserves disabled historical-version entries such as /7.6.1810/ and unrelated hosts.

Unchanged: AMD64 behavior, repository enablement and GPG/TLS settings, pinned TiKV/Rust base, all Go versions/profiles, sudo/user setup, workflows, package routing, and generated release files. No skip_if_unavailable, repo disabling, or new verification bypass is added.

Validation

  • RED: actual ARM64 CentOS 7 yum with the inherited base repository configuration reproduces the Vault 403.
  • GREEN: applying the actual Dockerfile repair refreshes all five affected CentOS/SCL repos, then installs sudo-1.8.23-10.el7_9.3.aarch64 and runs sudo --version successfully. HTTPS and package GPG checks remain enabled; the test imports and uses the installed CentOS/AArch64 signing keys.
  • The actual first Dockerfile RUN was also exercised in Linux with controlled arch/yum commands: ARM64 handles both $releasever and literal 7, cleans stale metadata before the original yum commands, and preserves historical/unrelated entries; AMD64 keeps the original repo bytes and original yum calls.
  • Skaffold 2.23.0 resolves all 6 TiDBx root/non-root × Go 1.27/1.26/1.25 configurations.
  • git diff --check.
  • Read-only pi-deepseek review-loop: CLEAN, round 1 (1m52s); separate final read-only verification PASS after GREEN completed.
  • Full pinned TiKV/Rust/Go TiDBx image builds are delegated to PR CI, not claimed passed locally. The focused local integration test uses a real ARM64 CentOS 7 container, the inherited base repo bytes, and matching SCL alias fixtures; it does not reproduce the entire parent image or test its untouched EPEL source.

Runtime line coverage: not measured (Dockerfile change); the architecture branches and failing yum/sudo operation were exercised as above. Local reproduction script and logs are kept outside the committed diff.

Publication impact

In the failed run:

  • tidbx:v2026.10.11-5-g29710e1-centos7-go1.27 already exists as an amd64 + arm64 manifest list.
  • The final tidbx:...-centos7-non-root-go1.27 manifest is missing; only its amd64 staging tag was pushed.
  • 15 later matrix jobs were cancelled by fail-fast, including Go 1.27 tiflow and the old-profile jobs. Their cancellation is not evidence of their own build defects.

This is a narrow TiDBx repair. After a separately authorized merge, the changed Dockerfile triggers a new TiDBx publish run and new SHA-derived tags, which still require manifest/label verification. Rerunning the original SHA would not include this code fix. Other components cancelled in the original run need a separate publication trigger if their new tags are required; this PR does not silently rebuild unrelated components or change routing.

Do not merge automatically; left for user review.

@ti-chi-bot ti-chi-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have already done a preliminary review for you, and I hope to help you do a better job.

Summary:
This PR fixes the ARM64 CentOS 7 repository URLs in the TiDBx Dockerfile by replacing old Vault URLs that produce HTTP 403 errors with Aliyun archive URLs for CentOS 7.9.2009. The fix is applied conditionally for ARM64 architecture before any yum operation, ensuring the cached repo metadata is cleared and the existing yum commands run cleanly. The approach is minimal, targeted, and well-explained in the description. The code change is concise and correctly scoped to the affected Dockerfile. Overall, the change appears sound and addresses a critical build failure for ARM64 CentOS 7 images.


Critical Issues

  • None identified. The fix correctly targets only ARM64, preserves existing repo settings, and clears yum cache to prevent stale metadata errors.

Code Improvements

  • Docker RUN command readability (dockerfiles/cd/builders/tidbx/centos7/Dockerfile, lines ~12-20):
    The inline shell commands in the RUN statement mix multiple operations with && \ and a long if block, which can be hard to read and maintain. Consider splitting into multiple RUN steps or using a shell script snippet for clarity. For example:

    RUN --mount=type=cache,target=/var/cache/yum /bin/bash -c '\
        if [ "$(arch)" = "aarch64" ]; then \
            sed -i "s#http://vault.centos.org/altarch/\(\$releasever\|7\)/#https://mirrors.aliyun.com/centos-vault/altarch/7.9.2009/#g" /etc/yum.repos.d/*.repo && \
            yum clean all; \
        fi && \
        yum update -y && \
        yum install -y openssl-devel && \
        yum clean all'

    This makes the shell logic explicit and easier to modify later.

  • Use of literal $releasever in sed regex:
    The PR description notes the importance of preserving the literal $releasever. Verify that the $ is escaped properly in the regex so it is not expanded by the shell during sed execution. The current usage inside single quotes appears correct, but a comment in the Dockerfile about this subtlety would help future maintainers.


Best Practices

  • Documentation and comments (dockerfiles/cd/builders/tidbx/centos7/Dockerfile, line 12):
    The single comment # Repair the pinned TiKV base's ARM64 Vault sources before accessing yum metadata. is helpful but could be expanded to mention why this is needed (HTTP 403 errors on old vault URLs) and that this fix is ARM64-specific, clarifying the rationale for future readers.

  • Testing coverage:
    The PR description documents thorough manual and CI validation steps but no automated tests or integration tests are added here (understandable for a Dockerfile change). Consider adding a minimal smoke test in CI that verifies the yum repos and package installation on ARM64 CentOS 7 images after this fix, if not already covered by existing tests.

  • Consistency in quoting and escaping:
    In the sed command, the parentheses are escaped as \(\$releasever\|7\). This is correct for basic regex, but since GNU sed allows extended regex with -r or -E, consider using extended regex for readability, e.g.:

    sed -r -i 's#http://vault\.centos\.org/altarch/(\$releasever|7)/#https://mirrors.aliyun.com/centos-vault/altarch/7.9.2009/#g' /etc/yum.repos.d/*.repo

    This avoids backslash clutter and improves maintainability.


Summary of actionable suggestions:

  • Add an explanatory comment in the Dockerfile about the $releasever literal preservation and the ARM64-specific nature of the fix.
  • Consider refactoring the RUN command for readability, possibly splitting into multiple RUN statements or using a shell script snippet.
  • Optionally use extended regex in sed to improve clarity.
  • Add or verify CI test coverage that exercises the ARM64 CentOS 7 yum repos after this fix.

These improvements will increase maintainability and clarity without changing the functional fix.

@ti-chi-bot ti-chi-bot Bot added the size/XS label Oct 16, 2026
@dillon-zheng

Copy link
Copy Markdown
Contributor Author

/approve

@ti-chi-bot

ti-chi-bot Bot commented Oct 16, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dillon-zheng

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added the approved label Oct 16, 2026
@ti-chi-bot
ti-chi-bot Bot merged commit e392e37 into PingCAP-QE:main Oct 16, 2026
16 checks passed
@dillon-zheng
dillon-zheng deleted the fix/tidbx-centos7-arm64-vault branch October 16, 2026 11:23
ti-chi-bot Bot pushed a commit that referenced this pull request Oct 16, 2026
## Purpose

Rebuild and publish the **full seven-component CentOS 7 Go-builder
matrix using the repaired code**, rather than stopping after TiDBx alone
was recovered.

- Original full publish run:
https://github.com/PingCAP-QE/artifacts/actions/runs/38860905596 —
TiDBx's ARM64 Vault failure cancelled 15 later matrix jobs.
- The fix in #1114 merged as
`e392e375` and successfully published **only TiDBx**:
https://github.com/PingCAP-QE/artifacts/actions/runs/38872606668 . It
did not recover the original action's other cancelled components.
- Rerunning the original run reuses its old `29710e1` SHA, without the
fix. The existing `workflow_dispatch` parser also does not support that
event, so a manual dispatch is not a working full-build recovery path
without broader workflow changes.

## Change

**Seven whitespace-only insertions, one blank line per existing
`skaffold.centos7.yaml`:**

- ng-monitoring
- pd
- ticdc
- tidb
- tidb-dashboard
- tidbx
- tiflow

The parsed configurations are unchanged. The merge push matches the
existing release paths and changed-builder parser, selecting **all seven
builders** for Go **1.27 / 1.26 / 1.25**, including **root + non-root**
images and **linux/amd64 + linux/arm64**.

This produces the existing **21 component/profile publish jobs**, each
building both image variants for both architectures. The branch starts
from `upstream/main` **after the TiDBx repository repair**.

No Dockerfile, version, profile, workflow logic, Rust/C++ builder,
routing, or generated release-file change is included.

## Validation

- [x] Exactly seven one-blank-line insertions; YAML parsing is identical
before/after for every changed file.
- [x] Replayed both actual existing PR/release changed-scope shell
parsers on the uncommitted diff: **exactly all seven Go builders
selected**, Rust/C++ arrays empty.
- [x] Both workflows' path filters, three Go profiles, and root/non-root
build steps cover the changes.
- [x] Skaffold **2.23.0**: **42/42** root/non-root × profile × component
effective configs resolve, preserving exact Go **1.27.2 / 1.26.9 /
1.25.14** and both architectures.
- [x] Branch contains merged repair `e392e375`; fetched/rebased against
upstream before submission.
- [x] `git diff --check`.
- [x] Read-only **pi-deepseek review-loop: CLEAN**, first round (23
seconds), no findings/fixes.

Runtime line coverage: not measured (whitespace-only trigger). Full
image builds are performed by PR CI and, after merge, the release
workflow; no publication success or future tag is claimed yet.

## Completion boundary

PR checks use `--push=false`; **merging this trigger PR is what starts
the full publication run**. Record exact tags from that successful
publish run, verify final root/non-root manifest lists contain both
architectures and the expected per-architecture Go labels, and only then
consider the builder publication complete. No downstream routing changes
are made here.

**Leave unmerged for user review; auto-merge is not enabled.**
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant