Nightly red on two counts: a stale submodule PR blocks Configure, and publish is deadlocked on an already-published version

[oleksandrlazarenko-pi]

Nightly Pipeline run 34306635583
(2026-09-09 03:18, scheduled) failed on two independent counts. Neither is a test
failure — the tests never ran on one side, and the packages were built fine on the
other.

---

1. Configure - 125 dies on a submodule merge conflict, so nothing is built

Checking out PR '#125 update-submodules/main->main : Update submodules'
Submodule path 'owid-java': checked out '764996e03ae231d871168cdec2787e99bac9160c'
Merging in any changes from main
hint: Recursive merging with submodules currently only supports trivial cases.
Failed to merge submodule owid-java (commits don't follow merge-base)
CONFLICT (submodule): Merge conflict in owid-java
Automatic merge failed; fix conflicts and then commit the result.
NativeCommandExitException: .../steps/checkout-pr.ps1:26
  26 |     git merge origin/$Branch
     | Program "git" ended with non-zero exit code: 1 (0x01).

Configure exits 1, so all twelve Build and Test jobs for PR 125 are skipped
and the nightly is red without a single test having run.

Why the conflict exists

PR #125 is the automated update-submodules/main PR, opened 2026-09-06 03:19. It
moves owid-java 694f633 -> 764996e. Later the same day, #126 ("BUILD: Move
owid-java to the commit that refuses redirects when fetching a key", commit
70f00438) moved main to 44d4e0b by hand. So base, ours and theirs all differ
on the same gitlink and GitHub now reports PR #125 as mergeable: CONFLICTING.

The content is not actually in conflict: 764996e is an ancestor of 44d4e0b,
so the merge is the trivial "take theirs" case. Git cannot see that in CI because
common-ci/steps/clone-repo.ps1 clones with --shallow-submodules, and
gh pr checkout --recurse-submodules then fetches the PR's pointer as a second
shallow fetch (* branch 764996e0... -> FETCH_HEAD). The submodule clone holds two
disconnected commits with no shared history, so git cannot compute a merge base and
reports exactly what it printed: commits don't follow merge-base.

Why it never heals on its own

common-ci/nightly-submodule-update.ps1 recreates the branch from main, updates the
submodules, and then:

} else {
    Write-Output "...No submodule changes, so not creating a pull request."
}

Since main already carries 44d4e0b, there are no changes, so the job exits early —
it does not push, and it neither refreshes nor closes the existing stale branch. Run
34306635583 shows exactly that annotation while PR #125 is what breaks it. So the
stale PR sits there red-lighting the nightly indefinitely: Configure - 125 failed
on 09-08, on 09-09 03:18 and again on 09-09 12:25 (run 34350954488).

Fixes

• Now, in this repo: close PR Update submodules #125 (merging it would downgrade owid-java from
  44d4e0b to 764996e), or reset update-submodules/main onto current main.
• In common-ci, so it does not come back:
  1. steps/clone-repo.ps1 — drop --shallow-submodules, or unshallow before the
     merge (git submodule foreach git fetch --unshallow), so git can resolve the
     trivial ancestor case instead of conflicting.
  2. nightly-submodule-update.ps1 — when there are no submodule changes, reconcile
     the existing update-submodules/<branch> PR (close it, or reset the branch onto
     main) rather than leaving it stale.
  3. Consider having the nightly skip a PR whose mergeable is CONFLICTING with a
     warning instead of failing the whole run — a conflicted PR is a fact about the
     PR, not a break in the pipeline.

---

2. Nightly Publish / Package is deadlocked on a version that was already published

Uploading the bundle...
Deployment id: 915c6ca0-bde7-49c9-a956-e7eea9bae61c
Checking status (1)...
Write-Error: .../ci/publish-package.ps1:8
     | Publishing failed: {
     |   "pkg:maven/com.51degrees/pipeline.core@4.5.26": [
     |     "Component with package url: 'pkg:maven/com.51degrees/pipeline.core@4.5.26' already exists" ],
     |   ... (every one of the ~25 artefacts, all @4.5.26)

Configure computed version=4.5.26, but 4.5.26 is already on Maven Central.

How it got stuck

Publish run 33769690244
(2026-09-03 15:02, on main) uploaded 4.5.26 and then died on the first status
poll:

Uploading the bundle...
Deployment id: 9d7cd85c-6830-4fc8-bb78-73ce0ce91cca
Checking status (1)...
Invoke-WebRequest: .../java/publish-package-maven.ps1:25
  25 |     $resp = Invoke-WebRequest @args -Uri "$api/status?id=$id" | Conve …
     | 502 Bad Gateway

publish-package-maven.ps1 loops on the deployment state but the
Invoke-WebRequest itself is unguarded, and the script runs under
$ErrorActionPreference = "Stop", so one transient 502 from Sonatype aborts it. The
deployment was AUTOMATIC and had already been accepted, so Central went on to
publish 4.5.26 — but in nightly-publish.package.ps1 the Update Tag group runs
after Publish Packages, so the abort meant no 4.5.26 tag and no release.

GitVersion derives the next version from tags. With the newest tag still 4.5.25, it
answered 4.5.26 every night, and Central rejected it every night:

Run	Date	Version computed	Package
33832753368	09-04	4.5.26	❌ already exists
33941451459	09-05	4.5.26	❌
34008682102	09-06	4.5.26	❌
34079295446	09-07 03:20	4.5.26	❌
34183073019	09-08	4.5.26	❌
34306635583	09-09 03:18	4.5.26	❌ (this run)
34350954488	09-09 12:25	4.5.27	✅ (release 4.5.27 at 12:33)

Six consecutive nights with no release. It only cleared once the 4.5.26 tag
appeared (some time between 03:20 and 12:25 on 09-09), after which GitVersion moved
to 4.5.27 and the publish went through. Note there is still no 4.5.26 GitHub
release, only the tag.

Fixes (these land in common-ci)

1. Retry the status poll. Wrap the Invoke-WebRequest in
   java/publish-package-maven.ps1 so an HTTP/network error is retried like any
   other non-terminal state instead of aborting a deployment that is already in
   flight.
2. Treat "already exists" as recoverable. When every component in the FAILED
   response says "already exists", the version is published and the run should tag it
   (or fail with a message that says exactly that) instead of repeating the same
   attempt every night.
3. Reconcile version and tag. Publishing to Central and tagging the repo need to
   agree, otherwise an abort in between deadlocks all future publishes. Either check
   Central for the computed version before uploading, or make the next-version
   calculation aware of what is actually published.

Related: common-ci#228 (packages pushed without being verified first) is the same
family of problem in the NuGet path.

---

Acceptance criteria

• Configure no longer fails the nightly because an automated submodule PR has gone
  stale; either the PR is refreshed/closed automatically or the run skips it with a
  warning.
• A transient error from Sonatype does not leave a version published but untagged,
  and a version that is already on Central does not repeat as a failure every night.
• Ten consecutive scheduled nightly runs green, with Build and Test actually
  executing for every open PR.

[oleksandrlazarenko-pi]

The publish half of this (section 2) is now tracked in common-ci as 51Degrees/common-ci#233, where the fix has to land: the status poll needs a retry, an "already exists" response needs reconciling rather than repeating, and publish-then-tag needs a recovery path. This issue keeps the pipeline-java side: PR #125 to close or reset, and the shallow-submodule / stale-PR behaviour behind section 1.