Linux/AppImage: bundled LD_LIBRARY_PATH still leaks into spawned git — stale bundled libnghttp2 breaks plain libcurl.so.4, blocking session creation (third recurrence of #644 / #1394)

[saurabh500]

Short summary

The Linux AppImage still exports its bundled lib directory on LD_LIBRARY_PATH and leaks it into spawned git. The bundled git's git-remote-https therefore binds the AppImage's stale bundled libnghttp2.so.14 instead of the system one, and every HTTPS fetch dies with a symbol lookup error, blocking session creation. This is the third occurrence of the same root cause after #644 (CLOSED) and #1394 (CLOSED, "fixed in v1.0.13").

Affected version or release

• GitHub Copilot app: 1.1.10 (from app logs: app version=1.1.10)
• Bundled desktop runtime: copilot-desktop-gh-2.96.0
• Bundled git: github-copilot-git-2.53.0-4 (git version 2.53.0) — logs confirm using bundled git
• Copilot CLI: 1.0.80
• AppImage build date: bundled libs dated Aug 13; usr/bin/github dated Aug 15

Installation context

• OS: Ubuntu 24.04.2 LTS, kernel 6.6.87.2-microsoft-standard-WSL2 (WSL2)
• AppImage extracted to disk at /home/saurabh/apps/squashfs-root (GitHub Copilot.desktop, Exec=github %u)
• System libcurl: /lib/x86_64-linux-gnu/libcurl.so.4 -> libcurl.so.4.8.0 (plain, not the gnutls variant)
• System nghttp2: /lib/x86_64-linux-gnu/libnghttp2.so.14 -> libnghttp2.so.14.26.0

What happened?

Creating any session fails during workspace initialization:

git ["-C", "/home/saurabh/work/mssql-rs", "-c", "credential.helper=", "fetch", "--progress",
     "--no-write-fetch-head", "https://x-access-token:[REDACTED]@github.com/microsoft/mssql-rs.git",
     "+refs/heads/<branch>:refs/remotes/origin/<branch>"] failed:
/home/saurabh/.cache/github-copilot-git-2.53.0-4/libexec/git-core/git-remote-https: symbol lookup error:
  /lib/x86_64-linux-gnu/libcurl.so.4: undefined symbol:
  nghttp2_option_set_no_rfc9113_leading_and_trailing_ws_validation
fatal: remote helper 'https' aborted session

Root cause (verified on this machine). The AppImage runtime exports to every child process:

LD_LIBRARY_PATH=/home/saurabh/apps/squashfs-root/usr/lib/:/home/saurabh/apps/squashfs-root/usr/lib/i386-linux-gnu/:/home/saurabh/apps/squashfs-root/usr/lib/x86_64-linux-gnu/:/home/saurabh/apps/squashfs-root/usr/lib32/:/home/saurabh/apps/squashfs-root/usr/lib64/:/home/saurabh/apps/squashfs-root/lib/:...

The bundled git-remote-https links the system libcurl.so.4, whose NEEDED list includes libnghttp2.so.14. Because the AppImage's lib dir takes precedence on LD_LIBRARY_PATH, the loader binds the stale bundled nghttp2, which is missing a symbol the system libcurl requires — so the helper dies before any network I/O.

Verified facts:

Fact	Value
Bundled libnghttp2.so.14	/home/saurabh/apps/squashfs-root/usr/lib/libnghttp2.so.14, 174816 bytes, dated Aug 13
nghttp2_option_set_no_rfc9113_leading_and_trailing_ws_validation in bundled lib	ABSENT (nm -D --defined-only -> 0 matches)
System libnghttp2.so.14	-> libnghttp2.so.14.26.0 (170352 bytes, Ubuntu)
Same symbol in system lib	PRESENT (nm -D --defined-only -> 1 match)
System libcurl	libcurl.so.4 -> libcurl.so.4.8.0 (plain, not gnutls)
Bundled git	github-copilot-git-2.53.0-4 (git 2.53.0)

Proof it is purely lib shadowing — not git, not network, not auth. The app's own bundled git works perfectly once the variable is dropped:

$ env -u LD_LIBRARY_PATH /home/saurabh/.cache/github-copilot-git-2.53.0-4/bin/git \
    ls-remote --heads https://github.com/microsoft/mssql-rs.git dev/saurabh/rowwriter-sink-api
890ef66d7d6d8f14147a18b66383aa530c11427d	refs/heads/dev/saurabh/rowwriter-sink-api

$ env -u LD_LIBRARY_PATH /home/saurabh/.cache/github-copilot-git-2.53.0-4/bin/git \
    ls-remote https://github.com/git/git HEAD
745601a9a94110d74769ab605ccd4f61339758d2	HEAD

Steps to reproduce

1. Run the GitHub Copilot AppImage (app 1.1.10) on Ubuntu 24.04 where system libcurl.so.4 is 4.8.0 and system libnghttp2.so.14 is newer than the bundled copy.
2. Create a session for any GitHub repo (triggers a workspace git fetch over HTTPS).
3. Workspace init fails with the symbol lookup error above.

Minimal repro of the mechanism, isolating only the nghttp2 shadowing (bundled lib exposed via a scratch dir, so nothing on the host is modified):

BGIT=~/.cache/github-copilot-git-2.53.0-4/bin/git
HELPER=~/.cache/github-copilot-git-2.53.0-4/libexec/git-core/git-remote-https
TMPD=$(mktemp -d)
ln -s /home/saurabh/apps/squashfs-root/usr/lib/libnghttp2.so.14 "$TMPD/libnghttp2.so.14"

# With the bundled nghttp2 on LD_LIBRARY_PATH (what the app does today):
LD_LIBRARY_PATH="$TMPD" ldd "$HELPER" | grep -E 'nghttp2|libcurl'
#   libcurl.so.4     => /lib/x86_64-linux-gnu/libcurl.so.4
#   libnghttp2.so.14 => /tmp/tmp.XXXX/libnghttp2.so.14      <-- bundled, stale

LD_LIBRARY_PATH="$TMPD" "$BGIT" ls-remote https://github.com/git/git HEAD
#   git-remote-https: symbol lookup error: /lib/x86_64-linux-gnu/libcurl.so.4:
#     undefined symbol: nghttp2_option_set_no_rfc9113_leading_and_trailing_ws_validation
#   fatal: remote helper 'https' aborted session

# Clean env — same bundled git, same URL:
env -u LD_LIBRARY_PATH "$BGIT" ls-remote https://github.com/git/git HEAD
#   745601a9a94110d74769ab605ccd4f61339758d2	HEAD

Both branches above were executed on this machine and produced exactly the quoted output.

Expected behavior

Session creation should succeed regardless of the host's libcurl/nghttp2 versions. Git spawned by the app (bundled or system) should resolve system libraries, not the AppImage's bundled ones.

What is new vs. #1394 (please don't close this as a duplicate of the earlier fix)

• Linux: AppImage leaks bundled LD_LIBRARY_PATH to spawned git (bundled git + libnghttp2 variant), blocking session creation — regression of #644 #1394 reported libcurl-gnutls.so.4; this occurrence is plain libcurl.so.4. Any fix that special-cased the gnutls variant does not cover this path.
• Linux: AppImage leaks bundled LD_LIBRARY_PATH to spawned git (bundled git + libnghttp2 variant), blocking session creation — regression of #644 #1394 was on bundled git github-copilot-git-2.53.0-3; this is -4. So the bundled-git refresh did not carry the fix forward either.
• Linux: AppImage leaks bundled LD_LIBRARY_PATH to spawned git (bundled git + libnghttp2 variant), blocking session creation — regression of #644 #1394 was closed as fixed in v1.0.13; this reproduces on 1.1.10. Combined with Bug: AppImage injects bundled libs into system git's environment, breaking on modern distros #644, the same root cause has now shipped and regressed three times, which suggests each fix targeted a symptom (one specific library or one specific git binary) rather than the underlying behavior.

The ask — please fix the behavior, not the library

The durable fix is to stop leaking the AppImage's LD_LIBRARY_PATH into spawned child processes. Concretely:

1. Primary: when the app spawns git (bundled or system), scrub LD_LIBRARY_PATH from the child environment — or reduce it to only what the bundled git genuinely requires. The AppImage's own GUI/runtime libs are irrelevant to git. As proven above, the shipped bundled git is already self-sufficient against system libs, so scrubbing costs nothing.
2. Defensive secondary: keep bundled transitive deps such as libnghttp2 at or above the versions the linked libcurl requires, or drop them from the bundle entirely. Shipping a bundled dependency that is older than the system library it shadows is what turns the leak into a hard failure.
3. Regression guard: a smoke test that spawns the bundled git under the AppImage's real spawn environment and does an HTTPS ls-remote would have caught all three occurrences.

Additional context

• Prior occurrences: Bug: AppImage injects bundled libs into system git's environment, breaking on modern distros #644 (CLOSED — system git loading bundled libssl/libpcre2 on Arch), Linux: AppImage leaks bundled LD_LIBRARY_PATH to spawned git (bundled git + libnghttp2 variant), blocking session creation — regression of #644 #1394 (CLOSED, "fixed in v1.0.13" — bundled git loading bundled libnghttp2 against libcurl-gnutls.so.4 on Ubuntu 24.04). Possibly related: [Bugs] Worktree creation fails hard with embedded Git if the expected version of lib... #1197 (OPEN — worktree creation fails with embedded git when the expected libcurl isn't installed, on Arch).

• Local workaround applied here (renaming the bundled lib so the loader falls through to the newer system one, which has the same .14 soname and a superset of symbols):

  mv /home/saurabh/apps/squashfs-root/usr/lib/libnghttp2.so.14 \
     /home/saurabh/apps/squashfs-root/usr/lib/libnghttp2.so.14.disabled

  After that, ldd on the helper resolves libnghttp2.so.14 => /lib/x86_64-linux-gnu/libnghttp2.so.14 and session creation succeeds with LD_LIBRARY_PATH still set. That confirms the bundled lib is the only thing breaking it — nothing else in the leaked path is load-bearing for git.

• The failure also affects any tooling the app spawns: every gh and git invocation from inside an app session has to be prefixed with env -u LD_LIBRARY_PATH to work, including the commands used to research and file this report.

• Filed via gh rather than the in-app issue reporter: the app's own "create issue" path refused with "This project's GitHub account is no longer available. Please re-link the project to a GitHub account before creating an issue." because the active project is an Azure DevOps repo with no linked GitHub account. Minor, separate papercut, but it means non-GitHub-backed projects cannot file feedback from inside the app.

[saurabh500]

Additional evidence: the shipped AppImage itself ships nghttp2 1.43.0 (Feb 2021), and the stale lib is not recoverable user-side

To rule out any possibility that this was corruption in my locally extracted squashfs-root, I re-checked the read-only FUSE-mounted AppImage the running app was actually launched from, at /tmp/.mount_githubcCHGNE:

github on /tmp/.mount_githubcCHGNE type fuse.github (ro,nosuid,nodev,relatime,user_id=1000,group_id=1000)

Check	Result
/tmp/.mount_githubcCHGNE/usr/lib/libnghttp2.so.14	present, 174816 bytes, root-owned
nm -D --defined-only for nghttp2_option_set_no_rfc9113_leading_and_trailing_ws_validation	0 matches (absent)
md5 vs my extracted squashfs-root copy	identical — debe00b7d7afa9d09861036ce3570914
SONAME (readelf -d)	libnghttp2.so.14
Version string embedded in the binary	1.43.0
System lib for comparison	libnghttp2.so.14 -> libnghttp2.so.14.26.0 (Ubuntu 24.04), symbol present (1 match)

Commands used (all read-only, nothing on the host modified):

M=/tmp/.mount_githubcCHGNE/usr/lib/libnghttp2.so.14
ls -la "$M"
nm -D --defined-only "$M" | grep -c 'no_rfc9113_leading_and_trailing_ws_validation'   # -> 0
md5sum "$M" /home/saurabh/apps/squashfs-root/usr/lib/libnghttp2.so.14.disabled       # -> identical
readelf -d "$M" | grep -i soname                                                      # -> libnghttp2.so.14
strings "$M" | grep -E '^1\.[0-9]+\.[0-9]+$' | sort -u                                # -> 1.43.0

Three conclusions

1. The bundled nghttp2 predates the required symbol by ~19 months, and the bundle itself is ~4.5 years old.

I verified the introducing release directly against upstream headers rather than relying on recollection:

curl -fsSL https://raw.githubusercontent.com/nghttp2/nghttp2/v1.49.0/lib/includes/nghttp2/nghttp2.h \
  | grep -c nghttp2_option_set_no_rfc9113_leading_and_trailing_ws_validation   # -> 0
curl -fsSL https://raw.githubusercontent.com/nghttp2/nghttp2/v1.50.0/lib/includes/nghttp2/nghttp2.h \
  | grep -c nghttp2_option_set_no_rfc9113_leading_and_trailing_ws_validation   # -> 1

• Absent in v1.49.0, present in v1.50.0 -> introduced in nghttp2 1.50.0.
• Upstream release dates (GitHub releases API): v1.50.0 published 2022-09-21; v1.43.0 published 2021-02-02.

So the bundle ships a Feb 2021 nghttp2, and the symbol arrived in Sep 2022. Any system libcurl built against nghttp2 >= 1.50.0 — which is every current mainstream distro — will fail to resolve against the bundled copy whenever the AppImage's lib dir wins on LD_LIBRARY_PATH.

2. The md5 match proves this is a build/packaging defect, not local corruption. The stale library is byte-identical between the read-only mounted AppImage and my extracted copy. It is what ships. Anyone running this AppImage on a distro with a modern libcurl is exposed — nothing about my machine made this file bad.

3. There is no durable user-side fix. Renaming the lib under an extracted squashfs-root repairs only the currently-running instance. The AppImage mount is read-only, and a normal launch remounts the same byte-identical stale file, so the failure returns on the next start. Users cannot work around this persistently; it has to be fixed in the build.

Sharpened ask

This does not change the primary fix, but it does add a second one that is independently necessary:

1. Primary (durable): scrub LD_LIBRARY_PATH from the environment of spawned children, git especially. This is the fix that survives future library drift and would have prevented Bug: AppImage injects bundled libs into system git's environment, breaking on modern distros #644, Linux: AppImage leaks bundled LD_LIBRARY_PATH to spawned git (bundled git + libnghttp2 variant), blocking session creation — regression of #644 #1394, and this issue alike.
2. Now also required: bring the bundled libnghttp2 to a version at or above what the bundled/linked libcurl requires (i.e. >= 1.50.0, realistically current stable), or drop it from the bundle so the loader uses the system copy. Shipping a 2021 transitive dependency that shadows the system library is a latent break on every modern distro, independent of the LD_LIBRARY_PATH question.

It is also worth auditing the rest of the bundled usr/lib for other libraries frozen at similarly old versions — libssl.so.3, libcrypto.so.3, and libk5crypto.so.3 sit in the same directory, and #644 was triggered by exactly that class of mismatch on libssl/libpcre2.