Skip to content

fix: TEM-89 ComfyUI-Manager pip failures (probe timeout, venv pip, torch index) - #163

Open
mariiachekmasova-runpod wants to merge 35 commits into
mainfrom
feat/TEM-89-no-pip-uv
Open

mariiachekmasova-runpod wants to merge 35 commits into
mainfrom
feat/TEM-89-no-pip-uv

Conversation

@mariiachekmasova-runpod

@mariiachekmasova-runpod mariiachekmasova-runpod commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

The ticket's main problem: ComfyUI-Manager refuses to work with Neither pip nor uv are available, so users on a running pod cannot install a single custom node.
Its second, from the same ticket's report #8: even with a healthy pip, installing
a node whose dependencies mention torch fails outright.

Two files: official-templates/comfyui/Dockerfile and scripts/start.sh.

1. The pip probe timeout

Manager's get_pip_cmd() runs python -m pip --version with a 5 s timeout and
catches every exception, so a timeout is indistinguishable from a missing pip.
The result is cached for the process lifetime: lose the race once and the Manager
is dead until the pod restarts. pip was there all along — it was just slow.

It was slow because it lived on the network volume. The venv is created under
/workspace, and python -m venv bootstraps its own copy of pip into it —
several hundred files read over the network on every invocation.

Two changes, in that order of importance:

Remove the cause. The venv is now created --without-pip, so pip resolves
from the image: local disk, bytecode compiled at build time.
--system-site-packages keeps it importable, and installs still land in the venv
because pip takes its target from sys.prefix. create_pip_shim writes bin/pip
only when absent, so a bare pip targets the venv while existing volumes stay
untouched. The ensurepip calls were redundant — venv bootstraps pip by
default — and are gone.

Raise the ceiling. The baked Manager's timeout goes from 5 s to 30 s, patched
before its git commit so its own tree stays clean; the grep -c assertion fails
the build if the upstream pattern moves. This is what reaches existing volumes,
which keep their own on-volume pip: the baked Manager is synced from the image on
every bundle change, --without-pip is not.

2. Report # 8: node installs could not resolve torch

/opt/comfyui-runtime-constraints.txt pins torch to the baked +cuXXX build and
start.sh exports it as PIP_CONSTRAINT, but the PyTorch index was only set
inside a single builder-stage RUN — it never existed in the finished image. pip
in the pod knew only PyPI, which serves no +cuXXX version, so the constraint was
unsatisfiable and any install that had to resolve torch died with
ResolutionImpossible.

The index is now set in the runtime stage, and added to the env-propagation
pattern in start.sh. That second half matters: PIP_CONSTRAINT was already
propagated to SSH sessions and the index was not, so a terminal had the half that
breaks installs without the half that fixes them.

3. Jupyter password failsafe

start_jupyter now generates a token when the env variable is empty and logs it, the way setup_ssh
already handles the root password.

4. Upgraded ComfyUI-RunpodDirect

New version fixes the silent-failure behaviour behind TEM-120. Pin moves from
809065c9d2f3 to 9e32b1a09577 (v1.0.12, "Fix HF auth").

Two defects found while verifying the above

Interactive sessions had no working Python. export_env_vars copies a fixed
set of variables into the login files, and it runs before the venv exists — so an
SSH session got a PATH without it. python did not resolve at all (the image
only ships python3.12) and pip resolved to /usr/local/bin/pip, installing
against the base interpreter into the container filesystem, which is discarded on
pod restart. start.sh now appends the venv activation to /etc/rp_environment
once the venv is ready; that file is already sourced from both bashrc files. It is
also cleared on boot now, like the three login files next to it — previously it
only ever grew.

A second CUDA-12.4 migration looped the pod. The migration moved the old venv
to a fixed .venv.bak. When that directory already existed — the script creates
it and only suggests deleting it — mv moved the venv inside it, failed, and
set -e killed the boot, so the pod restarted every ~15 s with no explanation.
The backup name is now timestamped, and a failed move warns and continues with a
fresh venv instead of aborting.

Verification

Manual, on rc images of this branch: cuda12.8 and cuda13.0, RTX 4090 / L4.

The timeout. python -m pip --version from the venv, same GPU model:

Volume state pip resolved from warm-up Manager prestartup
created by the published image 24.0 the volume 3.41 s / 3.45 s 13.0 s
created by this branch 26.1.2 the image 0.171–0.334 s 2.7–3.1 s

The old-volume row is the population --without-pip does not reach: 3.4 s against
a 5 s budget on a healthy pod, reproducible across two boots. No margin at all —
that is what the raised ceiling is for. Note the published image also ran a pip
two years older than the pinned one: ensurepip seeds the venv with Python's
bundled pip, so pip==26.1.2 never applied at runtime.

Report # 8, end to end through the Manager UI. Installing ComfyUI-Impact-Pack
on the published image fails at Installing build dependencies for sam2 with
ResolutionImpossible (torch>=2.5.1 against constraint torch==2.10.0+cu130).
On this branch it builds and installs, and the node loads after a pod restart with
its dependencies intact on the volume. Separately, pip index versions torch
lists 2.10.0+cuXXX with the index and only 2.10.0 without it.

Everything else:

  • pip install --ignore-installed six lands in .venv-cu128, and sysconfig's
    purelib points there. Bare pip works via the shim both by full path and
    through PATH after activation.
  • In an SSH session: python and pip both resolve into the venv, VIRTUAL_ENV
    is set, and pip install piexif lands in the venv.
  • All three pip-related variables reach an SSH session.
  • CUDA-12.4 migration: user-node dependencies install into the venv, the backup is
    timestamped, and a pre-existing backup no longer collides.
  • pip freeze --local still lists only venv-local packages.

Unrelated: the smoke test attached a Docker Hub credential nobody asked for

With no REGISTRY_AUTH_NAME set, the harness took whatever credential
GET /v2/registries happened to list first and attached it to every pod. After
the CI account switch that first entry held a stale Docker Hub login, and the
whole matrix died with unauthorized: incorrect username or password.

The implicit pick is gone. The new registry-auth-name action input names the
credential; empty (the default) means anonymous pulls, and a name the account
does not have exits before the first pod is created, listing the names it does
have. Existing callers need no change — none of the eight passes the input.

tests/unit/test_registry_auth.py covers the resolution rules, including that an
unresolved name attaches nothing and stops main before any pod is planned. The
Harness Unit Tests workflow runs them: 63 tests green.

@mariiachekmasova-runpod mariiachekmasova-runpod changed the title feat: TEM-89 fix: TEM-89 ComfyUI-Manager pip failures (probe timeout, venv pip, torch index) Sep 7, 2026
@MadiatorLabs

Copy link
Copy Markdown

Rechecked the latest update. The RunpodDirect SHA bump changes the bundle manifest, so existing workspaces now receive the Manager timeout patch. Both CUDA smoke tests passed too. Looks good from my side once #161 is merged.

Base automatically changed from feat/TEM-42-ComfyUI to main September 10, 2026 08:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants