fix: TEM-89 ComfyUI-Manager pip failures (probe timeout, venv pip, torch index) - #163
Open
mariiachekmasova-runpod wants to merge 35 commits into
Open
mariiachekmasova-runpod wants to merge 35 commits into
mariiachekmasova-runpod wants to merge 35 commits into
Conversation
|
Rechecked the latest update. The RunpodDirect SHA bump changes the bundle manifest, so existing workspaces now receive the Manager timeout patch. Both CUDA smoke tests passed too. Looks good from my side once #161 is merged. |
Madiator2011Work
approved these changes
Sep 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The ticket's main problem: ComfyUI-Manager refuses to work with
Neither pip nor uv are available, so users on a running pod cannot install a single custom node.Its second, from the same ticket's report #8: even with a healthy pip, installing
a node whose dependencies mention torch fails outright.
Two files:
official-templates/comfyui/Dockerfileandscripts/start.sh.1. The pip probe timeout
Manager's
get_pip_cmd()runspython -m pip --versionwith a 5 s timeout andcatches every exception, so a timeout is indistinguishable from a missing pip.
The result is cached for the process lifetime: lose the race once and the Manager
is dead until the pod restarts. pip was there all along — it was just slow.
It was slow because it lived on the network volume. The venv is created under
/workspace, andpython -m venvbootstraps its own copy of pip into it —several hundred files read over the network on every invocation.
Two changes, in that order of importance:
Remove the cause. The venv is now created
--without-pip, so pip resolvesfrom the image: local disk, bytecode compiled at build time.
--system-site-packageskeeps it importable, and installs still land in the venvbecause pip takes its target from
sys.prefix.create_pip_shimwritesbin/piponly when absent, so a bare
piptargets the venv while existing volumes stayuntouched. The
ensurepipcalls were redundant —venvbootstraps pip bydefault — and are gone.
Raise the ceiling. The baked Manager's timeout goes from 5 s to 30 s, patched
before its git commit so its own tree stays clean; the
grep -cassertion failsthe build if the upstream pattern moves. This is what reaches existing volumes,
which keep their own on-volume pip: the baked Manager is synced from the image on
every bundle change,
--without-pipis not.2. Report # 8: node installs could not resolve torch
/opt/comfyui-runtime-constraints.txtpins torch to the baked+cuXXXbuild andstart.shexports it asPIP_CONSTRAINT, but the PyTorch index was only setinside a single builder-stage
RUN— it never existed in the finished image. pipin the pod knew only PyPI, which serves no
+cuXXXversion, so the constraint wasunsatisfiable and any install that had to resolve torch died with
ResolutionImpossible.The index is now set in the runtime stage, and added to the env-propagation
pattern in
start.sh. That second half matters:PIP_CONSTRAINTwas alreadypropagated to SSH sessions and the index was not, so a terminal had the half that
breaks installs without the half that fixes them.
3. Jupyter password failsafe
start_jupyternow generates a token when the env variable is empty and logs it, the waysetup_sshalready handles the root password.
4. Upgraded ComfyUI-RunpodDirect
New version fixes the silent-failure behaviour behind TEM-120. Pin moves from
809065c9d2f3to9e32b1a09577(v1.0.12, "Fix HF auth").Two defects found while verifying the above
Interactive sessions had no working Python.
export_env_varscopies a fixedset of variables into the login files, and it runs before the venv exists — so an
SSH session got a
PATHwithout it.pythondid not resolve at all (the imageonly ships
python3.12) andpipresolved to/usr/local/bin/pip, installingagainst the base interpreter into the container filesystem, which is discarded on
pod restart.
start.shnow appends the venv activation to/etc/rp_environmentonce the venv is ready; that file is already sourced from both bashrc files. It is
also cleared on boot now, like the three login files next to it — previously it
only ever grew.
A second CUDA-12.4 migration looped the pod. The migration moved the old venv
to a fixed
.venv.bak. When that directory already existed — the script createsit and only suggests deleting it —
mvmoved the venv inside it, failed, andset -ekilled the boot, so the pod restarted every ~15 s with no explanation.The backup name is now timestamped, and a failed move warns and continues with a
fresh venv instead of aborting.
Verification
Manual, on rc images of this branch: cuda12.8 and cuda13.0, RTX 4090 / L4.
The timeout.
python -m pip --versionfrom the venv, same GPU model:The old-volume row is the population
--without-pipdoes not reach: 3.4 s againsta 5 s budget on a healthy pod, reproducible across two boots. No margin at all —
that is what the raised ceiling is for. Note the published image also ran a pip
two years older than the pinned one:
ensurepipseeds the venv with Python'sbundled pip, so
pip==26.1.2never applied at runtime.Report # 8, end to end through the Manager UI. Installing ComfyUI-Impact-Pack
on the published image fails at
Installing build dependenciesforsam2withResolutionImpossible(torch>=2.5.1against constrainttorch==2.10.0+cu130).On this branch it builds and installs, and the node loads after a pod restart with
its dependencies intact on the volume. Separately,
pip index versions torchlists
2.10.0+cuXXXwith the index and only2.10.0without it.Everything else:
pip install --ignore-installed sixlands in.venv-cu128, andsysconfig'spurelibpoints there. Barepipworks via the shim both by full path andthrough
PATHafter activation.pythonandpipboth resolve into the venv,VIRTUAL_ENVis set, and
pip install piexiflands in the venv.timestamped, and a pre-existing backup no longer collides.
pip freeze --localstill lists only venv-local packages.Unrelated: the smoke test attached a Docker Hub credential nobody asked for
With no
REGISTRY_AUTH_NAMEset, the harness took whatever credentialGET /v2/registrieshappened to list first and attached it to every pod. Afterthe CI account switch that first entry held a stale Docker Hub login, and the
whole matrix died with
unauthorized: incorrect username or password.The implicit pick is gone. The new
registry-auth-nameaction input names thecredential; empty (the default) means anonymous pulls, and a name the account
does not have exits before the first pod is created, listing the names it does
have. Existing callers need no change — none of the eight passes the input.
tests/unit/test_registry_auth.pycovers the resolution rules, including that anunresolved name attaches nothing and stops
mainbefore any pod is planned. TheHarness Unit Testsworkflow runs them: 63 tests green.