[siddhartha-quad]

Fix stuck jobs when machines are out of stock

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses issues with stuck jobs when GCE capacity is exhausted. It introduces a capacity circuit breaker for Slurm to manage nodesets during capacity errors and improves HTCondor's handling of VM shutdowns and job re-queuing. Additionally, it refines the HTCondor autoscaler logic to ensure more accurate scaling decisions.

Highlights

• Capacity Circuit Breaker: Introduced a new capacity circuit breaker mechanism for Slurm clusters to automatically drain and probe nodesets experiencing GCE capacity errors.
• HTCondor Shutdown Handling: Added shutdown scripts and job transformation rules to gracefully handle VM shutdowns and re-queue jobs that fail due to initialization errors on Windows.
• Autoscaler Improvements: Updated the HTCondor autoscaler to better account for all idle jobs when making scaling decisions and added logic to trigger negotiation cycles.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a capacity circuit breaker mechanism for the Slurm GCP v6 controller to temporarily remove GCE nodesets from scheduling after capacity errors, along with HTCondor improvements including shutdown scripts, automatic job re-queuing on Windows shutdown, and autoscaler refinements. The review feedback highlights several critical robustness improvements: preventing an AttributeError in resume.py by checking for _get_reason existence, avoiding a potential KeyError in capacity_circuit.py when accessing nodesets, safely handling cases where the node state is returned as a string rather than a list, and explicitly specifying UTF-8 encoding when reading the state file.

[gemini-code-assist]

Calling exc._get_reason() directly without checking if the attribute exists will raise an AttributeError if exc is a standard Python exception or any exception that doesn't implement _get_reason(). The original code safely handled this by checking hasattr(exc, "_get_reason") before calling it.

Suggested change

	failed_nodes = grouped_nodes[ident].nodes
	trip = trip_capacity_circuit(
	failed_nodes,
	capacity_circuit.exception_error_codes(exc),
	lkp,
	)
	down_nodes_notify_jobs(
	failed_nodes,
	f"GCP Error: {exc._get_reason()}", # type: ignore
	resume_data,
	trip,
	)
	failed_nodes = grouped_nodes[ident].nodes
	reason = exc._get_reason() if hasattr(exc, "_get_reason") else str(exc)
	trip = trip_capacity_circuit(
	failed_nodes,
	capacity_circuit.exception_error_codes(exc),
	lkp,
	)
	down_nodes_notify_jobs(
	failed_nodes,
	f"GCP Error: {reason}",
	resume_data,
	trip,
	)

[gemini-code-assist]

nodeset = lkp.cfg.nodeset[nodeset_name] will raise a KeyError if nodeset_name is not found in lkp.cfg.nodeset (for example, if it is a TPU nodeset or dynamic nodeset). Using .get(nodeset_name) and returning None if it is not found is much safer and prevents crashes.

Suggested change

	nodeset = lkp.cfg.nodeset[nodeset_name]
	powered_down = _powered_down_nodes(nodeset, lkp)
	nodeset = lkp.cfg.nodeset.get(nodeset_name)
	if nodeset is None:
	return None
	powered_down = _powered_down_nodes(nodeset, lkp)

[gemini-code-assist]

current_states = set(current.get("state") or []) assumes current.get("state") is always a list of strings. However, depending on the Slurm version or JSON schema, state can sometimes be returned as a single string (e.g., "DOWN"). If it is a string, set("DOWN") will split it into individual characters {'D', 'O', 'W', 'N'}, causing the intersection check with restricted_states to fail. Checking if it is a string and wrapping it in a set/list prevents this.

            current_reason = current.get("reason") or ""
            raw_state = current.get("state") or []
            current_states = {raw_state} if isinstance(raw_state, str) else set(raw_state)
            if current_states.intersection(
                restricted_states
            ) and not current_reason.startswith(reason_prefix):

[gemini-code-assist]

STATE_FILE.read_text() is called without specifying an encoding. This can lead to platform-dependent encoding issues (e.g., defaulting to CP1252 on some Windows environments or other locales). It is safer to explicitly specify encoding="utf-8", especially since _write_state_unlocked explicitly writes using encoding="utf-8".

Suggested change

	state = json.loads(STATE_FILE.read_text())
	state = json.loads(STATE_FILE.read_text(encoding="utf-8"))