[siddhartha-quad]

When autoscaler.py scales down the Managed Instance Group (MIG), it directly calls the GCP deleteInstances API. However, there is a time gap between that API call and when the Windows VM actually stops accepting new processes. If a job is dispatched during this window, Windows returns 0xC0000142 (STATUS_DLL_INIT_FAILED), which HTCondor treats as a completed job instead of a node failure.

Testing Environment & Verification

• OS Image: Windows Server 2022 Datacenter (windows-cloud:windows-2022) for Execute Points; Rocky Linux 8 for Central Manager and Access Point.
• HTCondor Version: HTCondor 23.0 LTS (23.0.x, Windows x64 MSI).

Test Scenarios:

1. Shutdown Script: Verified that triggering instance deletion in the MIG initiates the Windows shutdown script (Stop-Service condor -Force), removing the Execute Point from the collector pool immediately without accepting further jobs.
2. Job Transform & Retry Logic:
   • Submitted test jobs targeting Windows nodes simulating exit codes -1073741502 (0xC0000142) and -1073741205 (0xC000026B).
   • Verified that the RETRY_WINDOWS_SHUTDOWN job transform evaluated OnExitRemove = False, returning the jobs to Idle for re-execution.
   • Verified that retries are capped at 5 attempts (NumJobCompletions < 5) to prevent infinite retry loops on non-transient failures.
3. Autoscaler Verification: Verified that autoscaler.py successfully calculates scale-down targets considering all idle jobs and strips slot prefixes (slotX@) from node names.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses race conditions occurring during Managed Instance Group (MIG) downscaling. By implementing graceful shutdown procedures and improving the autoscaler's awareness of pending jobs, the changes prevent jobs from being incorrectly marked as completed when they fail due to VM termination during the shutdown window.

Highlights

• Graceful Shutdown: Added shutdown scripts for both Windows and Linux to ensure HTCondor services are stopped cleanly before VM termination.
• Job Re-queueing: Updated HTCondor configuration to automatically re-queue jobs that fail with specific Windows DLL initialization errors during shutdown.
• Autoscaler Logic: Enhanced the autoscaler to account for all idle jobs during scale-down decisions and added a verification step to prevent premature node deletion if new jobs arrive.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces graceful shutdown scripts for HTCondor execute points on both Windows and Linux, configures automatic job re-queuing for Windows initialization failures during shutdown, and refines the autoscaler logic to prevent premature scale-down when new idle jobs are waiting. The review feedback highlights a potential race condition where active nodes could be deleted during scale-down, and recommends adding a delay after condor_off to allow graceful eviction. Additionally, the feedback suggests defensive programming in the autoscaler script to handle missing RequestCpus attributes and prevent negative target sizes.

[saara-tyagi27]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces improvements to HTCondor job scheduling and autoscaling, including graceful shutdown scripts for execute points, automatic re-queuing of jobs that fail during Windows VM shutdown, and a more robust autoscaling logic that prevents premature scale-down when new jobs are queued. Feedback on these changes highlights two critical issues: first, the job transform expression in condor_config.tftpl incorrectly uses a configuration macro expansion that would overwrite custom user OnExitRemove expressions; second, a typo in autoscaler.py (AdType instead of AdTypes) will raise an AttributeError and crash the autoscaler during scale-down.

[sudheer-quad]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces graceful shutdown scripts for HTCondor execute points on both Windows and Linux, and adds a job transform to automatically re-queue Windows jobs that fail during VM shutdown. It also refines the Python autoscaler's scale-down logic to prevent deleting nodes when new idle jobs arrive or when nodes become claimed during GCP API calls. The review feedback highlights two critical issues in the autoscaler where calling int() on the RequestCpus attribute could crash the script if the value is undefined or a ClassAd expression, suggesting safer type-checking alternatives.

[siddhartha-quad]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces clean shutdown scripts for HTCondor execute points on both Windows and Linux, and adds a job transform to automatically re-queue jobs that fail to initialize on Windows during VM shutdown. It also refactors the autoscaler script to prevent premature scale-down by tracking all idle jobs (including newly submitted ones) and re-verifying node states before executing a scale-down. The review feedback highlights two instances in the autoscaler where casting CPU requests to integers truncates fractional CPU values to zero, which could lead to under-provisioning.

[siddhartha-quad]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces graceful shutdown scripts for HTCondor on both Windows and Linux, adds a job transform to automatically re-queue jobs failing during Windows VM shutdown, and refines the autoscaler logic to prevent premature scale-down and handle slot prefixes. The review feedback highlights a critical bug in autoscaler.py where slot prefixes are not stripped when constructing GCP instance URLs, potentially causing double-counting and API errors. Additionally, suggestions were made to improve robustness by handling unsigned representations of Windows exit codes, using safe-get logic to prevent crashes when calculating idle CPUs, and correcting minor Terraform formatting.

[siddhartha-quad]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces shutdown scripts for HTCondor execute points on both Windows and Linux, and configures the access point to automatically re-queue jobs that fail on Windows during VM shutdown. Additionally, it enhances the autoscaler by improving scale-down safety (re-querying the collector to avoid deleting newly claimed nodes), refining idle job counting to prevent premature scale-down, and handling machine names containing '@'. The review feedback recommends adopting defensive programming in the autoscaler by using .get() with safe defaults when accessing dictionary attributes like NODENAME_ATTRIBUTE and Machine to prevent potential KeyError or AttributeError exceptions.

[siddhartha-quad]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces several improvements to the HTCondor autoscaler and execute point modules. It adds shutdown scripts for both Windows and Linux to cleanly stop HTCondor services, configures automatic job re-queuing for Windows DLL initialization failures during VM shutdown, and enhances the Python autoscaler to prevent premature scale-down when new idle jobs are waiting. Additionally, the autoscaler now re-queries the collector to avoid deleting nodes that became claimed during GCP API calls. Feedback on the autoscaler script suggests using sys.exit() instead of the built-in exit() and adding defensive type checks to prevent potential AttributeErrors when parsing node names.

[siddhartha-quad]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces shutdown scripts for HTCondor execute points (both Windows and Linux) to gracefully stop services, and adds a job transform configuration to automatically re-queue jobs that fail during Windows VM shutdown. Additionally, the autoscaler script is updated to prevent premature scale-down when new idle jobs are waiting, re-query the collector for active nodes before deleting instances, and handle node name parsing more safely. The review feedback focuses on improving the robustness of the Python autoscaler by adding explicit type checks and safe attribute retrieval to prevent potential AttributeError or KeyError exceptions.

[siddhartha-quad]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces graceful shutdown scripts for HTCondor execute points on both Windows and Linux, and configures the access point to automatically re-queue Windows jobs that fail during VM shutdown. Additionally, the autoscaler is updated to prevent premature scale-down when new jobs are waiting, safely handle missing or malformed ClassAd attributes, and re-query HTCondor status during scale-down to avoid deleting newly claimed nodes. Feedback on the changes suggests removing the redundant re-query block in the autoscaler to reduce network latency and load on the HTCondor Collector and Scheduler.

[siddhartha-quad]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces graceful shutdown scripts for HTCondor execute points on both Windows and Linux, configures automatic re-queuing of Windows jobs failing during VM shutdown, and optimizes the autoscaler script to prevent premature scale-down and improve node name parsing. Feedback on the changes suggests explicitly checking for /usr/sbin/condor_off in the Linux shutdown script to ensure the command is reliably found even if the PATH environment variable is restricted during shutdown.

[arpit974]

Three fixes needed (details inline):

1. Cap the retries in the new retry rule (condor_config.tftpl L79).
2. Remove condor_off.exe and the sleep from the Windows script (main.tf L49–55).
3. Fix the terraform fmt failure (main.tf L56).
   Please also:

• Say in the PR description how you tested on Windows (which image and HTCondor version).
• Add README notes for the new shutdown scripts and the new retry rule, which applies to all jobs, Linux included.

[github-actions]

this PR has been inactive for 2 days and has changes requested. @siddhartha-quad, please address the requested changes or close the PR if it's no longer needed.

[siddhartha-quad]

/gemini review

[gemini-code-assist]

Code Review

This pull request introduces Windows support and graceful shutdown handling for HTCondor execute and access points, including VM shutdown scripts, automatic job retries for Windows shutdown exit codes, and improved autoscaling logic. Feedback on these changes highlights several issues: an undefined local variable (local.is_windows) in main.tf that will cause Terraform errors, unnecessary duplication of installation logic in download-condor-config.ps1.tftpl, risky backtick line continuations in PowerShell, and a hardcoded 25.0 version in the HTCondor installer download URL that prevents dynamic version selection.

[github-actions]

this PR has been inactive for 2 days and has changes requested. @siddhartha-quad, please address the requested changes or close the PR if it's no longer needed.