Repository navigation
fix(job): set explicit CPU and memory requests on pathways-proxy and pathways-rm - #6527
Conversation
…pathways-rm In pkg/orchestrator/gke/templates/pathways_jobset.tmpl, pathways-proxy and pathways-rm previously specified resources.limits (16 CPU / 100Gi and 8 CPU / 32Gi) without a resources.requests block. Because Kubernetes defaults requests = limits when requests are omitted, every Pathways head pod reserved 24 CPU and 132Gi for the two sidecars alone (26 CPU and 140Gi total with workload-container), blocking scheduling on smaller CPU head node pools and over-reserving Kueue CPU and memory quota. Set explicit requests on pathways-proxy (cpu: "8", memory: "32Gi") and pathways-rm (cpu: "2", memory: "8Gi") while keeping their existing limits, reducing head pod requests to 12 CPU / 48Gi (10 CPU / 40Gi in headless mode) while preserving sidecar burst capacity.
Neelabh94
added
the
release-bugfix
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses an issue where missing explicit resource requests for sidecar containers resulted in inefficient cluster resource allocation. By defining specific request values, the change reduces the overall footprint of head pods, improving scheduling flexibility and quota management while preserving necessary burst capacity for the sidecars. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates the GKE pathways jobset template to specify resource requests (CPU and memory) alongside the existing resource limits for the pathways containers. Additionally, the corresponding unit tests in manifest_generator_test.go have been updated to reflect and verify these changes. I have no feedback to provide as the changes are correct and properly tested.
872b5a0
into
GoogleCloudPlatform:develop
5 participants
Summary
In
pkg/orchestrator/gke/templates/pathways_jobset.tmpl,pathways-proxyandpathways-rmpreviously specifiedresources.limits(16CPU /100Giand8CPU /32Gi) without an explicitresources.requestsblock. Because Kubernetes defaultsrequests = limitswhenrequestsare omitted, every Pathways head pod reserved 24 CPU and 132Gi for the two sidecars alone (26 CPU and 140Gi total withworkload-container), blocking scheduling on smaller CPU head node pools and over-reserving Kueue CPU and memory quota.This change adds explicit
requeststopathways-proxy(cpu: "8",memory: "32Gi") andpathways-rm(cpu: "2",memory: "8Gi") while keeping their existinglimits(16CPU /100Giand8CPU /32Gi), reducing head pod requests to 12 CPU / 48Gi (10 CPU / 40Gi in headless mode) while preserving sidecar burst capacity.Testing
TestGeneratePathwaysManifest_HeadContainerResourcesinpkg/orchestrator/gke/manifest_generator_test.goto verify bothrequestsandlimitsonpathways-proxyandpathways-rmacross default and headless modes (KUBECONFIG=/dev/null go test -race -count=1 ./pkg/orchestrator/gke/...).