Repository navigation
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## 4.22 #14138 +/- ##
============================================
+ Coverage 17.86% 18.84% +0.98%
- Complexity 16037 17304 +1267
============================================
Files 5928 5939 +11
Lines 534479 562677 +28198
Branches 65410 76557 +11147
============================================
+ Hits 95468 106043 +10575
- Misses 428173 444788 +16615
- Partials 10838 11846 +1008
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
|
@blueorangutan package |
|
@kiranchavala a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19246 |
weizhouapache
changed the title
kiranchavala
left a comment
kiranchavala
left a comment
There was a problem hiding this comment.
Still hitting the following issues
Issue 1
-
Deploy a cks with etcd node with size 1
-
scale the cks cluster size 2
``
(localcloud) 🐱 > scale kubernetescluster id=fc2459db-4e65-48d4-9c91-34c479346417 size=2
cks cluster scaled successfully
3. Downscale the cks cluster back to size 1 > Exception> cks cluster goes into alert state > vm is expunged
(localcloud) 🐱 > scale kubernetescluster id=fc2459db-4e65-48d4-9c91-34c479346417 size=1
{
"account": "admin",
"accountid": "994826d5-d36a-11f1-bb47-1e003c000122",
"cmd": "org.apache.cloudstack.api.command.user.kubernetes.cluster.ScaleKubernetesClusterCmd",
"completed": "2026-10-30T05:59:16+0000",
"created": "2026-10-30T05:59:00+0000",
"domainid": "3d20f963-d36a-11f1-bb47-1e003c000122",
"domainpath": "ROOT",
"jobid": "ce7a745b-a5b6-4a35-ba18-2d45baf0109f",
"jobprocstatus": 0,
"jobresult": {
"errorcode": 530,
"errortext": "Scaling failed for Kubernetes cluster KubernetesCluster {"id":2,"name":"cks2","uuid":"fc2459db-4e65-48d4-9c91-34c479346417"}, unable to update network rules"
},
"jobresultcode": 530,
"jobresulttype": "object",
"jobstatus": 2,
"userid": "9948cdb5-d36a-11f1-bb47-1e003c000122"
}
🙈 Error: async API failed for job ce7a745b-a5b6-4a35-ba18-2d45baf0109f
2026-10-30 05:59:16,594 ERROR [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-36:[ctx-4702e3ef, job-62, ctx-35c81aea]) (logid:ce7a745b) Scaling failed for Kubernetes cluster KubernetesCluster {"id":2,"name":"cks2","uuid":"fc2459db-4e65-48d4-9c91-34c479346417"}, unable to update network rules com.cloud.exception.ManagementServerException: Failed to provision firewall rules for SSH access for the Kubernetes cluster : cks2
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterActionWorker.addFirewallRulesForNodes(KubernetesClusterActionWorker.java:1037)
at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterResourceModifierActionWorker.createFirewallRules(KubernetesClusterResourceModifierActionWorker.java:668)
at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.cloud.exception.NetworkRuleConflictException: The range specified, 2222-2223, conflicts with rule FirewallRule {"id":17,"networkId":205,"purpose":"Firewall","state":"Active","uuid":"3198c529-f76d-45aa-9d83-e74f90dffd06"} which has 2222-2224
at com.cloud.network.firewall.FirewallManagerImpl.detectRulesConflict(FirewallManagerImpl.java:486)
at com.cloud.network.firewall.FirewallManagerImpl.lambda$createFirewallRule$0(FirewallManagerImpl.java:265)
at com.cloud.utils.db.Transaction.execute(Transaction.java:38)
at com.cloud.network.firewall.FirewallManagerImpl.createFirewallRule(FirewallManagerImpl.java:255)
at com.cloud.network.firewall.FirewallManagerImpl.createIngressFirewallRule(FirewallManagerImpl.java:207)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:109)
2026-10-30 05:59:16,599 INFO [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-36:[ctx-4702e3ef, job-62, ctx-35c81aea]) (logid:ce7a745b) Revoking launch permission for systemVM Template
2026-10-30 05:59:16,609 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-36:[ctx-4702e3ef, job-62]) (logid:ce7a745b) Complete async job-62, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Scaling failed for Kubernetes cluster KubernetesCluster {"id":2,"name":"cks2","uuid":"fc2459db-4e65-48d4-9c91-34c479346417"}, unable to update network rules"}
Issue 2
Scale to common offering
create kubernetescluster name=cks-npe zoneid= kubernetesversionid= networkid=
controlnodes=1 size=1 hypervisor=KVM
nodeofferings[0].node=control nodeofferings[0].offering=
nodeofferings[1].node=worker nodeofferings[1].offering=
mysql> select control_node_service_offering_id,worker_node_service_offering_id,etcd_node_service_offering_id from kubernetes_cluster where id=3;
+----------------------------------+---------------------------------+-------------------------------+
| control_node_service_offering_id | worker_node_service_offering_id | etcd_node_service_offering_id |
+----------------------------------+---------------------------------+-------------------------------+
| 13 | 13 | NULL |
+----------------------------------+---------------------------------+-------------------------------+
1 row in set (0.00 sec)
Stop the cks cluster
scale kubernetescluster id=<> serviceofferingid=<>
mysql> select control_node_service_offering_id,worker_node_service_offering_id,etcd_node_service_offering_id from kubernetes_cluster where id=3;
+----------------------------------+---------------------------------+-------------------------------+
| control_node_service_offering_id | worker_node_service_offering_id | etcd_node_service_offering_id |
+----------------------------------+---------------------------------+-------------------------------+
| 14 | 14 | 14 |
+----------------------------------+---------------------------------+-------------------------------+
1 row in set (0.00 sec)
Cloudstack changed the etcd node service offering
Issue 3
still hitting npe when try to scale a running/stopped cks cluster
(localcloud) 🐱 > scale kubernetescluster id=985d339e-0744-4d91-82a8-de95fcfac3cc nodeofferings[0].key=worker nodeofferings=19948eb4-4ef3-42f0-87a6-ec621718f704
🙈 Error: (HTTP 530, error code 9999) Cannot invoke "java.util.Map.put(Object, Object)" because "mapValue" is null
2026-10-30 06:43:15,369 ERROR [c.c.a.ApiServer] (qtp253011924-16:[ctx-64f43a55, ctx-83720331, ctx-f4b35cf7]) (logid:38be252b) unhandled exception executing api command: [Ljava.lang.String;@2027d8b4 java.lang.NullPointerException: Cannot invoke "java.util.Map.put(Object, Object)" because "mapValue" is null
at com.cloud.api.dispatch.ParamUnpackWorker.handle(ParamUnpackWorker.java:104)
at com.cloud.api.dispatch.DispatchChain.dispatch(DispatchChain.java:37)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:814)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:697)
at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:409)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:194)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:191)
at com.cloud.api.ApiServlet.doPost(ApiServlet.java:150)
|
@Pearl1594 could please you look into the issue again |
…fering update for absent node types
|
@kiranchavala I've addressed issues 1 and 2, while issue 3 is a non issue, atleast not pertaining to CKS. The issue occured due to incorrect usage of nodeofferingids. Its a map and you passed a map as well as a UUID, and that's the reason for the failure. This would fail for any API parameter of map type. |
|
@blueorangutan package |
|
@Pearl1594 a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
|
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 19902 |
Description
This PR #14122 #14148
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
How Has This Been Tested?
Deployed a CKS cluster with dedicated service offerings for the node types (v1.35.6) - this isnt reproducible via UI, because via UI passing the common service offering is mandatory (so the kubernetes_cluster.service_offering_id column is never null). Scaled the offering to a default/common one - successfully scaled. (for KVM the cluster needs to be stopped prior upgrade)
Also tried upgrading the cluster - form v1.35.6 to 1.36.2 - Successfully upgraded
How did you try to break this feature and the system with this change?