kvm: bound Ceph operations by default

Default rados.osd.op.timeout to 60 and rados.mon.op.timeout to 30. With the
options left unset, an operation that never completes still parks the agent
thread, and with it the host's storage command queue, for anyone who does not
know to set them. A failed operation can be retried; a wedged host cannot.

Setting either property to 0 still leaves the option unset and keeps the
librados behaviour of waiting forever, and a cluster whose single operations
legitimately run longer can raise them.

Signed-off-by: Brad House <bhouse@nexthop.ai>

Author: xxxxxxxxxxxxx

agent/conf/agent.properties

@@ -463,18 +463,17 @@ iscsi.session.cleanup.enabled=false
 
 # Time, in seconds, that a single Ceph OSD operation may block before it fails. This bounds one operation
 # and not a whole request, so a long running copy or flatten is made up of many operations that are each
-# well inside the limit.
-# The default of 0 leaves the option unset, which is the librados behaviour of waiting forever. An
-# operation that never completes parks the agent thread that issued it, and because the agent runs a
-# host's storage commands in sequence, every command queued behind it stops with it.
-# Setting a limit is recommended. 60 is a reasonable starting point: it is far above a normal operation
-# on a healthy cluster, while still bounding one that will never return.
-# rados.osd.op.timeout=0
+# well inside the limit. 60 is far above a normal operation on a healthy cluster, while still bounding one
+# that will never return.
+# Set to 0 to leave the option unset, which is the librados behaviour of waiting forever. An operation
+# that never completes then parks the agent thread that issued it, and because the agent runs a host's
+# storage commands in sequence, every command queued behind it stops with it. Raise the limit instead on
+# a cluster where single operations legitimately run longer, for example during heavy recovery.
+# rados.osd.op.timeout=60
 
 # Time, in seconds, that a single Ceph monitor operation may block before it fails.
-# The default of 0 leaves the option unset, which is the librados behaviour of waiting forever.
-# 30 is a reasonable starting point.
-# rados.mon.op.timeout=0
+# Set to 0 to leave the option unset, which is the librados behaviour of waiting forever.
+# rados.mon.op.timeout=30
 
 # Time, in seconds, that the Ceph client may spend connecting to the monitors.
 # Set to 0 to leave the option unset and keep the librados default.

agent/src/main/java/com/cloud/agent/properties/AgentProperties.java

@@ -918,22 +918,22 @@ public Property<Integer> getWorkers() {
      * Time, in seconds, that a single Ceph OSD operation may block before it fails.<br>
      * This bounds each operation, not a whole request, so a long running copy or flatten is made up of
      * many operations that are each well inside the limit.<br>
-     * The default of <code>0</code> leaves the option unset, which keeps the librados behaviour of waiting
+     * Set to <code>0</code> to leave the option unset, which keeps the librados behaviour of waiting
      * forever. A stuck operation then blocks the agent thread that issued it, and with it every storage
-     * command queued behind it for that host, so setting a limit is recommended.<br>
+     * command queued behind it for that host.<br>
      * Data type: Integer.<br>
-     * Default value: <code>0</code>
+     * Default value: <code>60</code>
      */
-    public static final Property<Integer> RADOS_OSD_OP_TIMEOUT = new Property<>("rados.osd.op.timeout", 0);
+    public static final Property<Integer> RADOS_OSD_OP_TIMEOUT = new Property<>("rados.osd.op.timeout", 60);
 
     /**
      * Time, in seconds, that a single Ceph monitor operation may block before it fails.<br>
-     * The default of <code>0</code> leaves the option unset, which keeps the librados behaviour of waiting
+     * Set to <code>0</code> to leave the option unset, which keeps the librados behaviour of waiting
      * forever.<br>
      * Data type: Integer.<br>
-     * Default value: <code>0</code>
+     * Default value: <code>30</code>
      */
-    public static final Property<Integer> RADOS_MON_OP_TIMEOUT = new Property<>("rados.mon.op.timeout", 0);
+    public static final Property<Integer> RADOS_MON_OP_TIMEOUT = new Property<>("rados.mon.op.timeout", 30);
 
     /**
      * Time, in seconds, that the Ceph client may spend connecting to the monitors.<br>