bug(multi-agent): Python agent council times out at 120s due to serial Ollama calls, bypassed rule engine fast-path & missing DAG/caching #418

Bug Description

When running kdm analyze --multi-agent (or kdm analyze --explain with the Ollama backend) against a Kubernetes cluster or Minikube with failing workloads, the Python Multi-Agent Council fails with a timeout watchdog error:

⚠ Multi-agent council failed, using direct completion: Python agent council timed out after 120000ms

While #383 previously increased the timeout threshold from 45s to 120s, the underlying architectural bottlenecks remain unaddressed:

1. Serial Specialist Execution: agents/council.py executes 4 blocking Ollama calls in strict sequence (RuntimeLogAgent -> ConfigDependencyAgent -> ClusterResourceAgent -> SynthesizerAgent). With local LLMs (e.g., 9.5 GB gemma4:e4b-mlx or llama3.1), each inference takes 35–45s, yielding a total runtime of 140s–160s, which mathematically breaches the 120s watchdog.
2. Bypassed Deterministic Rule Engine: council.py:run_council() directly calls council.run() rather than council.diagnose(). Obvious deterministic failures (CrashLoopBackOff, ImagePullBackOff, OOMKilled, missing ConfigMap) that can be diagnosed in < 2ms with 100% confidence are never checked and are forced into the 2.5-minute serial LLM pipeline.
3. Inverted Fallback Ladder in Node.js: src/analysis/analysis.ts waits for the 120s Python council timeout before attempting applyRuleEngineFallback(). Deterministic rule checks should be evaluated upfront before spawning the Python subprocess.
4. Lack of Per-Agent Timeouts & Cache: An individual agent can block indefinitely without a timeout, and duplicate failures across multiple pod replicas trigger redundant LLM council runs.

Steps to Reproduce

1. Start Minikube and deploy failing workloads (e.g., CrashLoopBackOff pod):

   kubectl run test-crash --image=busybox --restart=Always -- sh -c "echo 'fatal startup'; exit 1"

2. Ensure Ollama is running with a local model (e.g., gemma4:e4b-mlx or llama3.1):

   ollama list

3. Run KDM multi-agent diagnostic analysis:

   kdm analyze deployment/test-broken-deployment --multi-agent
   # or
   kdm analyze -n default --explain

4. Observe that after 120 seconds of blocking, the command fails with:

   ⚠ Multi-agent council failed, using direct completion: Python agent council timed out after 120000ms
   

Expected Behavior

• Deterministic failures (CrashLoopBackOff, ImagePullBackOff, OOMKilled, CreateContainerConfigError) should be diagnosed instantly in < 5ms by the Rule Engine fast-path without invoking the LLM.
• If LLM deliberation is required for non-deterministic issues, independent specialist agents (Runtime, Config, Resource) should execute concurrently in parallel rather than serially.
• The total council runtime should stay well under 45–60s on local hardware.
• Duplicate pod failures should be resolved in $O(1)$ via an in-memory signature cache.

Actual Behavior

• The Rule Engine is bypassed by run_council().
• Four serial LLM calls run in sequence, taking ~140–160s.
• Node.js ProcessLifecycleWatcher triggers a SIGTERM/SIGKILL at 120,000ms.
• Fallback ladder is delayed by a 2-minute freeze.

Root Cause Analysis

Proposed Architectural Solution & Data Structures

1. Directed Acyclic Graph (DAG) / Fork-Join Pattern (concurrent.futures)

The 3 specialist agents have zero dependencies between each other (in-degree = 0). They should be dispatched concurrently in a thread pool:

with ThreadPoolExecutor(max_workers=3) as executor:
    futures = {
        executor.submit(runtime_agent.analyze, failure_text, ctx): "runtime",
        executor.submit(config_agent.analyze, failure_text, ctx): "config",
        executor.submit(resource_agent.analyze, failure_text, ctx): "resource",
    }
    # Wait with per-agent timeout (≤ 20s)

• Latency impact: Drops total execution from $\sum T_i$ (160s) to $\max(T_i) + T_{\text{synth}}$ (45s–55s).

2. Wire Deterministic Rule Engine Fast-Path in run_council

Evaluate council.rule_engine before triggering any LLM calls:

rule_match = council.rule_engine.evaluate_text(failure_text, context)
if rule_match and council.rule_engine.should_bypass_llm(rule_match):
    diagnosis = ConsensusDiagnosis.from_rule_match(rule_match, target_name=context.get("name", ""))
    emit_event("complete", {"consensus": diagnosis.to_dict()})
    return diagnosis.to_dict()

3. Radix / Trie Pattern Matcher ($O(L)$)

Implement a trie/prefix lookup for known error signatures (CrashLoopBackOff, ImagePullBackOff, OOMKilled, CreateContainerConfigError, exit code 137) for instantaneous $O(L)$ deterministic categorization.

4. In-Memory LRU Cache ($O(1)$)

Cache consensus diagnoses keyed by Hash(kind, namespace, normalized_error_signature):

• Replica 1 of a crashing deployment incurs the investigation.
• Replicas 2–10 hit the LRU cache in $< 0.1\text{ms}$, preventing cluster-wide analysis denial-of-service.

5. Short-Circuit in Node.js (src/analysis/analysis.ts)

Run applyRuleEngineFallback() before spawning the Python subprocess bridge:

if (applyRuleEngineFallback(fallbackCtx)) {
  return { success: true, response: fallbackCtx.result.details };
}

Environment

• OS: macOS Darwin arm64 (Apple Silicon)
• Node.js: v22.x
• Ollama: v0.31.x
• Local Model: gemma4:e4b-mlx (9.5 GB) / llama3.1
• Kubernetes / Minikube: v1.35.1 / v1.38.1
• KDM Version: 3.1.0 / 4.0.0