Issues

• qa_2 (HotpotQA): string_match_part matches the gold answer "no" inside other words (not, know, cannot), so hedged or refusing answers get full credit

  NVIDIA/RULER#112 · Filter by author shaurya416 opened on Sep 23, 2026

• Optional EvalPort export for RULER's per-task predictions/summary.csv

  NVIDIA/RULER#111 · Filter by author adhabnr-ux opened on Sep 22, 2026

• {Suggestion] - Table results update

  NVIDIA/RULER#104 · Filter by author Avroboros opened on Mar 15, 2026
  1

• Question: could "long-horizon tension crash tests" be a complementary workload for RULER?

  NVIDIA/RULER#103 · Filter by author onestardao opened on Feb 6, 2026
  3

• Inquiry about Model Evaluation and Leaderboard Inclusion

  NVIDIA/RULER#101 · Filter by author zhuangzhong10 opened on Feb 3, 2026

• 超过128k是如何做截断的？

  NVIDIA/RULER#97 · Filter by author llllllaz opened on Oct 15, 2025
  2

• string_match_all only evaluates recall but not precision

  NVIDIA/RULER#95 · Filter by author seyuboglu opened on Jul 28, 2025
  2

• Add support for multi-node vllm serving

  NVIDIA/RULER#89 · Filter by author surprisedPikachu007 opened on May 29, 2025

• Issue with getting the Docker Container Operational

  NVIDIA/RULER#83 · Filter by author heoun opened on Mar 26, 2025
  1

• Weighted Average Values

  NVIDIA/RULER#82 · Filter by author bowencohere opened on Feb 21, 2025
  2

• Can't reproduce results from the paper

  NVIDIA/RULER#80 · Filter by author eldarkurtic opened on Jan 16, 2025

• Does it really take 2 days to evaluate llama-3.1-8b at 128k length?

  NVIDIA/RULER#79 · Filter by author eldarkurtic opened on Jan 16, 2025
  3

• Rough runtime benchmarks across tasks and context-lengths on any hardware setups

  NVIDIA/RULER#76 · Filter by author girishbalaji opened on Dec 3, 2024

• Evaluate on Jamba-1.5-Mini

  NVIDIA/RULER#69 · Filter by author coranholmes opened on Sep 24, 2024
  19

• Note about Mistral models

  NVIDIA/RULER#67 · Filter by author inflatebot opened on Sep 20, 2024
  1

• Gemini flash 1.5 results

  NVIDIA/RULER#43 · Filter by author augusto-rehfeldt opened on Jul 31, 2024
  1

• gpt-4o results?

  NVIDIA/RULER#12 · Filter by author the21st opened on May 16, 2024
  3

• Raw scores?

  NVIDIA/RULER#10 · Filter by author WesleyYue opened on May 14, 2024
  3