Issues

• Add sliding windowed attention and different block sizes per layer.

  ReaLLMASIC/ReaLLM-Forge#434 · Filter by author gkielian opened on Apr 22, 2025

• Option to monitor specific learned parameters during training

  ReaLLMASIC/ReaLLM-Forge#429 · Filter by author gkielian opened on Apr 12, 2025

• Bugs when running on Multiple A100

  ReaLLMASIC/ReaLLM-Forge#426 · Filter by author Mars-Cat2023 opened on Apr 7, 2025

• Ssm has following error when running locally on tip of tree

  ReaLLMASIC/ReaLLM-Forge#412 · Filter by author klei22 opened on Mar 4, 2025

• Add entropy colorization of output

  ReaLLMASIC/ReaLLM-Forge#407 · Filter by author gkielian opened on Feb 26, 2025
  

• Add Softcapping implementation

  ReaLLMASIC/ReaLLM-Forge#397 · Filter by author gkielian opened on Feb 13, 2025

• Add visualizations for attention

  ReaLLMASIC/ReaLLM-Forge#389 · Filter by author gkielian opened on Feb 6, 2025

• Implement Early Exit

  ReaLLMASIC/ReaLLM-Forge#388 · Filter by author gkielian opened on Feb 5, 2025
  

• Add Delta Attention variation

  ReaLLMASIC/ReaLLM-Forge#376 · Filter by author gkielian opened on Jan 27, 2025

• Add support for Maximal Update of Parameters - MUP

  ReaLLMASIC/ReaLLM-Forge#372 · Filter by author gkielian opened on Jan 23, 2025
  

• Learned KV Cache Prefix

  ReaLLMASIC/ReaLLM-Forge#369 · Filter by author klei22 opened on Jan 21, 2025

• Add CJK IPA Tokenizer

  ReaLLMASIC/ReaLLM-Forge#367 · Filter by author gkielian opened on Jan 19, 2025
  

• Create a variation with mixins of RNN / RWKV style components

  ReaLLMASIC/ReaLLM-Forge#365 · Filter by author gkielian opened on Jan 19, 2025

• Add support for HookPoints

  ReaLLMASIC/ReaLLM-Forge#364 · Filter by author gkielian opened on Jan 19, 2025

• Add Semantic Factorization

  ReaLLMASIC/ReaLLM-Forge#363 · Filter by author gkielian opened on Jan 19, 2025
  1

  

• Add a Reweighted Cross-Entropy Loss for Improved Handling of Zipfian Distributions

  ReaLLMASIC/ReaLLM-Forge#357 · Filter by author gkielian opened on Jan 10, 2025

• Add means for gauging average average loss per token and token subsets

  ReaLLMASIC/ReaLLM-Forge#356 · Filter by author gkielian opened on Jan 10, 2025

• Add model variation for Gaussian Attention

  ReaLLMASIC/ReaLLM-Forge#355 · Filter by author gkielian opened on Jan 10, 2025

• Add model variations for Megalodon innovations

  ReaLLMASIC/ReaLLM-Forge#354 · Filter by author gkielian opened on Jan 10, 2025

• Add model variation for RPC attention

  ReaLLMASIC/ReaLLM-Forge#353 · Filter by author gkielian opened on Jan 10, 2025

• Add BAdam optimizer option

  ReaLLMASIC/ReaLLM-Forge#351 · Filter by author gkielian opened on Jan 9, 2025

• Add compatibility with CREMA-D Dataset

  AudioDataset
  ReaLLMASIC/ReaLLM-Forge#349 · Filter by author gkielian opened on Jan 6, 2025
  

• Add compatibility with Textrol Speech Dataset

  AudioDataset
  ReaLLMASIC/ReaLLM-Forge#348 · Filter by author gkielian opened on Jan 6, 2025
  

• Add compatibility with DailyTalk dataset

  AudioDataset
  ReaLLMASIC/ReaLLM-Forge#347 · Filter by author gkielian opened on Jan 6, 2025
  

• Add separate ckpt saving for tracking ICL accuracy

  ReaLLMASIC/ReaLLM-Forge#346 · Filter by author gkielian opened on Jan 6, 2025