Skip to content
View zigzagcai's full-sized avatar
🎯
Evolving
🎯
Evolving
89 followers · 166 following

Achievements

Achievement: Pull Sharkx3Achievement: YOLOAchievement: Pair Extraordinairex2Achievement: Quickdraw

Achievements

Achievement: Pull Sharkx3Achievement: YOLOAchievement: Pair Extraordinairex2Achievement: Quickdraw

Highlights

Block or report zigzagcai

Block user

Prevent this user from interacting with your repositories and sending you notifications. Learn more about blocking users.

You must be logged in to block users.

Maximum 250 characters. Please don’t include any personal information such as legal names or email addresses. Markdown is supported. This note will only be visible to you.
Report abuse

Contact GitHub support about this user’s behavior. Learn more about reporting abuse.

Report abuse
zigzagcai/README.md

Hi there 😄

Short Bio

I am Zheng Cai, nickname zigzagcai, and AI Infra engineer, working on (M)LLM pre/post-train and Automatic Kernel Optimization.

My Thinking

I love the general idea of open source (code/knowledge/and others) and love to learn from open source community and try my best to contribute back.

Selected thoughts I have ever written:

  1. CPU memory optimization when using PyTorch Dataloader over very large-scale datasets: pytorch/pytorch#13246 (comment) — 2024/08
  2. Analysis of numerical stability between Ring and Tree All-Reduce: NVIDIA/nccl#1055 - 2023/11
  3. Implement variable-length training with Mamba State Space Models: state-spaces/mamba#244 - 2024/03
  4. Avoid deadlock when training with ColossalAI over very large-scale GPU clusters: hpcaitech/ColossalAI#5625 - 2024/04
  5. DeepSeek V3 671B trainable with FSDP+EP by hacking two lines of PyTorch FSDP codes: https://github.com/zigzagcai/DeepSeekV3 - 2025/04
  6. Support nogil feature in NumPy-1.18.5 in the experimental CPython ecosystem: https://github.com/colesbury/numpy/commit/0d6ef2770268711ee6417792ba0da35fcb264bf5 - 2022/07
  7. 由A800平台训练InternLM-7B无法收敛引发的思考 - 2023/11
  8. 支持变长序列的Mamba-1训练 - 2024/08

Pinned Loading

  1. varlen_mamba varlen_mamba Public

    Forked from state-spaces/mamba

    Mamba SSM architecture that supports training on variable-length sequences. Implementation of ECCV24' <PackMamba: Efficient processing of variable-length sequences in mamba training>

    Python 12 1

  2. DeepSeekV3 DeepSeekV3 Public

    Simple and efficient implementation of 671B DeepSeek V3 that trainable with FSDP+EP and minimal requirement of 256x A100/H100, targeted for HuggingFace ecosystem

    Python 8 2

  3. InternLM/InternEvo InternLM/InternEvo Public

    InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.

    Python 421 67

  4. hpcaitech/ColossalAI hpcaitech/ColossalAI Public

    Making large AI models cheaper, faster and more accessible

    Python 41.4k 4.5k

  5. algorithmicsuperintelligence/openevolve algorithmicsuperintelligence/openevolve Public

    Open-source implementation of AlphaEvolve

    Python 7.3k 1.1k