🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading

InternEvo is an open-sourced lightweight training framework aims to support model pre-training without the need for extensive dependencies.

Slicing a PyTorch Tensor Into Parallel Shards

LLM inference engine from scratch — paged KV cache, continuous batching, chunked prefill, prefix caching, speculative decoding, CUDA graph, tensor parallelism, OpenAI-compatible serving

Decentralized LLMs fine-tuning and inference with offloading

Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*

An Efficient and Versatile Inference Engine for Distributed LLM Serving

JORA: JAX Tensor-Parallel LoRA Library (ACL 2024)

A distributed training framework for large language models powered by Lightning.

Fast and easy distributed model training examples.

Tensor Parallelism with JAX + Shard Map

Tencent Hy3 295B MoE (NVFP4) on 2x NVIDIA DGX Spark — TP2 over 200GbE, 256K context, 26 tok/s end-to-end. Tuned, benchmarked, agent-ready.

GPU Memory Calculator for LLM Training - Calculate GPU memory requirements for training Large Language Models with support for multiple training engines including PyTorch DDP, DeepSpeed ZeRO, Megatron-LM, and FSDP.

Communication cost modeling for tensor parallel LLM inference with TP vs PP vs hybrid comparison, VRAM analysis, pipeline bubble modeling, regime detection, and cost-efficiency. Shows TP dominates on NVLink, PP has 47% bubble at 8 GPUs, and LLaMA-70B needs 8× A100 or 2× H100 for VRAM.

A production-grade, native Rust speculative inference engine for Apple Silicon with Metal GPU acceleration and paged attention.

MiniMax-M3 (428B MoE) running on 3× RTX PRO 6000 Blackwell at TP=3 with 240K context, FP8 KV cache, and working multimodal vision input. Includes dist_utils.py patch for non-divisible attention heads.

Multi-GPU tensor/context parallel diffusion on AMD ROCm — with the patch that makes it actually work.

This repository outlines a comprehensive guide for training a distributed deep learning model.

Optimized dual AMD Strix Halo (gfx1151) vLLM MoE inference: TP=2 over USB4, tuned int4 MoE kernel, ~8us interconnect, serialized serving

This repository focuses on distributed and parallel computing with PyTorch, covering model parallelism, data parallelism, and advanced optimization techniques. It provides resources for scaling AI training and inference efficiently across multiple devices.