llm.c
https://github.com/karpathy/llm.c
Cuda
LLM training in simple, raw C/CUDA
Triage Issues!
When you volunteer to triage issues, you'll receive an email each day with a link to an open issue that needs help in this project. You'll also receive instructions on how to triage issues.
Triage Docs!
Receive a documented method or class from your favorite GitHub repos in your inbox every day. If you're really pro, receive undocumented methods or classes and supercharge your commit history.
Cuda not yet supported2 Subscribers
Add a CodeTriage badge to llm.c
Help out
- Issues
- Add 🦀 Rust port with safety + CUDA to notable forks
- Fix: Offset cast to int in fseekCheck breaks datasets >2GB
- added RoPE/GQA/SwiGLU
- Assignment Submission: RMSNorm Implementation
- Issue #243: Add GPT-2 initialization from scratch
- Is there any pretrain weights for this project?
- Can we use Flash Attention 2?
- Runs under Windows
- [Feature Req]: Any plan on parallel training like deepseeed(zero3, offload nvme) and megatron(tensor parallem, layer pileline)?
- MPI run with upto 3xH100 GPU works but starts giving an "illegal memory access" error when run using "mpirun -np 4 ./train_gpt2cu"
- Docs
- Cuda not yet supported