karpathy / llm.c

LLM training in simple, raw C/CUDA

Repository from Github https://github.comkarpathy/llm.c

karpathy/llm.c Issues

ERROR on the AMD GPU
Closed a year ago4
Modal script - benchmarking, profiling and libraries
Updated a year ago6
Model Export & Inference
Updated a year ago3
Cudnn error cudnn_att.cpp on train_gptcu
Closed a year ago5
BitNet (b1.58) support
Updated a year ago2
python dev/data/fineweb.py --version 10B
Closed a year ago2
Deleting Conda/Python as a dependency entirely to dramatically decrease "latency to step"
Updated a year ago4
Recalculating the activations in the backwards pass to conserve memory
Updated a year ago3
Mismatch of dweight at layernorm_backward.cu
Closed a year ago
ThunderKittens Backend
Updated a year ago1
2D and 3D tile divisions so that permutation coordinates can be read from threadIdx and blockIdx
Updated a year ago3
Broader vendor support for hardware acceleration
Updated a year ago5
compute sanitizers
Updated a year ago1
Is there a plan to support 8bits (FP8 or INT8)?
Updated a year ago2
`make` fails to autodetect GPU compute capability
Updated a year ago3
Llm on small models
Closed a year ago
more detailed explanation of Multi GPU
Updated a year ago4
How to do Inference on the trained weight of GPT 2 model after finishing the training on CPU using train_gpt2.py and train_gpt2 ?
Updated a year ago1
MultiGPU training hangs
Closed a year ago9
make: *** [Makefile:194: train_gpt2] Error 2 on Windows
Closed 2 years ago6
Assertion `graph->check_support(cudnn_handle).is_good()' failed
Updated a year ago21
When will llama and other frameworks be supported?
Updated 2 years ago1
Error: make: *** [Makefile:203: train_gpt2cu] Error 255
Updated 4 months ago8
About pull request of custom kernel implementation
Closed 2 years ago
What would be the main design trade-offs when re-implementing in clean modern C++?
Updated 2 years ago2
Possible bugs in the data loading functions
Updated a year ago1
void tokenizer_init failed
Closed a year ago3
Possible NULL Pointer Dereference
Closed 2 years ago2
delete use of cooperative groups in kernels
Updated 2 years ago2
inf loss at big batch
Updated a year ago1
Hardcoded block_size in kernels
Updated 2 years ago14
cuda code that approaches cublas performance
Updated 2 years ago
WikiText 103 evaluation
Closed a year ago1
Refactoring all of the shared cuda helper methods to the shared common file
Closed 2 years ago
Splitting cuda dev files to use smaller sizes for cpu validation compared to profiling
Updated 2 years ago
init from scratch
Updated 9 months ago4
CI Mac issue with resources for Python?
Closed 2 years ago5
Error while running "make train_gpt2fp32cu" on Ubuntu
Updated a year ago15
Windows Github actions / workflow is successfully building including Cuda 12.4 builds
Closed a year ago12
test_gpt2.cu correctness bounds tune per-parameter
Updated a year ago1
CI tests are dependent on connectivity/uptime of huggingface
Closed 2 years ago3
bug: something goes wrong at larger batch sizes
Closed 2 years ago6
Suddenly "Out of memory" on train/python and train, test/CUDA on 4090
Closed 2 years ago3
Input token length question
Updated 2 years ago2
Is there a final bin file that can input some question and output some answer?
Updated 2 years ago1
[.gitignore] gitignore does not ignore all binary files
Closed 2 years ago3
layernorm_backward.cu: atomicAdd
Updated 2 years ago
__stcs in layernorm_forward_kernel3 function in train_gpt2.cu
Closed 2 years ago3
CPU: atomicAdd(float*)
Updated 2 years ago12
Should we use 'T' instead of 'BT' here? I'm not sure either
Updated 2 years ago3