karpathy / llm.c

LLM training in simple, raw C/CUDA

Repository from Github https://github.comkarpathy/llm.c

layernorm_backward.cu: atomicAdd

azret opened this issue 2 years ago · comments

Azret Botash commented 2 years ago

llm.c/dev/cuda/layernorm_backward.cu

Line 208 in 7b79fbb

dinp_bt[i] += dval;

Should all of these use atomicAdd?