DefTruth / CUDA-Learn-Notes

🎉CUDA/C++ 笔记 / 技术博客: fp32、fp16/bf16、fp8/int8、flash_attn、sgemm、sgemv、warp/block reduce、dot prod、elementwise、softmax、layernorm、rmsnorm、hist etc.

Home Page:https://github.com/DefTruth/cuda-learn-notes

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

您好,请问sigmoid算子这里为啥没有考虑指数溢出问题

Phoenix8215 opened this issue · comments

// Sigmoid x: N, y: N y=1/(1+exp(-x))
// grid(N/128), block(K=128) 
__global__ void sigmoid(float* x, float* y, int N) {
  int idx = blockIdx.x * blockDim.x + threadIdx.x;
  if (idx < N) y[idx] = 1.0f / (1.0f + expf(-x[idx]));
}

我看ncnn他是做了相关的处理,链接:https://github.com/Tencent/ncnn/blob/02ba6766bd2ecd08600736c033e7346142313466/src/layer/sigmoid.cpp#L41

欢迎提交PR进行优化~

欢迎提交PR进行优化~

好咧,谢谢你的repo🤞