Is there an efficient way to apply NAtten to 1D data?
Guaishou74851 opened this issue · comments
Hi, your work is really inspiring and interesting!
I am wondering if there is a simple way to apply your Natten block to 1D data with a shape of (B, C, L), where L is the sequence length?
I make a copy of NAtten folder, but I have no idea about adjusting the source codes, for making the block act like Conv1d, with a 1D kernel (e.g. 49*1 kernel)?
Hello and thank you for your interest,
We plan to release a complete torch extension in the near future with support for both single and multi-dimensional data, as well as other relevant properties.
In the case of 1D data, it is actually relatively easy to do, so we will release a version of that in the coming days.
I'll keep this issue open until then, so you get notified when we merge the PR.
Actually I just went ahead and did it, considering how simple it was.
Now you can just import NeighborhoodAttention1d from natten which will work on 4D tensors (Batch, Head, Length, Dim) instead of 5D.
I'll keep this open in case you have any issues, but feel free to close it if it works out.
It works well in my program, thanks for your so timely responses and efforts~