Question about discrepancy between implementations available in the repo and related papers

Question

Question about discrepancy between implementations available in the repo and related papers

sylee0124 opened this issue a year ago · comments

Hi, I'm bit confused about current implementations of the repo and implementations used/discussed in related papers. I'll just state what I think is true. Please correct me if I'm wrong.

Flashconv from h3
Fused kernel is implemented at fftconv_cuda.cu but it is not using block FFT.
FlashButterfly in "Simple Hardware-Efficient Long Convolutions for Sequence Modeling"
long_conv.py uses BlockFFT (which is same as Butterfly Decomposition) with support for learnable parameters for dft_matrix. But not using fused kernel and Three-pass algorithm is also not implemented.

Dan Fu · Answer 1 · Tue Mar 14 2023 19:57:19 GMT+0800 (China Standard Time)

These are correct, we are cleaning up the code/algorithms for the fast block FFT and three pass algorithm across into a single package, this repository is focused on the architecture pieces for now. Will update this issue when released!

…

On Tue, Mar 14, 2023 at 6:52 AM Lee Seung Yul ***@***.***> wrote: Hi, I'm bit confused about current implementations of the repo and implementations used/discussed in related papers. I'll just state what I think is true. Please correct me if I'm wrong. - Flashconv from h3 Fused kernel is implemented at fftconv_cuda.cu but it is *not using block FFT*. - FlashButterfly in "Simple Hardware-Efficient Long Convolutions for Sequence Modeling" long_conv.py uses BlockFFT (which is same as Butterfly Decomposition) with support for learnable parameters for dft_matrix. But *not using fused kernel* and *Three-pass algorithm* is also not implemented. — Reply to this email directly, view it on GitHub <#5>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABDDIIX33MA6IYWP65TVIALW4BEWFANCNFSM6AAAAAAV2IUUYI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

Lee Seung Yul · Answer 2 · Tue Mar 14 2023 20:39:44 GMT+0800 (China Standard Time)

Thanks for verifying :)
When can I expect this performance update? Will it happen anytime soon?

Dan Fu · Answer 3 · Tue Mar 14 2023 20:45:59 GMT+0800 (China Standard Time)

Hopefully soon! I’ve been traveling for a bit, but have some time to code again soon.

…

On Tue, Mar 14, 2023 at 8:39 AM Lee Seung Yul ***@***.***> wrote: Thanks for verifying :) When can I expect this performance update? Will it happen anytime soon? — Reply to this email directly, view it on GitHub <#5 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABDDIIQDGU7RRW6X2BNZCX3W4BRJXANCNFSM6AAAAAAV2IUUYI> . You are receiving this because you commented.Message ID: ***@***.***>