Mission of EldritchWeave centers on exploring potential of combining existing knowledge in the field of AI and natural language processing to push LLM inference, training and general capabilities further. Moreover, the project is equally committed to pioneering novel methodologies and technologies, with promising ideas already in development and poised to contribute significantly to the field. While in its nascent stages, the project is poised to set a new benchmark in AI research and application, addressing both current limitations and future possibilities.
Within the rapidly evolving field of AI and natural language processing, certain novel techniques have emerged as particularly noteworthy. These techniques, characterized by their innovative approaches and potential to significantly advance the discipline, include:
- LongRope : https://arxiv.org/abs/2402.13753
- Latent attention : https://arxiv.org/abs/2402.17512
- Draft and verify : https://arxiv.org/abs/2309.08168
- ResLORA : https://arxiv.org/abs/2402.18039
- Big Bird sparse attention : https://arxiv.org/abs/2007.14062
- Exploiting persistence of importance hypothesis : https://arxiv.org/abs/2305.17118
- LAyer-SElective Rank reduction (LASER) : https://arxiv.org/abs/2312.13558
- PowerInfer : https://arxiv.org/abs/2312.12456
- KV-quant : https://arxiv.org/abs/2401.18079
- Ring Attention with blockwise transformers : https://arxiv.org/abs/2310.01889
- Recurrent-memory transformers : https://arxiv.org/abs/2402.10790
- Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding : https://arxiv.org/abs/2402.12374
- Fast LLM Inference without Auxiliary Models : https://arxiv.org/abs/2402.11131
- Find and Replace transformers : https://openreview.net/forum?id=MGWsPGogLH
Incorporating these cutting-edge techniques underscores EldritchWeave’s dedication to push the frontiers of AI and natural language processing. This project is a testament to the power and innovative rigor of open research in shaping the future of technology, driving us towards a horizon where AI's full potential can be realized.