ymd-h / tsax

TSax

Warning
This module is still under development, and doesn't work yet.

TSax is a Python package providing Time Series Forecasting Models.

The name comes from Time Series on JAX / Flax

Algorithms

• Transformer
  • A. Vaswani et al., "Attention is All you Need", NIPS 2017, arXiv:1706.03762
  • Notes: This is not intended for Time Series Forecasting, but for understanding of Transformer basics.
• Reformer
  • N. Kitaev et al., "Reformer: The Efficient Transformer", ICLR 2020, arXiv:2001.04451
• (WIP) Informer
  • H. Zhou et al., "Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting", AAAI 2021, Vol. 35, No. 12, arXiv:2012.07436
• (WIP) Autoformer
  • H. Wu et al., "Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting", NeurIPS 2021, arXiv:2106.13008
• FEDformer
  • T. Zhou et al., "FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting", ICML 2022 aeXiv:2201.12740
• ...

Avoid to Vanishing Gradient

• ResiDual
  • S. Xie et al., "ResiDual: Transformer with Dual Residual Connections", arXiv:2304.14802
• Admin
  • L. Liu et al., "Understanding the Difficulty of Training Transformers", Empirical Methods in Natural Language Processing (EMNLP), (2020) arXiv:2004.08249
• Sin2-max-shifted, Sin-Softmax
  • S. Wang et al., "Escaping the Gradient Vanishing: Periodic Alternatives of Softmax in Attention Mechanism", IEEE Access Vol.9, 168749--168759 (2021), doi: 10.1109/ACCESS.2021.3138201 arXiv:2108.07153
• ...

Development

docstring

We obey Numpy's Style guide, and the docstring is treated by Sphinx Napoleon extension.

Annotation

Follow JAX Typing Best Practices. Use jax.typing.ArrayLike for inputs, jax.Array for outputs.