Synthesizer
Paper Reading Notes: “Synthesizer: Rethinking Self-Attention for Transformer Models”
Efficient-Llms
Reformer
Paper Reading Notes: “Reformer: The Efficient Transformer”
FNet: Mixing Tokens with Fourier Transforms
Paper-reading notes: FNet
Linformer: Self-Attention with Linear Complexity
Paper-reading notes: Linformer
Rethinking Attention with Performers
Paper-reading notes: Performers
ATLAS: Learning to Optimally Memorize the Context at Test Time
Paper-reading notes: ATLAS
Titans: Learning to Memorize at Test Time
Paper-reading notes: Titans
Roformer: Enhanced Transformer With Rotary Position Embedding
Paper-reading notes: Roformer
Hyena Hierarchy: Towards Larger Convolutional Language Models
Paper-reading notes: Hyena Hierarchy
Mamba: Linear-Time Sequence Modeling with Selective State Spaces
Paper-reading notes: Mamba