Dao-AILab

flash-attention: Compute Exact Attention Faster and with Less Memory

FlashAttention provides GPU-optimized implementations of exact attention for deep learning, reducing memory use while improving speed. It is aimed at practitioners training or serving Transformer models who can build and run GPU kernels.

View on GitHub Compare
Stars
25.1k
Forks
3.1k
Stars, last 30 days
+159

Our take

Health score

63/100

How it is scored
Activity
73
Community
72
Issues
60
Pull requests
46

+159 stars in the last 30 days

  • Releases more often than 82% of the projects we track
  • More contributors than 78% of the projects we track

Project history

Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.

Selected snapshot

Stars

25,320

Forks

3,241

Change since Nov 18, 2026: +5 stars·+1 forks

Overview

FlashAttention is a GPU attention library for PyTorch and a reference implementation of the FlashAttention family of research methods. It computes exact attention while reducing the memory traffic and intermediate storage that make standard attention costly, particularly for long sequences.

It is useful when attention is a training or inference bottleneck and the workload can run on supported NVIDIA or AMD GPUs. The repository includes multiple generations of implementations, with hardware-specific options, so the appropriate package and backend depend on your accelerator and CUDA or ROCm setup.

Key Features

  • Provides FlashAttention and FlashAttention-2 implementations for exact scaled dot-product attention.
  • Supports causal masks, sliding-window attention, ALiBi, and dropout in core interfaces.
  • Handles multi-query and grouped-query attention, plus variable-length sequences.
  • Includes KV-cache attention for incremental decoding, with optional in-place cache updates and paged caches.
  • Offers NVIDIA CUDA and AMD ROCm paths, including a Triton backend for AMD.
  • Includes FlashAttention-3 beta support for Hopper GPUs and FlashAttention-4 using CuTeDSL for Hopper and Blackwell.
  • Provides PyTorch modules and model/training code alongside kernel tests.

Use Cases

  • Train Transformer models with long sequences when standard attention's memory use limits batch size or context length.
  • Serve autoregressive language models, where optimized KV-cache attention can help with incremental decoding.
  • Add attention with causal, local-window, or grouped-query behavior to a PyTorch model on compatible GPU hardware.
  • Benchmark or develop GPU attention kernels across NVIDIA and AMD backends.

Getting Started

For the main package, install with:

pip install flash-attn --no-build-isolation

This requires a compatible GPU toolchain and PyTorch. See the README for hardware-specific requirements, backend instructions, and API examples.

Alternatives

  • TensorRT-LLM: TensorRT-LLM is a full NVIDIA-focused LLM inference framework with optimized attention kernels, while FlashAttention focuses on exact attention implementations.
ProjectLanguageLicenseStarsStatus
flash-attentionPythonBSD-3-Clause25.1kNot checked yet
TensorRT-LLMPython–15kActive

Considerations

  • This is a GPU kernel library, not a hardware-independent attention implementation. The README lists supported GPU architectures and toolkit versions for each backend.
  • The main installation requires Linux, PyTorch 2.2 or newer, and a CUDA or ROCm toolkit. Windows support is described as needing more testing.
  • Building from source can be resource-intensive. The README recommends Ninja and documents limiting parallel jobs with MAX_JOBS.
  • FlashAttention-3 is labeled a beta release and is limited to Hopper GPUs with CUDA 12.3 or newer. FlashAttention-4 has separate installation and hardware guidance.
  • AMD backend capabilities differ by backend, and the README notes that sliding-window attention in the Triton backend is a work in progress.

Found this useful?

Share it with someone who would like flash-attention.

Comparisons