Lighthouse Attention: A Training-Only Hierarchical Selection Method for Faster LLM Pretraining

Large language models (LLMs) struggle with long sequences because the standard attention mechanism—scaled dot-product attention (SDPA)—scales quadratically in both compute and memory with sequence length. FlashAttention reduced memory but didn't fix the compute scaling. Researchers at Nous Research have introduced Lighthouse Attention, a training-only method that uses a selection-based hierarchical approach to achieve a 1.4–1.7× pretraining speedup while maintaining or improving final model quality. Unlike prior sparse methods, Lighthouse keeps selection outside the attention kernel, allowing reuse of optimized dense kernels.

1. What is the core problem with standard attention for long sequences?

Standard scaled dot-product attention (SDPA) has O(N²) computational and memory cost for sequence length N. This means that as models are trained on longer contexts, the cost grows quadratically, making long-context training prohibitively expensive. While FlashAttention uses IO-aware tiling to avoid materializing the full attention matrix in memory, it doesn't reduce the underlying compute; the quadratic scaling remains a bottleneck. For pretraining, where sequences can be hundreds of thousands of tokens long, this cost becomes a major limitation. Lighthouse Attention targets this specific problem by reducing the amount of compute needed during the forward and backward passes of training, without sacrificing model performance.

2. Why do existing sparse attention methods fall short for pretraining?

Most prior sparse attention methods—like NSA, HISA, DSA, and MoBA—share two shortcomings. First, they apply asymmetric compression: they pool only keys and values while keeping queries at full resolution, which can lose important information for the selection mechanism. Second, their selection logic is baked into custom attention kernels, preventing reuse of the highly optimized dense-attention kernels on modern GPUs. This custom kernel approach hurts training speed and makes it difficult to integrate with existing frameworks. Additionally, these methods are often designed for inference, where the sparse model is compared against its dense counterpart. For training, the challenge is that the resulting weights must still work well with dense attention at inference time. Lighthouse treats this correctness criterion as central.

3. How does Lighthouse Attention differ from previous sparse methods?

Lighthouse makes two key design changes. First, it applies symmetric compression by pooling queries, keys, and values together across a multi-level pyramid. This creates coherent triples at each level, ensuring that selection is based on both query and key information. Second, it places the entire selection process outside the attention kernel. After selection, the chosen entries are gathered into a contiguous, dense sub-sequence and processed using a standard FlashAttention kernel—the same one used in dense baselines. This means teams can leverage existing optimized kernels without custom code. By keeping selection separate, Lighthouse also simplifies engineering and allows for easy adoption. The method is training-only; after training, the model uses dense attention as usual.

4. What is the four-stage pipeline of Lighthouse Attention?

A Lighthouse attention layer wraps around standard SDPA without modifying it. The pipeline operates in four stages: Pyramid Construction, Scoring, Selection, and Dense Attention. First, average pooling builds an L-level pyramid from queries, keys, and values symmetrically, producing compressed triples at each level. Second, a parameter-free scorer assigns two scalar scores to each pyramid entry: a query score and a key score, based on per-head L2 norms. Coarser levels inherit scores from finer levels. Third, the selection stage picks the top entries using these scores to form a contiguous, dense sub-sequence. Fourth, this sub-sequence is fed into standard FlashAttention for both forward and backward passes. The entire pipeline is efficient, with pyramid construction costing O(N) time and memory.

5. How does the symmetric pyramid pooling work?

The symmetric pyramid pooling is a key innovation in Lighthouse Attention. For each attention layer, the method applies average pooling with a factor p to build L levels. At level ℓ, the sequence length reduces to N/p^ℓ, with each token summarizing p^ℓ base positions. Crucially, the pooling is applied identically to queries (Q), keys (K), and values (V), producing coherent (Q^(ℓ), K^(ℓ), V^(ℓ)) triples at every level. This symmetry ensures that when selecting which tokens to attend to, the method considers both query and key representations together, avoiding the information loss from asymmetric compression. The pyramid structure allows the selection to be hierarchical: finer levels capture local details, while coarser levels capture global context. The total cost of building the pyramid scales linearly with sequence length, adding minimal overhead.

6. How does the selection process avoid custom kernels?

In Lighthouse Attention, the selection logic is implemented as a separate step outside the attention kernel. After scoring and ranking the pyramid entries, the method gathers the selected tokens into a contiguous, dense sequence using standard gather operations (e.g., torch.gather). This dense sub-sequence is then passed to the same optimized FlashAttention kernel used by dense baselines. Because selection does not require custom GPU kernel code, it can leverage existing libraries (e.g., FlashAttention, cuDNN) and benefits from their ongoing optimizations. This design also makes Lighthouse easy to integrate into existing training pipelines—developers only need to add the small, modular selection logic before the attention call. The training speedup comes from the fact that the attention kernel processes a much shorter sequence (typically 10–25% of original length), while the selection overhead is linear and minor.

7. What speedups does Lighthouse Attention achieve, and does it maintain model quality?

In experiments by Nous Research, Lighthouse Attention delivers a 1.40× to 1.69× end-to-end wall-clock speedup compared to a cuDNN-backed SDPA baseline during pretraining. This is achieved without any loss in model quality; in fact, the method shows matching or lower final training loss across multiple configurations. The speedup comes from reducing the effective sequence length for the attention operation, while the hierarchical selection ensures that important tokens are still attended to. The method is training-only—after pretraining, the model uses standard dense attention at inference, so there is no trade-off in deployment quality. This makes Lighthouse a practical solution for accelerating long-context LLM training while maintaining performance.