FlashAttention
A widely used technique that makes the attention step of a transformer far faster by moving less data around on the GPU.
Attention is the part of a transformer where every token looks at every other token to decide what matters. Done naively it produces a giant intermediate table and then spends most of its time shuffling that table between fast and slow memory on the graphics card. FlashAttention rearranges the computation so the table never has to be written out in full, which means the same result with a fraction of the memory traffic.
It is one of those rare optimisations that changed what was practical rather than just what was fast: longer context windows became affordable largely because of it. Successive versions, up to FlashAttention-4, are the baseline any new attention kernel gets measured against.