Untied Ulysses: Memory-Efficient Context Parallelism via Headwise Chunking
Efficiently processing long sequences with Transformer models usually requires splitting the computations across accelerators via context parallelism.
Research
Academic or research source. Check the methodology, sample size, and whether it's been replicated.