Solar Open 2 uses hybrid attention for million-token context
Upstage released a 250B mixture-of-experts model activating 15B parameters per token. Three linear-attention layers for every softmax layer reduce KV-cache pressure while supporting one-million-token context.
Why it matters: Hybrid attention is becoming a credible alternative to full softmax attention at every layer for long-context workloads.