What changed
Upstage released a 250B mixture-of-experts model activating 15B parameters per token. Three linear-attention layers for every softmax layer reduce KV-cache pressure while supporting one-million-token context.
Why it matters
Hybrid attention is becoming a credible alternative to full softmax attention at every layer for long-context workloads.
What to do
Evaluate it through hosted inference for large archives and persistent research traces before considering self-hosting.
Caveat
Recommended deployment still needs roughly eight H200 or B200-class GPUs, and the custom licence includes naming and attribution conditions.