Technology

microbatching

1 synchronized story carry this tag.

llama.cpp b11223 fixes long-input causal reranking

Signed prerelease b11223 allows RANK-pooling batches to split for causal LLM rerankers such as Qwen3 and Qwen3-VL. The server now checks the effective runtime attention mode instead of hardcoded architectures, so long-document and multimodal reranking can use chunked prefill rather than failing above the microbatch limit.

Why it matters: The change removes a correctness and availability boundary for long or multimodal reranking while respecting runtime attention overrides. It matters to retrieval pipelines whose inputs exceed `n_ubatch`.

Read storyOriginal source · github.com