Alibaba Qwen · mixture of experts · Qwen Community License 1.0

What runs Qwen3.8 Flash-Next

Sparse MoE with 125B parameters and 6B active per token, plus a large n-gram embedding table the card describes as suited to offloading. Aimed at 128 GB unified-memory machines and multi-GPU rigs.

Parameters

125B, 6B active

4-bit file

111.4 GB

Q4_K_M

Needs at 8k context

113.1 GB

context memory estimated

Longest context

262,144 tokens

Context memory: 24,576 bytes per token, an estimate from the config: 12 attention layers x 2 KV heads x 256 x 2 (K,V) x 2 bytes; 36 linear layers hold a fixed state. Gated DeltaNet + Qwen Sparse Attention (QSA): 12 of 48 layers are attention layers (2 KV heads, head dim 256); the rest are linear attention with a fixed-size state. The card does not publish a per-token KV figure. The card also lists 51B of n-gram embedding parameters and 4B MTP on top of the 125B. How context memory works.