Parameters

Falcon 2 11B Parameters: The Full Configuration

Falcon 2 11B has 11 billion parameters, which tells you very little on its own. This page lists the configuration values that actually predict how the model behaves.

Back to the overview

The published configuration

  • Hidden size: 4,096.
  • Layers: 60.
  • Attention: 32 query heads, 8 key/value heads, multi-query attention enabled.
  • Feed-forward hidden size: 16,384 with a factor of 4.
  • Vocabulary: 65,024 tokens, untied embeddings.
  • Rotary position embeddings with theta 500,042.
  • Context window: 8,192 tokens.

Why the head count matters

Thirty-two query heads sharing eight key/value heads means the KV cache is a quarter of what a full multi-head configuration would need. That is why an 11B model can serve a useful amount of context on a single 24 GB GPU without the cache becoming the dominant cost.

What is not published

TII does not break down the parameter count by component, so there is no published figure separating embeddings, attention and feed-forward parameters. Anyone quoting a split is estimating, not reporting.

Related pages

Falcon 2 11BContext lengthVRAM requirements

Sources

falcon2.lol is an independent third-party site. It is not affiliated with, endorsed by, sponsored by or operated by the Technology Innovation Institute, and it does not speak for TII. Every number on this page is attributed to the source listed above.

Falcon 2 11B Parameters: The Full Configuration