Falcon 2 11B Model Card: Parameters, Context and Licence

Architecture at a glance

Architecture: FalconForCausalLM, a causal decoder-only transformer with parallel attention.
Layers: 60. Attention heads: 32 query heads against 8 key/value heads.
Hidden size: 4,096. Feed-forward size: 16,384.
Positional encoding: rotary embeddings with a theta of 500,042.
Context window: 8,192 tokens.
Vocabulary: 65,024 tokens. Tied embeddings: off.
Default precision: bfloat16.

Training data

TII trained Falcon 2 11B on more than 5,000 billion tokens, drawing on the RefinedWeb dataset and additional curated corpora. The model card lists English, German, Spanish, French, Italian, Portuguese, Polish, Dutch, Romanian and Czech, with Swedish also named in the model detail section.

How to run it

The weights are about 22 GB in bfloat16, which puts the model on a single 24 GB GPU. Text Generation Inference is the fastest supported path, and 4-bit quantisations bring the memory requirement down to roughly 7-8 GB.

Licence

Falcon 2 11B is published under the TII Falcon License 2.0. Hugging Face labels the repository licence as unknown because the licence text lives on TII's site rather than in a standard SPDX file, so read the licence at the source before you ship.

Sources

falcon2.lol is an independent third-party site. It is not affiliated with, endorsed by, sponsored by or operated by the Technology Innovation Institute, and it does not speak for TII. Every number on this page is attributed to the source listed above.