Context length

Falcon 2 11B Context Length: 8,192 Tokens and Its Costs

Falcon 2 11B ships with an 8,192 token maximum position embedding. That is the window, and it is a configuration value rather than a suggestion.

Back to the overview

The number

  • Maximum position embeddings: 8,192 tokens for the text checkpoint.
  • The VLM config carries its own maximum length value in the text sub-config, which is lower.
  • Anything longer must be chunked, summarised or retrieved into, not pasted in.

Why 8,192 is a real constraint

For chat and summarisation this is comfortable. For long-document question answering, code repositories or long transcripts it is not: 8,192 tokens is roughly 6,000 English words, minus whatever the prompt and the answer need. Models released later in 2024 and 2025 commonly offer several times that, which is one reason comparisons against newer models are not like for like.

Working inside the window

  • Retrieve the relevant passages rather than the whole document.
  • Summarise incrementally and carry the summary, not the source.
  • Budget the answer as well as the prompt: generation consumes the same window.
  • Measure actual token counts rather than estimating from word counts.

Related pages

VRAM requirementsFalcon 2 vs Llama 3.1 8BParameters

Sources

falcon2.lol is an independent third-party site. It is not affiliated with, endorsed by, sponsored by or operated by the Technology Innovation Institute, and it does not speak for TII. Every number on this page is attributed to the source listed above.

Falcon 2 11B Context Length: 8,192 Tokens and Its Costs