Falcon 2Key values
- Vocabulary size: 65,024 tokens, comparable to other contemporary open models.
- Beginning and end of sequence token id: 11.
- Untied input and output embeddings, so the embedding tables are not shared.
What this means in practice
A 65,024 vocabulary sits in the middle of the range for models of this size: larger vocabularies shorten sequences but cost more embedding parameters, and smaller ones do the reverse. For Falcon 2 the practical consequence is that English text costs roughly the token counts you would expect from a modern BPE tokenizer, and languages with less training data cost more.
The 8,192 token context window is expressed in these tokens. If you reserve the full window for a request, you are reserving 8,192 positions of cache, not 8,192 words.
Prompt formatting
The text checkpoint is a raw pretrained model, so there is no official instruction template to copy. The VLM checkpoint does have one: User:<image> followed by the instruction, then Falcon: to open the reply. If you build a chat experience on the text model, you are choosing the format, which also means you own the consequences of that choice.
Related pages
Sources
falcon2.lol is an independent third-party site. It is not affiliated with, endorsed by, sponsored by or operated by the Technology Innovation Institute, and it does not speak for TII. Every number on this page is attributed to the source listed above.