Training data

Falcon 2 11B Training Data: RefinedWeb and 5,000B Tokens

Falcon 2 11B was trained on more than 5,000 billion tokens, primarily from RefinedWeb with additional curated corpora. That is the disclosure; it is deliberately broad.

Back to the overview

What is disclosed

  • More than 5,000 billion training tokens.
  • RefinedWeb as the main dataset, published by TII as tiiuae/falcon-refinedweb.
  • Additional curated corpora, not itemised.
  • A published technical report, arXiv 2407.14885.

RefinedWeb in one paragraph

RefinedWeb is a large-scale web dataset that TII built and released, filtered and deduplicated rather than curated by hand from a fixed source list. It is a web corpus, which is the honest way to understand the model: it reflects the distribution of the filtered web, including its biases and its gaps.

What is not disclosed

There is no published token count per language, no breakdown of the curated corpora, and no cutoff date stated on the model card. TII also warns that although the model handles several languages, it was trained mostly on English and will not generalise well to languages outside the mix.

Related pages

LanguagesFalcon 2 11BFalcon 2 benchmarks

Sources

falcon2.lol is an independent third-party site. It is not affiliated with, endorsed by, sponsored by or operated by the Technology Innovation Institute, and it does not speak for TII. Every number on this page is attributed to the source listed above.

Falcon 2 11B Training Data: RefinedWeb and 5,000B Tokens