Falcon 2 benchmarks

Falcon 2 11B Benchmarks: What Was Published and What Was Not

Falcon 2 11B has one widely quoted benchmark number and very little else. This page collects what was published and is explicit about what was not.

Back to the overview

The published record

  • Hugging Face Open LLM Leaderboard average: 64.28 for Falcon 2 11B.
  • The same leaderboard placed Google Gemma 7B at 64.29.
  • TII states Falcon 2 11B outperformed Meta Llama 3 8B on that leaderboard.
  • A technical report exists as arXiv 2407.14885.

What is missing

There is no published per-task breakdown from TII on the model card, no latency or throughput benchmark, and no vision benchmark for the VLM checkpoint. The leaderboard that produced the headline number has since been replaced by newer evaluation suites, so the figure is not reproducible from the original harness today.

How to benchmark it yourself

  • Fix your prompt set and decoding parameters, and record them alongside the scores.
  • Measure the quantisation you will actually deploy, not bfloat16.
  • Measure tokens per second at your real concurrency, not batch size one.
  • Publish the date with any number you cite, including ours.

Related pages

Falcon 2 vs Gemma 7BParametersFalcon 2 11B

Sources

falcon2.lol is an independent third-party site. It is not affiliated with, endorsed by, sponsored by or operated by the Technology Innovation Institute, and it does not speak for TII. Every number on this page is attributed to the source listed above.

Falcon 2 11B Benchmarks: What Was Published and What Was Not