Quantisation

Falcon 2 11B Quantisation: 4-bit, 8-bit and What You Lose

Quantisation is the difference between running Falcon 2 11B on a workstation and running it on nothing. It is also the change most likely to quietly degrade your output.

Back to the overview

The size ladder

  • bfloat16: about 22 GB, the reference configuration.
  • 8-bit: about 11-12 GB, usually close to the reference in output quality.
  • 4-bit: about 7-8 GB, the common laptop-class choice.
  • Below 4-bit: expect visible degradation and test carefully before deploying.

What degrades first

Aggressive quantisation tends to show up in long-form coherence, arithmetic and multi-step reasoning before it shows up in short factual answers. A quantised model can look fine on a demo prompt and fail on the task you actually care about.

How to test it properly

  • Build a prompt set from your own real tasks, not from a public benchmark.
  • Run both the quantised build and a bfloat16 reference on the same set.
  • Score them blind where you can, and keep the prompts with the scores.
  • Re-run after any runtime or quantisation change, because the result is specific to the build.

Related pages

VRAM requirementsFalcon 2 with llama.cppGPU requirements

Sources

falcon2.lol is an independent third-party site. It is not affiliated with, endorsed by, sponsored by or operated by the Technology Innovation Institute, and it does not speak for TII. Every number on this page is attributed to the source listed above.

Falcon 2 11B Quantisation: 4-bit, 8-bit and What You Lose