Falcon 2 with Ollama

Run Falcon 2 with Ollama: Quantised and Local

Ollama runs quantised GGUF builds, which is the shortest path to Falcon 2 11B on a laptop or a CPU server. The trade-off is that you are not running the original bfloat16 weights.

Back to the overview

Why a GGUF build is required

Ollama does not load the original safetensors checkpoint. You need a GGUF conversion of Falcon 2 11B, which the community publishes at several quantisation levels. Confirm the conversion came from tiiuae/falcon-11B before you rely on it.

Choosing a quantisation

  • Q4 class builds land around 7 GB and fit comfortably in 8-12 GB of memory.
  • Q5 and Q6 builds trade memory for fidelity if you have the headroom.
  • Q8 approaches the original size and stops being attractive on small machines.

The honesty caveat

Quantisation changes the model. Any benchmark number you read for Falcon 2 11B was produced on the original weights, so it does not transfer to your local build automatically. If a task matters, compare the quantised build against a hosted bfloat16 run on the same prompts.

Related pages

QuantisationFalcon 2 with llama.cppRun Falcon 2 locally

Sources

falcon2.lol is an independent third-party site. It is not affiliated with, endorsed by, sponsored by or operated by the Technology Innovation Institute, and it does not speak for TII. Every number on this page is attributed to the source listed above.

Run Falcon 2 with Ollama: Quantised and Local