Falcon 2The memory arithmetic
- bfloat16 weights: about 22 GB, plus KV cache for the context you use.
- 4-bit quantisation: roughly 7-8 GB of weights.
- 8-bit quantisation: roughly 11-12 GB of weights.
- Practical bf16 target: a 24 GB card such as an RTX 4090, L4 24GB or A10G 24GB.
Throughput versus quality
Quantisation buys memory and often speed at the cost of some quality, and the loss is not uniform: reasoning and long-form generation usually degrade before short factual answers do. If you quantise, test on your own prompts rather than trusting a leaderboard number.
Serving on one GPU
Text Generation Inference is the supported high-throughput path for the text checkpoint. It handles continuous batching, which matters as soon as more than one person uses the model, and it is what most hosted Falcon 2 deployments are built on.
What one GPU does not give you
One GPU is one point of failure and one queue. If you are serving other people rather than yourself, the sizing question stops being "does it fit" and becomes "how many concurrent requests before latency is unacceptable". That number comes from your own load test, not from a model card.
Related pages
Sources
falcon2.lol is an independent third-party site. It is not affiliated with, endorsed by, sponsored by or operated by the Technology Innovation Institute, and it does not speak for TII. Every number on this page is attributed to the source listed above.