Falcon 2Weights first
- bfloat16: about 22 GB of weights, so 24 GB of VRAM is the practical minimum.
- 8-bit: roughly 11-12 GB, so a 16 GB card has room to work.
- 4-bit: roughly 7-8 GB, which brings 12 GB cards into range.
- CPU with 16 GB or more of system memory can run a 4-bit build without a GPU.
Then the KV cache
Everything above is weights only. Serving a request also allocates KV cache, which grows with context length and concurrency. Falcon 2 11B has 60 layers with 8 key/value heads, which keeps the per-token cache smaller than a same-size model with full multi-head attention, but it is still not free.
The trap is sizing for one request at a time. A 24 GB card that runs a single 8K-token conversation comfortably can still fall over when several arrive at once.
Picking hardware
For evaluation and small internal use, a 24 GB workstation card is the simplest answer. For anything public, size from a load test at your expected concurrency, and remember that a quantised model on cheaper hardware is a quality decision, not just a cost one.
Related pages
Sources
falcon2.lol is an independent third-party site. It is not affiliated with, endorsed by, sponsored by or operated by the Technology Innovation Institute, and it does not speak for TII. Every number on this page is attributed to the source listed above.