VRAM requirements

Falcon 2 11B VRAM Requirements Including KV Cache

Almost every Falcon 2 11B sizing answer online quotes weights only. Weights are the floor, not the requirement. Here is the rest of the arithmetic.

Back to the overview

The three components

  • Weights: 22 GB at bfloat16, 11-12 GB at 8-bit, 7-8 GB at 4-bit.
  • KV cache: grows with the context window you reserve and the number of concurrent requests.
  • Runtime overhead: the inference engine, tokenizer and activation buffers need their own room.

Why the context window matters

Falcon 2 11B is configured at 8,192 tokens. A server that reserves the full window for every request allocates far more cache than one that reserves 2,048, and the difference can be the margin between fitting and not fitting. Set the limit you actually need.

A simple planning rule

Take the weight size, add the cache for your context length multiplied by how many users you expect at once, then add about ten percent. If that total exceeds your VRAM, change one of the three inputs: quantise, shorten the context, or limit concurrency. There is no fourth option at a fixed memory size.

Related pages

GPU requirementsContext lengthQuantisation

Sources

falcon2.lol is an independent third-party site. It is not affiliated with, endorsed by, sponsored by or operated by the Technology Innovation Institute, and it does not speak for TII. Every number on this page is attributed to the source listed above.

Falcon 2 11B VRAM Requirements Including KV Cache