Falcon 2 with vLLM

Run Falcon 2 with vLLM: What Works and What to Watch

vLLM is a common choice for serving Falcon 2 11B because its paged attention and continuous batching keep throughput reasonable under load.

Back to the overview

Sizing the deployment

  • Single 24 GB GPU for bfloat16, with the remainder of memory going to KV cache.
  • Set the maximum model length to what you actually need: Falcon 2 11B is configured at 8,192 tokens, and reserving the full window in KV cache costs memory you may not have.
  • Use tensor parallelism only when you have more than one GPU: on a single card it adds overhead and no capacity.

What usually goes wrong

First, memory: the model loads, then the server crashes under concurrent long prompts because KV cache was never budgeted. Second, the chat template: vLLM will happily serve a raw pretrained model, and a raw pretrained model will happily continue your prompt instead of answering it. That is a behaviour problem, not a serving problem.

Throughput expectations

Continuous batching is the reason to use a serving engine at all: it lets one GPU work on several requests at once instead of serialising them. Batch size one latency and batched throughput are different numbers, and only the second one predicts what your users feel under load.

Related pages

Deploy as an APIVRAM requirementsRun Falcon 2 locally

Sources

falcon2.lol is an independent third-party site. It is not affiliated with, endorsed by, sponsored by or operated by the Technology Innovation Institute, and it does not speak for TII. Every number on this page is attributed to the source listed above.

Run Falcon 2 with vLLM: What Works and What to Watch