How to Run Falcon 2 Locally: Step by Step

1. Decide where it will run

bfloat16 needs about 22 GB of weights, so a 24 GB GPU is the straightforward target.
4-bit quantisation needs roughly 7-8 GB, which fits a 12 GB card or a large-memory CPU box.
Below 8 GB of free memory, do not start: the model will load and then fail on the first long prompt.

2. Pick the weights

The canonical repository is tiiuae/falcon-11B. Community quantised builds exist for llama.cpp and for 4-bit GPU inference, and they are the practical choice below 24 GB. Quantised builds change the numerics, so validate output quality on your own prompts rather than assuming parity.

3. Load and generate

The model card uses transformers with the falcon architecture and requires a recent PyTorch. The important detail is that the checkpoint is a raw pretrained model: it continues text. For conversation you add the formatting yourself, or you start from the VLM checkpoint, which was built from a chat-finetuned backbone.

4. Serve it rather than scripting it

For repeated use, put a server in front. Text Generation Inference gives you an HTTP API and continuous batching; llama.cpp gives you a small single binary and CPU support. A one-off Python script is fine for a test, not for a product.

Sources

falcon2.lol is an independent third-party site. It is not affiliated with, endorsed by, sponsored by or operated by the Technology Innovation Institute, and it does not speak for TII. Every number on this page is attributed to the source listed above.