Falcon 2 with llama.cpp

Run Falcon 2 with llama.cpp: CPU, GPU or Both

llama.cpp is the option that does not assume a GPU. With a 4-bit GGUF build, Falcon 2 11B becomes a file you can run on a CPU and offload to a GPU as far as your memory allows.

Back to the overview

How offload works

llama.cpp lets you choose how many layers live on the GPU. A small card can hold part of the model and the CPU handles the rest, which turns a hard memory limit into a latency slope: the fewer layers on the GPU, the slower the tokens.

Realistic expectations

  • CPU-only, 4-bit, modern desktop: a few tokens per second, usable for batch work, painful for chat.
  • Partial offload on a 8-12 GB card: a workable middle ground for a single user.
  • Full offload on a 24 GB card: the fast small-machine configuration.

When llama.cpp is the right answer

Choose it when the alternative is not running the model at all: offline machines, CPU-only servers, or a laptop where you want a local assistant and can accept slow responses. Choose a serving engine when other people depend on the endpoint.

Related pages

Falcon 2 with OllamaQuantisationVRAM requirements

Sources

falcon2.lol is an independent third-party site. It is not affiliated with, endorsed by, sponsored by or operated by the Technology Innovation Institute, and it does not speak for TII. Every number on this page is attributed to the source listed above.

Run Falcon 2 with llama.cpp: CPU, GPU or Both