Run the VLM locally

Run Falcon 2 11B VLM Locally: Memory and Setup

The VLM checkpoint carries the same 11B language model plus a vision tower, so everything you know about the text model memory applies and then some.

Back to the overview

Memory

  • Language weights are the same 11B as the text checkpoint, so budget the same 22 GB at bfloat16.
  • Add the CLIP ViT-L/14 vision encoder and the projection layer.
  • Add image tokens: a tiled high-resolution image consumes context, sometimes several hundred tokens per tile.
  • A 24 GB GPU is the comfortable target; below that, quantisation becomes necessary and quality testing becomes mandatory.

Loading it

The published path uses LlavaNextForConditionalGeneration with LlavaNextProcessor and a tokenizer class of PreTrainedTokenizerFast. The architecture is standard enough that the generic LLaVA-NeXT code path works; the prompt format is Falcon-specific.

The practical trap

Image tokens compete with your text for the same context window. A high-resolution tiled image can consume a large share of it before you have written a single question, which is why the loaded config carries a lower maximum length than the text checkpoint. Plan the budget per request, not per model.

Related pages

Falcon 2 11B VLMVRAM requirementsFalcon 2 VLM vs LLaVA

Sources

falcon2.lol is an independent third-party site. It is not affiliated with, endorsed by, sponsored by or operated by the Technology Innovation Institute, and it does not speak for TII. Every number on this page is attributed to the source listed above.

Run Falcon 2 11B VLM Locally: Memory and Setup