Falcon 2How the vision half is built
TII took the chat-finetuned Falcon 2 11B and attached a pretrained CLIP ViT-L/14 vision encoder, then trained the pair on image-text data. The published architecture is LlavaNextForConditionalGeneration.
The vision tower is a CLIP model with a 336 pixel input, 24 layers, 1,024 hidden size and a 768 dimensional projection into the language model.
High-resolution handling
Small detail is the usual failure mode of vision models. Falcon 2 11B VLM uses dynamic encoding at high resolution: the config lists image grid pinpoints at 336x672, 672x336, 672x672 and 1008 pixels, so a large image can be tiled instead of squashed into one small square.
The prompt format matters
The model card uses a plain, specific format: User:<image> followed by the instruction, then " Falcon:" to open the reply. Getting the format wrong is the most common reason the model appears to ignore the image.
- Prompt: User:<image> + newline + instruction + " Falcon:"
- Load with LlavaNextForConditionalGeneration and LlavaNextProcessor.
- Language: English only. The text checkpoint is multilingual, the VLM is not.
Frequently asked questions
Can Falcon 2 11B VLM read text in a screenshot?+
It can attempt it, and high-resolution tiling is designed for exactly this kind of fine detail, but TII publishes no OCR benchmark for the model. Test it on your own documents before relying on it.
Related pages
Sources
falcon2.lol is an independent third-party site. It is not affiliated with, endorsed by, sponsored by or operated by the Technology Innovation Institute, and it does not speak for TII. Every number on this page is attributed to the source listed above.