Florence-2

Microsoft Research released Florence-2 in June 2024 as a leading open-source model in multi-task computer vision. The model features a sequence-to-sequence architecture based on a Transformer that integrates a DaViT image encoder with a text decoder, unifying tasks such as captioning, object detection, segmentation, OCR, and grounding under a single prompt-based interface. It comes in two main sizes: Florence-2-base (0.23B parameters) and Florence-2-large (0.77B parameters), offering flexibility for different compute budgets. Trained on the FLD-5B dataset comprising 5.4 billion annotations across 126 million images, Florence-2 delivers strong zero-shot performance and competitive fine-tuning results. The model is freely available on Hugging Face with pretrained weights, code, and demo notebooks, making it ideal for researchers and developers building advanced vision applications without licensing costs.

Architecture and Performance

Florence-2 employs a dual-transformer architecture: a hybrid vision encoder (DaViT) extracts visual features, and a standard text decoder (BART-like) generates text. Each vision task is converted into a text format via specific prompts (e.g., <CAPTION>, <OD>, <OCR>), allowing the model to handle multiple tasks without architectural changes. This design reduces engineering complexity and facilitates scaling. The model has demonstrated superior performance on benchmarks like COCO and Flickr30k, achieving up to 85% accuracy in captioning and competitive object detection accuracy. Florence-2 also supports OCR in multiple languages, though Arabic support is limited and requires fine-tuning for optimal results.

Key Performance Features

The model offers high inference speed on mid-range GPUs: Florence-2-base runs on 8GB VRAM, while large requires 16GB. It provides a REST API via Hugging Face Inference Endpoints for easy integration into web and mobile applications. Zero-shot learning enables immediate use without additional training, reducing development costs. The community provides helper tools like Python libraries (transformers) and ready-made examples on GitHub, accelerating adoption.