ferret: refer and ground anything anywhere at any granularity

Ferret is a new Multimodal Large Language Model (MLLM) that enables referring and grounding capabilities for multimodal large language models (LLM). It allows users to refer to regions or objects in point, box, or any free-form shape, and accurately grounds any open-vocabulary descriptions. The hybrid representation of the region in the input is replaced by the proposed representation before being fed into the LLM. The bounding box coordinates in the output denote the predicted coordinates.