tao-generate-image-grounding

Extract referring expressions and bounding boxes from image-caption pairs using Vision-Language Models.

83|20|Updated Apr 16, 2026
One-click install
npx skills add https://github.com/NVIDIA-TAO/tao-skill-bank --skill tao-generate-image-grounding
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tao-generate-image-grounding
Source: https://github.com/NVIDIA-TAO/tao-skill-bank/tree/main/skills/data/tao-generate-image-grounding
Command: npx skills add https://github.com/NVIDIA-TAO/tao-skill-bank --skill tao-generate-image-grounding

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires nvcr.io/nvidia/tao/tao-toolkit:6.26.3-pyt, and includes scripts (resource) and references (resource) components.

What problem does it solve?

This Skill automates the process of grounding captions to pixel-space bounding boxes, generating phrase-grounded annotations, and auto-labeling images for grounding. It uses Vision-Language Models (VLMs) to extract referring expressions from image-caption pairs.

Core Features & Use Cases

  • Image Grounding: Extracts referring expressions from (image, caption) pairs and grounds them to pixel-space bounding boxes.
  • Phrase Grounding: Generates phrase-grounded annotations for each expression.
  • Auto-labeling: Automatically labels images for grounding.
  • Use Case: Ideal for generating phrase-grounded training data for referring-expression and grounding models.

Quick Start

Use the tao-generate-image-grounding skill to extract referring expressions and bounding boxes from the image 'cat.jpg' and its caption 'A cute cat'.

Frequently Asked Questions about tao-generate-image-grounding

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I automate image grounding to generate pixel-space bounding boxes from captions?▼

Image grounding extracts referring expressions from image-caption pairs and grounds them to pixel-space bounding boxes. Vision-Language Models automate this extraction, generating phrase-grounded annotations for each expression.

Can I use vLLM or Gemini for auto-labeling phrase grounding annotations?▼

Yes, auto-labeling for phrase grounding supports Gemini, NIM, vLLM, and other OpenAI-compatible endpoints. These Vision-Language Models automate the extraction of referring expressions from image-caption pairs.

What is phrase grounding and how does it work with Vision-Language Models?▼

Phrase grounding generates phrase-grounded annotations for referring expressions within captions. Vision-Language Models process image-caption pairs to extract these expressions and ground them to corresponding pixel-space bounding boxes.

Do I need Python and specific image files to generate referring expression annotations?▼

Yes, generating referring expression annotations requires Python and access to an image file alongside its corresponding caption file. These inputs are processed through Vision-Language Model endpoints to produce grounded annotations.

What's the best way to create phrase-grounded training data for grounding models?▼

Automated phrase-grounded training data is generated by extracting referring expressions and bounding boxes from image-caption pairs. This approach is ideal for producing training data for referring-expression and grounding models.

Does this image grounding approach work with OpenAI-compatible endpoints?▼

Yes, the image grounding mechanism supports Gemini, NIM, vLLM, and other OpenAI-compatible endpoints. This allows flexible integration with various Vision-Language Models to extract referring expressions.