skills.rest
Nzettodess
NzettodessCOMMUNITY
Explore all skills by this author →
GitHub

llava

Runs vision-language conversations and visual question answering with LLaVA models.

2|Updated Jan 26, 2026
Search Tags:#fine-tuning#multimodal#vision-language#vqa#visual question answering#llava#image chat
One-click install
npx skills add https://github.com/Nzettodess/Awesome-Agent-Skills --skill llava-nzettodess
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llava
Source: https://github.com/Nzettodess/Awesome-Agent-Skills/tree/main/Skills/Agentic%20AI%20Development/AI-research-SKILLs-1.1.0/18-multimodal/llava
Command: npx skills add https://github.com/Nzettodess/Awesome-Agent-Skills --skill llava-nzettodess

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires transformers, torch, pillow, and includes references (resource) components.

What problem does it solve? Building image-understanding chatbots and visual question answering systems requires integrating a vision encoder with a language model, which involves complex setup, model selection, and VRAM management. This Skill provides ready-to-use instructions for running LLaVA models for conversational image analysis. ## Core Features & Use Cases - Visual Question Answering: Ask natural-language questions about images and receive detailed answers using LLaVA 7B, 13B, or 34B models. - Multi-Turn Image Chat: Maintain conversation context across multiple questions about the same image via CLI, Python API, or Gradio web UI. - Custom Fine-Tuning: Train LLaVA on your own instruction data with full fine-tuning or memory-efficient LoRA, including DeepSpeed multi-GPU support. - Use Case: A team building a document-understanding assistant loads llava-v1.5-7b with 4-bit quantization on a single GPU, then queries scanned document images for their main topics through a Gradio interface. ## Quick Start Ask the agent to load the llava-v1.5-7b model and describe what is in a provided image file.

Frequently Asked Questions about llava

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run visual question answering with LLaVA in Python?▼

Load a pretrained model with llava.model.builder.load_pretrained_model, process the image with the image processor, build a conversation prompt with the image token, and call model.generate. The response is decoded from the output token IDs.

Which LLaVA model size should I use for image chat?▼

Start with LLaVA-v1.5-7B, which needs about 14 GB VRAM in FP16 or 4 GB with 4-bit quantization. Use 13B or 34B for higher quality if you have 28-70 GB of VRAM available.

Can LLaVA run on a GPU with limited VRAM?▼

Yes, enable 4-bit quantization with load_4bit=True to reduce VRAM usage by roughly 4x, or 8-bit for about 2x reduction. A 7B model then fits in about 4 GB of VRAM.

LLaVA vs GPT-4V vs CLIP for image understanding?▼

LLaVA is open-source and self-hostable for conversational image analysis. GPT-4V offers higher quality via API, while CLIP suits simple zero-shot classification rather than dialogue.

How do I fine-tune LLaVA on custom image data?▼

Format your data as JSON with image paths and human/gpt conversation pairs, then run the fine-tune script with DeepSpeed. LoRA fine-tuning reduces memory needs about 10x, fitting a 7B model on one A100.

What are the limitations of LLaVA for image analysis?▼

LLaVA can hallucinate objects not present in images, struggles with precise spatial reasoning and small text, and counts objects imprecisely. It also requires a GPU, as CPU inference is extremely slow.

Related Skills

Explore More Education & Research Skills→
SJTU-IPADSSJTU-IPADS
7

image-utils

Automate deterministic image editing tasks on image collections with Pillow.

Official
Intermediate
X-School-AcademyX-School-Academy
19

media-image-analysis

Analyze images with AI vision models to answer questions about content.

Official
Basic
davidrrowleydavidrrowley

azure-ai-vision-imageanalysis-java

Extract captions, OCR text, tags, objects, people, and smart crops from images via Azure AI Vision SDK.

Community
Intermediate
VYRE-StudiosVYRE-Studios

clip

Classify images and retrieve matching visual content using pretrained CLIP models.

Official
Intermediate