skills.rest
Orchestra-Research
Orchestra-ResearchOFFICIAL
Explore all skills by this author →
GitHub

llava

Enables visual instruction tuning and multi-turn image conversations using CLIP and Vicuna models.

13.0k|930|Updated Nov 3, 2025
Search Tags:#multimodal#instruction tuning#clip#vision-language#visual question answering#llava#image chat
One-click install
npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill llava-orchestra-research
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llava
Source: https://github.com/Orchestra-Research/AI-research-SKILLs/tree/main/18-multimodal/llava
Command: npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill llava-orchestra-research

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires transformers, torch, pillow, and includes references (resource) components.

What problem does it solve? Running open-source vision-language models for image understanding requires complex setup of vision encoders, language models, and conversation templates, which this Skill simplifies into ready-to-use workflows. ## Core Features & Use Cases - Visual Question Answering: Ask natural language questions about images and receive detailed answers using LLaVA models from 7B to 34B parameters. - Multi-Turn Image Chat: Maintain conversational context across multiple questions about the same image for interactive analysis. - Custom Model Training: Fine-tune LLaVA on your own instruction data with full training or memory-efficient LoRA approaches. - Use Case: A team building a document understanding chatbot can load LLaVA-v1.5-7B with 4-bit quantization on a single GPU and answer questions about scanned document images through a Gradio web interface. ## Quick Start Ask the AI to load the LLaVA-v1.5-7B model and describe what is in an attached image file.

Frequently Asked Questions about llava

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I use LLaVA for visual question answering?▼

Load a pretrained model like liuhaotian/llava-v1.5-7b with load_pretrained_model, process the image with the image processor, and generate a response using a conversation template with your question. GPU inference is required for practical speed.

What is the difference between LLaVA and CLIP?▼

CLIP performs zero-shot image classification by matching images to text labels, while LLaVA combines a CLIP vision encoder with a language model for open-ended conversations, detailed descriptions, and instruction following about images.

How much VRAM does LLaVA need to run?▼

The 7B model needs about 14 GB in FP16 or 4 GB with 4-bit quantization. The 13B model needs 28 GB FP16 or 8 GB quantized, and the 34B model needs 70 GB FP16 or 18 GB quantized.

Can I fine-tune LLaVA on my own image data?▼

Yes, prepare instruction data as JSON with image paths and human-gpt conversation pairs, then run the fine-tune script with DeepSpeed. LoRA fine-tuning reduces memory needs to a single A100 for the 7B model.

What are the limitations of LLaVA for image analysis?▼

LLaVA can hallucinate objects not present in images, struggles with precise spatial reasoning and counting many objects, and has difficulty reading small text. It also requires a powerful GPU and runs slower than CLIP.

Related Skills

Explore More Education & Research Skills→
matlabmatlab
995

matlab-use-cameras

Connect to cameras and acquire images in MATLAB using Image Acquisition Toolbox videoinput.

Official
Intermediate
X-School-AcademyX-School-Academy
19

media-image-analysis

Analyze images with AI vision models to answer questions about content.

Official
Basic
tassiovaletassiovale
13

liteparse

Extract text and bounding boxes from PDFs, Office files, and images.

Community
Intermediate
VYRE-StudiosVYRE-Studios

clip

Classify images and retrieve matching visual content using pretrained CLIP models.

Official
Intermediate