skills.rest
Clay-HHK
Clay-HHKCOMMUNITY
Explore all skills by this author →
GitHub

llava

Run vision-language conversations and visual question answering with LLaVA models.

1|Updated Mar 5, 2026
Search Tags:#fine-tuning#multimodal#clip#vision-language#visual question answering#llava#image chat
One-click install
npx skills add https://github.com/Clay-HHK/claude-skills --skill llava-clay-hhk
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: llava
Source: https://github.com/Clay-HHK/claude-skills/tree/main/llava
Command: npx skills add https://github.com/Clay-HHK/claude-skills --skill llava-clay-hhk

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires transformers, torch, pillow, and includes references (resource) components.

What problem does it solve? Building image-understanding chatbots and visual question answering systems requires integrating vision encoders with language models, which involves complex setup, model loading, and conversation management that this Skill streamlines. ## Core Features & Use Cases - Multi-turn Image Conversations: Load LLaVA models (7B to 34B) and conduct contextual multi-turn dialogues about images using CLIP vision encoders with Vicuna/LLaMA backbones. - Visual Question Answering & Captioning: Perform VQA, image captioning, scene understanding, and document understanding through simple prompt-based queries. - Custom Training & Fine-tuning: Fine-tune on custom instruction data with full training or memory-efficient LoRA, plus 4-bit/8-bit quantization for reduced VRAM. - Use Case: Deploy a Gradio web interface where users upload product photos and ask questions like "What defects are visible?" using a 4-bit quantized LLaVA-13B model on a single GPU. ## Quick Start Ask the assistant to load the llava-v1.5-7b model and describe what is in a provided image file.

Frequently Asked Questions about llava

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run LLaVA for visual question answering in Python?▼

Load a pretrained model with llava.model.builder.load_pretrained_model, process the image with the image processor, build a conversation using conv_templates, and call model.generate with the image tensor. Pretrained checkpoints like liuhaotian/llava-v1.5-7b are available on Hugging Face.

What is the difference between LLaVA, CLIP, and BLIP-2?▼

LLaVA supports multi-turn conversational image understanding and instruction following, while CLIP handles simple zero-shot classification and BLIP-2 is better suited for captioning only. GPT-4V offers higher quality but is API-based rather than open-source.

How much VRAM does LLaVA need for inference?▼

The 7B model needs about 14 GB in FP16 or 4 GB with 4-bit quantization, the 13B needs about 28 GB or 8 GB quantized, and the 34B needs about 70 GB or 18 GB quantized. GPU inference is required since CPU inference is extremely slow.

Can I fine-tune LLaVA on my own image dataset?▼

Yes, format your data as JSON with image paths and human/gpt conversation pairs, then run the fine-tuning script with DeepSpeed. LoRA fine-tuning reduces memory needs to a single A100 for the 7B model.

What are the limitations of LLaVA for image understanding?▼

LLaVA can hallucinate objects not present in images, struggles with precise spatial reasoning and small text, and counts objects imprecisely. It also requires substantial GPU memory and runs slower than CLIP.

Related Skills

Explore More Education & Research Skills→
matlabmatlab
995

matlab-use-cameras

Connect to cameras and acquire images in MATLAB using Image Acquisition Toolbox videoinput.

Official
Intermediate
NzettodessNzettodess

llava

Runs vision-language conversations and visual question answering with LLaVA models.

Community
Intermediate
spideynolovespideynolove
2

llm-evaluation

Evaluate LLM outputs with automated metrics and human evaluation checklists.

Community
Advanced
arsityarsity

blip-2-vision-language

Generate captions, answer visual questions, and retrieve images with BLIP-2.

Community
Advanced