clip

Compute image-text embeddings with CLIP for zero-shot classification and cross-modal retrieval.

13.0k|930|Updated Nov 3, 2025
One-click install
npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill clip-orchestra-research
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: clip
Source: https://github.com/Orchestra-Research/AI-research-SKILLs/tree/main/18-multimodal/clip
Command: npx skills add https://github.com/Orchestra-Research/AI-research-SKILLs --skill clip-orchestra-research

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires transformers, torch, pillow, and includes references (resource) components.

What problem does it solve? Classifying images or searching image collections normally requires labeled training data and model fine-tuning. CLIP removes that requirement by matching images to natural-language descriptions, enabling zero-shot classification, semantic image search, and content moderation out of the box. ## Core Features & Use Cases - Zero-Shot Image Classification: Classify images into arbitrary text categories without any training data, using models like ViT-B/32. - Semantic Image Search & Retrieval: Encode images and text into a shared embedding space for text-to-image and image-to-text search, with optional vector database integration (Chroma/FAISS). - Content Moderation: Score images against safety categories such as NSFW or violent content using softmax probabilities. - Use Case: Index a folder of product photos with CLIP embeddings, then let users search the catalog with queries like "a red sneaker on white background" and return the top matching images. ## Quick Start Use the clip skill to classify the attached image against the labels "a dog", "a cat", and "a car" and show the probability for each.

Frequently Asked Questions about clip

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I do zero-shot image classification with CLIP?▼

Load a model with clip.load("ViT-B/32"), preprocess the image, tokenize candidate text labels, then call model(image, text) and apply softmax to the logits to get per-label probabilities. No training data is required.

How to build semantic image search with CLIP embeddings?▼

Encode each image with model.encode_image, normalize the embeddings, and store them. Encode a text query the same way, then rank images by cosine similarity. Embeddings can also be stored in vector databases like Chroma or FAISS.

CLIP vs BLIP-2 vs LLaVA for vision-language tasks?▼

CLIP is best for zero-shot classification, image-text matching, and retrieval. BLIP-2 produces better image captions, while LLaVA supports vision-language chat. For segmentation, use Segment Anything instead.

Which CLIP model should I use: RN50, ViT-B/32, or ViT-L/14?▼

ViT-B/32 is the recommended default, balancing speed and quality at 151M parameters. RN50 is fastest, while ViT-L/14 (428M parameters) gives the best quality but is the slowest.

What are the limitations of CLIP for image understanding?▼

CLIP works best for broad categories, not fine-grained tasks. It produces no bounding boxes, has weak spatial and counting understanding, requires descriptive text labels, and may carry biases from its web training data.