model-pick

Selects current LLM candidates via live registry queries, benchmarks, and pricing research.

Updated Jul 14, 2026
One-click install
npx skills add https://github.com/raian-pollock/operator-skills --skill model-pick-raian-pollock
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: model-pick
Source: https://github.com/raian-pollock/operator-skills/tree/main/model-pick
Command: npx skills add https://github.com/raian-pollock/operator-skills --skill model-pick-raian-pollock

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? AI agents default to whichever LLM was best at their training cutoff, silently missing newer, cheaper, or better models released since. This Skill forces a 5-10 minute fresh research pass before any code change that adds or modifies an external-LLM API call. ## Core Features & Use Cases - Live registry refresh: Pulls current models and pricing from the OpenRouter models endpoint and mechanically selects the latest releases per vendor family instead of relying on memory. - Task-type benchmark mapping: Classifies the task (agentic coding, long context, multilingual, etc.) and looks up the benchmarks most predictive for it, with per-locale slicing rules for multi-language deployments. - Channel and tier cost analysis: Enumerates direct API, OpenRouter, BYOK credit pools, and batch/flex service tiers, computing gross-cost-equivalents so "free" channels don't silently exhaust credit pools. - Use Case: Before writing a new script that calls the Gemini API, run the pick flow to discover a model released last week offers equivalent quality at half the price, then log the verdict to a JSONL pick log. ## Quick Start Before adding any external LLM call to my code, run the model-pick flow to refresh the model registry, compare 3-5 candidates across cost tiers, and log the chosen model with rationale.

Frequently Asked Questions about model-pick

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I choose which LLM to use for a new feature?▼

Classify the task by primary lens (quality, cost, speed, or capability), refresh a live model registry from the OpenRouter models endpoint, then compare 3-5 candidates across ultra-cheap, mid-range, and premium tiers using Arena ELO and task-specific benchmarks before logging a verdict.

What benchmarks should I check when picking an LLM for coding tasks?▼

For agentic coding use SWE-Bench Verified, Terminal-Bench, and Aider Polyglot; for code completion use HumanEval and MBPP. Cross-reference with LMSYS Arena ELO to catch models that score high on benchmarks they were likely trained on.

Should I wait for an upcoming model release or ship with current models?▼

Wait only if a release is vendor-confirmed or above roughly 40% probability within 4 weeks, the current frontier shows real quality headroom on your task, and nothing else is blocked. Otherwise ship with the current frontier and re-evaluate when the release lands.

Does model selection matter for multilingual or localized output?▼

Yes. Models that win on average can regress 4-6 points on minority locales, hidden by blended averages. Slice evaluation results per locale and flag any model dropping more than 3 points from its best-locale score on any minority language.

Why is a BYOK or free-tier model not actually free at scale?▼

BYOK channels draw from finite credit pools, so always compute the gross-cost-equivalent at sticker rate times actual volume. A run showing $0 marginal cost can still exhaust a credit pool mid-batch and silently spill into paid metered billing.

When is a lightweight model pick not enough?▼

Escalate to a full production bake-off when the batch is 50 or more items, user-facing, and has a high quality bar. The lightweight flow is research-only and does not substitute for scored evaluation against real production data.