tritonkit-local-mlx-vlm-provider

Implements and validates local mlx-swift-lm VLM provider integration for TritonKit grounding and model cache workflows.

1|Updated May 15, 2026
One-click install
npx skills add https://github.com/NeptuneKit/TritonKit --skill tritonkit-local-mlx-vlm-provider-neptunekit
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: tritonkit-local-mlx-vlm-provider
Source: https://github.com/NeptuneKit/TritonKit/tree/main/.agents/skills/tritonkit-local-mlx-vlm-provider
Command: npx skills add https://github.com/NeptuneKit/TritonKit --skill tritonkit-local-mlx-vlm-provider-neptunekit

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Integrating a local MLX Swift VLM provider into the TritonKit CLI requires strict separation between the main CLI and heavy ML dependencies, plus disciplined model cache, evidence, and grounding-failure handling that is easy to get wrong. ## Core Features & Use Cases - Helper Contract Enforcement: Delegates real MLX execution to an external triton-mlx-provider helper resolved via --helper, TRITON_MLX_HELPER, or TRITON_MLX_SWIFT_LM_HELPER, keeping mlx-swift-lm out of the main CLI package. - Model Cache Management: Covers triton vlm model list/inspect/preflight/download/prune/remove with idempotent ready entries, fail-closed missing-helper behavior, and explicit removal semantics. - Evidence-Gated Grounding: Ensures VLM grounding writes request, raw output, parsed point, transform, overlay, and policy artifacts, converting failures into vlm_grounding_failed recovery proposals instead of executing actions. - Use Case: When adding Qwen2-VL local smoke validation, use this Skill to build the helper, copy the Metal metallib runtime, run focused swift test --filter VLMMlxSwiftLM tests, and capture compact evidence without committing model weights. ## Quick Start Ask the agent to implement and validate a local mlx-swift-lm VLM grounding flow for the TritonKit CLI, including helper resolution, model cache preflight, and evidence artifacts for a fixture screenshot.

Frequently Asked Questions about tritonkit-local-mlx-vlm-provider

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run local VLM grounding with mlx-swift-lm in the Triton CLI?▼

Run `triton vlm ground --provider mlx-swift-lm --model <id> --image <fixture> --target <text>` with the helper set via `--helper` or `TRITON_MLX_HELPER`. The helper writes a raw point to stdout while Triton owns coordinate transform, overlay, and evidence artifacts.

How do I manage the local MLX model cache with triton vlm model commands?▼

Use `triton vlm model list`, `inspect`, `preflight`, `download`, `prune`, and `remove` with `--json`. The cache defaults to `~/.cache/triton/mlx-models`, ready entries return `already-ready`, prune never deletes ready models, and deletion requires explicit `remove`.

Why does MLX execution fail with a missing metallib error?▼

SwiftPM pulls MLX Swift/C++ sources but may not colocate `mlx.metallib` beside the helper binary. Rebuild and copy the Metal runtime with `Tools/TritonMLXProvider/Scripts/build-mlx-metallib.sh debug`, and do not vendor the generated metallib into git.

Can the main triton CLI link mlx-swift-lm or swift-transformers directly?▼

No. The main CLI must not depend on `mlx-swift-lm`, `swift-transformers`, or `swift-huggingface` directly, and model weights or caches must never be committed. Real execution is delegated to the external `Tools/TritonMLXProvider` helper package.

What happens when VLM grounding fails during a workspace run?▼

The runtime action provider is not called. Triton writes a VLM failure artifact, failed action evidence, and a `vlm_grounding_failed` recovery proposal, then pauses for evidence-backed repair instead of falling back silently.