cluster-tooling

Manages k3s cluster operations, Zot registry, BuildStream builds, and GPU inference workloads.

11|9|Updated May 25, 2026
One-click install
npx skills add https://github.com/projectbluefin/lab --skill cluster-tooling-projectbluefin
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: cluster-tooling
Source: https://github.com/projectbluefin/lab/tree/main/docs/skills/cluster-tooling
Command: npx skills add https://github.com/projectbluefin/lab --skill cluster-tooling-projectbluefin

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Operating a two-node k3s homelab cluster involves many failure-prone areas: BuildStream distributed builds that get preempted, Zot registry sync-prefix changes that silently break image pulls, GPU memory misconfiguration on Strix Halo nodes, and wedged nodes that report Ready while pods hang. This Skill consolidates verified operational runbooks, measured hardware findings, and GitOps-safe procedures so cluster changes are made correctly the first time. ## Core Features & Use Cases - Cluster and registry operations: Covers kubectl/k3s workflows, Zot OCI registry sync-prefix changes with metric-based verification, external-secrets, and K8sGPT MCP-based cluster analysis. - BuildStream and Buildbarn distributed builds: Enforces USB4 admission gating, correct cache configuration (override-project-caches, Buildbarn frontend endpoints), PriorityClass-based preemption avoidance, and RECC pilot evidence handling. - Hardware and storage runbooks: Documents AMD GPU topology and GTT sizing on Strix Halo, NVMe/XFS migration procedures, FCOS memory-limit quirks, and wedged-node recovery without SSH. - Use Case: Before changing a Zot sync prefix, an operator follows the Skill to inventory served repositories via zot_repo_downloads_total metrics, applies the config-version bump through GitOps, and verifies every repository still reports non-zero download counters instead of trusting a misleading skopeo timeout. ## Quick Start Ask the assistant to check the cluster's USB4 link admission state and BuildBarn worker readiness before submitting a Dakota BuildStream build.

Frequently Asked Questions about cluster-tooling

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I safely change Zot registry sync prefixes in Kubernetes?▼

Inventory actually-served repositories via zot_repo_downloads_total metrics before editing, since the prefix list also filters on-demand pulls. Bump the config-version annotation in the same commit, then verify every pre-change repository still reports non-zero download counters after rollout.

How do I run BuildStream distributed builds with Buildbarn?▼

Point artifact writes at the Buildbarn frontend gRPC endpoint with push enabled, keep upstream caches as read-only fallbacks, and set override-project-caches to false for artifacts. Require a fresh USB4 link label and Ready workers on both nodes before admission.

Why does my BuildStream build pod get deleted on a small cluster?▼

Build pods are preempted when higher-priority VM pods need memory. Set the bst-build PriorityClass above lab-test-vm, keep the build semaphore at live worker capacity, and check kubectl events for Preempted reasons before retrying.

Can I use skopeo inspect to verify a Zot cache rollout?▼

No. A timeout only means Zot is syncing multi-GB blobs on first read, while a fast 404 means the prefix is rejected. Verify with zot_repo_downloads_total counters and the absence of error series instead.

Why does committing replicas: 0 deadlock my ArgoCD deployment?▼

A WaitForFirstConsumer local-path PVC never binds without a pod, so ArgoCD blocks on PVC health and the Deployment is never created. This only applies before first binding; once the PVC is Bound, committing replicas: 0 is a safe durable pause.

How do I recover a wedged Kubernetes node without SSH?▼

Compare the node lease freshness against stale status heartbeats, query the kubelet's own pod view via the API proxy, and force-delete orphaned pods only after confirming containers are gone. Cordon the node and patch Released local-path PVs to Retain until it returns.