skills.rest
timchap
timchapCOMMUNITY
Explore all skills by this author →
GitHub

observability-stack

Deploy and troubleshoot a Prometheus, Grafana, cAdvisor, and Phoenix monitoring stack on homelab hosts.

Updated Jul 10, 2026
Search Tags:#monitoring#observability#grafana#prometheus#ansible#homelab#cadvisor
One-click install
npx skills add https://github.com/timchap/dot-hermes --skill observability-stack-timchap
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: observability-stack
Source: https://github.com/timchap/dot-hermes/tree/main/skills/devops/observability-stack
Command: npx skills add https://github.com/timchap/dot-hermes --skill observability-stack-timchap

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve? Running a homelab monitoring stack across multiple hosts involves scattered configuration, subtle platform pitfalls (Docker 29 breaking cAdvisor, Raspberry Pi 5 disabling memory cgroups), and tedious dashboard authoring. This Skill consolidates deployment commands, PromQL patterns, and verified fixes for the full observability stack on pi-services-0. ## Core Features & Use Cases - Stack Deployment: Ansible-based deployment of Prometheus, Grafana, cAdvisor, node exporters, Phoenix tracing, and Lemonade metrics scraping across homelab hosts. - Dashboard Authoring: Conventions and ready-to-use PromQL queries for Grafana dashboards covering CPU, memory, disk, containers, GPU, and LLM token throughput. - Pitfall Resolution: Documented fixes for cAdvisor breakage under Docker 29 containerd-snapshotter and Raspberry Pi 5 cgroup memory disable, with verification steps. - Use Case: Container memory metrics show zero in Grafana after a Docker upgrade — follow the documented containerd-snapshotter fix, redeploy via Ansible, and verify the cAdvisor Docker factory registers successfully. ## Quick Start Ask the agent to deploy the observability stack to pi-services-0 and verify that all Prometheus scrape targets are up.

Frequently Asked Questions about observability-stack

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I deploy Prometheus and Grafana on a Raspberry Pi homelab?▼

Run the Ansible playbook with the observability tag: ansible-playbook playbooks/pi-services.yml --limit pi-services-0 --tags observability. This provisions Prometheus, Grafana, cAdvisor, and Phoenix via Docker Compose with preconfigured scrape targets and dashboards.

Why are container memory metrics zero in Grafana with cAdvisor?▼

Docker 29+ enables containerd-snapshotter by default, which breaks cAdvisor's Docker factory because the overlay2 layerdb metadata is missing. Disable containerd-snapshotter in /etc/docker/daemon.json and use cAdvisor v0.53.0 or newer.

How do I fix Docker mem_limit warnings on Raspberry Pi 5?▼

Raspberry Pi 5 firmware injects cgroup_disable=memory into the kernel command line. Append cgroup_enable=memory to /boot/firmware/cmdline.txt and reboot; the kernel processes arguments left-to-right so the last setting wins.

What PromQL query shows CPU usage per host in Grafana?▼

Use 100 - (avg by (host) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) to compute per-host CPU usage from node exporter metrics. Similar documented queries cover memory, disk, container memory, GPU utilization, and token throughput.

How do I monitor LLM token throughput with Prometheus?▼

Lemonade 10.9.0+ exposes metrics at port 13305 including lemonade_output_tokens_total and lemonade_model_tokens_per_second. Query sum by (model_name) (rate(lemonade_output_tokens_total[5m])) for per-model token throughput.

Why is a Prometheus scrape target down and how do I diagnose it?▼

Check the target's service: verify the node exporter container runs on the target host, inspect docker logs cadvisor on pi-services-0, or confirm Lemonade is reachable on port 13305 via the tailnet. Query the Prometheus targets API to identify which job is failing.

Related Skills

Explore More Software Engineering Skills→
logo-solutionslogo-solutions

nas-logo

Provision an Apple Silicon Mac Mini as an Immich NAS using Ansible roles.

Official
Advanced
rayvoidxrayvoidx

monitoring

Automate end-to-end monitoring of AI workflows with Langfuse, Prometheus, and Grafana.

Community
Advanced
prototypdigitalprototypdigital

infrastructure-drift-check

Verify infrastructure-as-code consistency with deployed Terraform, Ansible, Kubernetes, and Docker Compose configurations.

Official
Advanced
chflechfle
3

linux-monitoring-setup

Generate monitoring stack configurations with Prometheus, Grafana, Loki, and Alertmanager.

Community
Advanced