prod-auth-triage

Diagnose 401 and token audience failures on AWS EKS caused by recreated Entra app registrations.

Updated Apr 16, 2025
One-click install
npx skills add https://github.com/damoke012/eks_code --skill prod-auth-triage-damoke012
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: prod-auth-triage
Source: https://github.com/damoke012/eks_code/tree/main/.claude/skills/prod-auth-triage
Command: npx skills add https://github.com/damoke012/eks_code --skill prod-auth-triage-damoke012

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Production APIs suddenly return blanket 401 errors while pods stay healthy, and teams waste hours redeploying services or rolling back Kubernetes when the real cause is a recreated Entra app registration that invalidated every consumer's client ID and role assignment. ## Core Features & Use Cases - Root-cause fingerprinting: Compare AWS Secrets Manager CreatedDate vs LastChangedDate and CloudTrail DeleteSecret/CreateSecret events to prove an app registration was destroyed and rebuilt. - Fleet-wide consumer mapping: Sweep ConfigMaps and Secrets Manager for stale client ID GUIDs across every API and SPA, not just the reported one. - Pre-fix validation: Acquire a client_credentials token to distinguish AADSTS500011 (dead registration) from AADSTS501051 (missing app-role assignment) before changing anything. - Use Case: After an orders-api outage with 100% 401s, use this Skill to confirm the registration was recreated, find three silently broken consumer APIs, and sequence dependency-ordered releases that restore access. ## Quick Start Triage why the orders-api on the prod EKS cluster returns 401 on every request even though the pods are healthy.

Frequently Asked Questions about prod-auth-triage

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I troubleshoot 401 errors on a Kubernetes API when pods are healthy?▼

Check the istio-proxy sidecar access logs, not app logs, since most apps log no HTTP requests. Then verify whether the Entra app registration was recreated by comparing Secrets Manager CreatedDate and LastChangedDate, which indicates a new client ID broke all consumers.

What causes IDX10214 audience validation failed in Azure AD?▼

IDX10214 occurs when the token's audience does not match the API's expected client ID, typically because the API's app registration was destroyed and recreated with a new client ID. Consumers still hold the stale ID, so their tokens fail validation.

How do I tell AADSTS500011 apart from AADSTS501051?▼

Acquire a client_credentials token against the API's scope. AADSTS500011 means the resource principal was deleted and the consumer points at a dead identity; AADSTS501051 means the scope is right but the app-role assignment was destroyed, which a config change cannot fix.

Why is my SPA getting 401s but the client_credentials test shows AADSTS501051?▼

SPAs use delegated scopes via authorization-code plus PKCE and are never assigned app roles, so AADSTS501051 on a client_credentials test is expected and proves nothing. Instead compare the ConfigMap VITE_AUTH_CLIENT_ID against the current Entra app registration.

Should I redeploy or restart a service to fix auth errors?▼

Redeploy only when the config itself is wrong, such as a stale client ID in a ConfigMap; use rollout restart when config is right but the process is stuck. Never do a clean release, since it rebuilds the app registration and breaks every consumer.

Why does kubectl rollback not fix authentication failures?▼

Auth values flow from Secrets Manager through ESO into Secrets and envFrom, so every ReplicaSet reads the same current Secret. Rolling back Kubernetes objects changes nothing because the stale credential lives outside the cluster state.