langfuse-feedback-analyst

What problem does it solve?

Manually analyzing user feedback from Langfuse annotation queues and translating it into specific configuration changes for template.yaml, style.yaml, or tools.yaml is a complex, time-consuming, and error-prone process. This skill automates that analysis and recommendation, ensuring your AI system continuously improves based on real-world input.

Core Features & Use Cases

• Automated Feedback Retrieval: Pulls scores and comments from Langfuse annotation queues, providing a comprehensive view of user sentiment.
• Pattern Analysis: Identifies recurring issues and groups them by affected configuration file and topic, highlighting critical areas for improvement.
• Surgical Recommendations: Generates precise YAML edits with before/after diffs and clear rationales, targeting only fields actually used by the generation architecture for maximum impact.
• Use Case: If users consistently report a "too formal" tone in AI-generated content, this skill will analyze the feedback, identify style.yaml's signatures.tone_rubric.grader_prompt as the likely culprit, and suggest an exact YAML modification to make the tone more approachable.

Quick Start

Retrieve all completed annotations from Langfuse queue cm42abc123xyz. Analyze the feedback patterns and generate specific recommendations for template.yaml, style.yaml, and tools.yaml to address the identified quality issues.