flaky-smoke-tests

Diagnose intermittent VS Code Electron smoke-test failures from Azure DevOps pipeline runs.

Updated Sep 3, 2026
One-click install
npx skills add https://github.com/az0512124155azz-sys/CodeForge --skill flaky-smoke-tests-az0512124155azz-sys
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: flaky-smoke-tests
Source: https://github.com/az0512124155azz-sys/CodeForge/tree/main/.github/skills/flaky-smoke-tests
Command: npx skills add https://github.com/az0512124155azz-sys/CodeForge --skill flaky-smoke-tests-az0512124155azz-sys

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Intermittent Electron smoke-test failures in the Azure DevOps Flaky Smoke Tests pipeline (definition 700) are hard to investigate because iteration tasks use continueOnError, logs are shared across iterations, and artifacts can be overwritten or lost on cancellation. This Skill provides a systematic procedure to isolate the exact failed iteration, gather authoritative evidence, and trace the root cause. ## Core Features & Use Cases - Failed Iteration Discovery: Query the build timeline with Azure CLI to find iteration tasks with succeededWithIssues or failed results, and download exact per-task logs. - Artifact Correlation: Download platform log and crash artifacts, then correlate task timestamps with smoke-test-runner.log, mock-server requests, extension-host logs, screenshots, and Playwright traces. - Root Cause and Validation: Trace the introducing commit with git log and blame, then queue focused definition-700 validation runs with controlled iteration counts and platform selection. - Use Case: A chat smoke test fails intermittently on macOS. Use this Skill to pull the failed iteration's task log, confirm the mock server never received the scenario tag, identify the commit that introduced the dispatch race, and queue a six-iteration validation run to confirm the fix. ## Quick Start Ask the assistant to investigate the failed iterations of Azure DevOps build <BUILD_ID> from the Flaky Smoke Tests pipeline and identify the root cause.

Frequently Asked Questions about flaky-smoke-tests

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I find failed smoke test iterations in an Azure DevOps build?▼

Download the build timeline with az devops invoke using the build timeline resource, then filter records whose names contain Smoke test iteration. Treat tasks with result succeededWithIssues, failed, or error issues as failed iterations and record their task id, log id, and time range.

How do I download Azure Pipelines task logs with Azure CLI?▼

Use az devops invoke with the build logs resource, passing the buildId and the timeline record's logId. The response is JSON with a value array of log lines, which you can flatten to plain text with jq or PowerShell.

Why is the smoke test logs artifact missing after a canceled run?▼

Canceling a platform job can skip the artifact publish steps entirely, even when failed iteration task logs already exist. An absent artifact after cancellation is expected behavior, not evidence of a wrong artifact name.

Why does the smoke-test-runner.log not match the failed iteration?▼

The logs artifact is job-scoped, not iteration-scoped, so the runner log can contain multiple iterations and later iterations can overwrite files. Use the failed task's exact timestamps and test title to isolate the matching Test start and Test end interval.

How do I queue a focused flaky smoke test validation run?▼

Use az pipelines run with definition id 700, passing the branch, commit SHA, and an iterations parameter such as [1,2,3,4,5,6], plus platform flags like VSCODE_BUILD_MACOS. Six iterations suit quick checks; use the default 20 for rare failures.

When should I not just increase a smoke test timeout?▼

Increasing a timeout does not fix race conditions such as stale session DOM matches or duplicate in-flight requests. Wait for the actual state transition, use unique scenario markers, and verify the gesture, dispatch, request, and rendering chain instead.