skills.rest
flagos-ai
flagos-aiOFFICIAL
Explore all skills by this author →
WebsiteGitHub

transformers-test

Runs HuggingFace transformers tests on custom chips with triage, verification, and issue preview.

12|18|Updated Apr 7, 2026
Search Tags:#testing#PyTorch#github issues#HuggingFace#Transformers#triage#accelerator
One-click install
npx skills add https://github.com/flagos-ai/Torch-FL --skill transformers-test-flagos-ai
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: transformers-test
Source: https://github.com/flagos-ai/Torch-FL/tree/main/.claude/skills/transformers-test
Command: npx skills add https://github.com/flagos-ai/Torch-FL --skill transformers-test-flagos-ai

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill requires torch, transformers, accelerate, pytest, and includes scripts (resource) components.

What problem does it solve? Measuring whether real HuggingFace transformers models actually run on a custom accelerator (via the flagos device) is slow and error-prone: crashes poison the device, CPU fallbacks hide missing operators, and raw failures are not actionable for maintainers. This Skill automates the full measurement pipeline from test execution to verified, deduplicated GitHub issue previews. ## Core Features & Use Cases - Automated test sweeps: Run official HuggingFace architecture tests on the flagos device with resilient batch mode for crash recovery on unstable chips. - Triage and verification: Classify failures (OP_UNSUPPORTED, OP_CPU_FALLBACK, PRECISION, CRASH, etc.), isolate findings in fresh subprocesses, and measure CPU fallback per operator. - Safe mode for weak models: A validated wrapper prevents weaker AI models from installing packages, editing source files, or running unapproved commands. - Use Case: Run /transformers-test --model qwen3 --chip MUSA to execute the qwen3 architecture test suite on a MUSA accelerator, triage every failure to a root-cause operator, and generate GitHub issue previews requiring explicit per-finding approval before filing. ## Quick Start Ask the AI to run the transformers test for the bert model on your GCU chip using the transformers-test skill.

Frequently Asked Questions about transformers-test

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run HuggingFace transformers tests on a custom AI chip?▼

Invoke the skill with a model and chip label, for example /transformers-test --model bert --chip GCU. It injects the flagos device via TRANSFORMERS_TEST_DEVICE_SPEC, runs the official tests for that architecture, and reports triaged findings.

How do I test transformers models on unstable accelerators that crash?▼

Use resilient mode, which runs tests in batches with crash isolation, automatic device reset, and incremental result saving. Adjust --batch-size and --batch-timeout based on crash frequency and model size.

Does the skill file GitHub issues automatically?▼

No. The pipeline only generates issue previews after triage, verification, and deduplication. Every GitHub write requires explicit per-finding approval, and TEST_ERROR or ENVIRONMENT_ERROR results are never filed.

Why do some transformers tests skip on PrivateUse1 devices like flagos?▼

Flex Attention tests skip because torch.nn.attention.flex_attention requires a CUDA or ROCm Triton backend, and require_torch_gpu skips compare against cuda literally. These skips document upstream PyTorch limitations, not coverage gaps.

What does exit code 2 mean in the transformers test runner?▼

Exit code 2 means nothing was measured due to an environment, source, or preflight failure. It is never a coverage result and must not be summarized as tests passing; fix the environment and re-run.

When should I use safe mode for running transformers tests?▼

Use safe mode when the executing AI model is weak (e.g., Qwen-27B, Sonnet) or pass --safe explicitly. The safe wrapper blocks package installs, source edits, and unapproved commands, allowing only validated operations.

Related Skills

Explore More Software Engineering Skills→
bensonlee5bensonlee5

act-so101-pen

Automate pick-and-place 'pass the pen' tasks on a real SO-101 arm.

Community
Intermediate
anthropicsanthropics
9.1k

auto-updater

Compare pinned commit SHAs and apply community legal skill updates after diff review.

Official
Advanced
Brilly-BohyunBrilly-Bohyun

obliteratus

Remove refusal behaviors from open-weight LLMs using diff-in-means, SVD, and SAE decomposition.

Community
Advanced
WangWindowWangWindow

triton-npu-convert

Convert PyTorch operators into Triton Ascend NPU kernel-backed operators with validated correctness.

Community
Advanced