skills.rest
arsity
arsityCOMMUNITY
Explore all skills by this author →
WebsiteGitHub

hqq-quantization

Quantize LLM weights without calibration data across 8/4/3/2/1-bit configurations.

Updated Mar 16, 2026
Search Tags:#llm#pytorch#quantization#inference#backends#memory-optimization#calibration-free
One-click install
npx skills add https://github.com/arsity/scholar-tools --skill hqq-quantization-arsity
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: hqq-quantization
Source: https://github.com/arsity/scholar-tools/tree/main/vendor/ai-research-skills/10-optimization/hqq
Command: npx skills add https://github.com/arsity/scholar-tools --skill hqq-quantization-arsity

SYSTEM DOCUMENTATION & REQUIREMENTS

💡 This Skill includes references (resource) components.

What problem does it solve?

Half-Quadratic Quantization enables weight quantization for large language models without requiring calibration data, reducing memory footprint and speeding up inference.

Core Features & Use Cases

  • Calibration-free quantization across 8/4/3/2/1-bit configurations with configurable group sizes.
  • Multiple backends (PyTorch, TorchAO, BitBlas, Marlin) for flexible deployment.
  • Mixed-precision per-layer configurations and easy integration with HuggingFace Transformers.
  • Production-ready workflows with vLLM for efficient serving and deployment.
  • Clear paths for model compression and deployment across diverse hardware.

Quick Start

Install hqq and quantize a model with a 4-bit config using the default backend, then test with a sample prompt.

Frequently Asked Questions about hqq-quantization

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I quantize LLM weights without calibration data?▼

You can quantize LLM weights without calibration data using Half-Quadratic Quantization, which directly processes model weights to reduce memory footprint and accelerate inference without needing a calibration dataset.

Can I use mixed-precision quantization per layer for PyTorch models?▼

Yes, mixed-precision quantization supports per-layer configurations for PyTorch models, allowing you to assign different bit-widths across layers and integrate seamlessly with HuggingFace Transformers.

Does hqq quantization support vLLM for production inference?▼

Yes, hqq quantization supports production-ready workflows with vLLM for efficient model serving and deployment across diverse hardware configurations.

What bit-width configurations are available for calibration-free LLM quantization?▼

Calibration-free LLM quantization supports 8, 4, 3, 2, and 1-bit configurations with configurable group sizes to optimize memory usage and inference speed.

Which inference backends are compatible with Half-Quadratic Quantization?▼

Half-Quadratic Quantization is compatible with multiple backends including PyTorch, TorchAO, BitBlas, and Marlin, enabling flexible deployment across various hardware environments.

What do I need to install to start quantizing models with hqq?▼

You need Python, PyTorch, and the hqq library installed, along with optional dependencies for specific backends and HuggingFace Transformers integration to begin quantizing models.

Related Skills

Explore More Software Engineering Skills→
zhuangbiaoweizhuangbiaowei
2

hqq-quantization

Quantize LLMs to 8/4/3/2/1-bit precision without calibration using HQQ.

Community
Advanced
davpatel605-beepdavpatel605-beep

serving-llms-vllm

Deploy and optimize vLLM inference servers with OpenAI-compatible APIs.

Community
Advanced
tadod12tadod12

hqq-quantization

Quantize large language models without calibration data for HuggingFace Transformers or vLLM.

Community
Advanced
MesferAliMesferAli

hqq-quantization

Quantize Large Language Model weights to 4/3/2-bit precision without calibration data.

Community
Intermediate