skills.rest
KxSystems
KxSystemsOFFICIAL
Explore all skills by this author →
Website𝕏@kxsystemsGitHub

run-nyse-taq-benchmarks

Runs the KX NYSE TAQ benchmark suite across in-memory query engines and KDB-X table formats.

13|7|Updated Jun 16, 2026
Search Tags:#benchmark#performance testing#parquet#duckdb#query engines#kdb-x#nyse taq
One-click install
npx skills add https://github.com/KxSystems/NYSETAQBenchmarks --skill run-nyse-taq-benchmarks-kxsystems
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: run-nyse-taq-benchmarks
Source: https://github.com/KxSystems/NYSETAQBenchmarks/tree/main/.claude/skills/run-nyse-taq-benchmarks
Command: npx skills add https://github.com/KxSystems/NYSETAQBenchmarks --skill run-nyse-taq-benchmarks-kxsystems

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve? Running the KX NYSE TAQ benchmark suite involves a strict four-step pipeline (selecting a data size, downloading PSV files, generating kdb+/Parquet databases, and executing benchmark scripts) where skipping or misordering steps causes confusing failures. This Skill guides the workflow end-to-end, including reusing already-generated data to avoid repeating slow downloads. ## Core Features & Use Cases - Guided four-step workflow: Walks through SIZE selection, PSV download via getPSVs.sh, kdb+/Parquet database generation via generateDB.sh, and benchmark execution via queryEngines.sh or kdbAttributes.sh. - Engine and format mapping: Matches engine subsets (kdb, kdbxsql, pykx, duckdb, chdb, polars, pandas) to the required database format so only the necessary binaries are generated. - Data reuse detection: Checks ${NYSEBENCHMARKDIR}/${SIZE} for existing kdb and Parquet directories so users can re-run queries with different thread counts or query filters without regenerating data. - Results interpretation: Explains the results.psv columns (run times, memory, status, threadcount) and troubleshooting for common failures like missing submodules or OOM on Community Edition. - Use Case: A performance engineer wants to compare DuckDB and KDB-X query latency on TAQ data at medium size with 4 and 16 threads; the Skill verifies prerequisites, generates both database formats, runs the benchmark, and explains the warm-run timing columns. ## Quick Start Ask the assistant to run the NYSE TAQ benchmark at small size comparing KDB-X and DuckDB with 4 and 16 threads, reusing any data already on disk.

Frequently Asked Questions about run-nyse-taq-benchmarks

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I run the KX NYSE TAQ benchmark suite?▼

Run four steps in order: export a SIZE (tiny through full), download PSV files with getPSVs.sh, generate kdb+ and/or Parquet databases with generateDB.sh, then execute queryEngines.sh or kdbAttributes.sh with --db-dir, --param-dir, --datadate, and --threads. Results land in a results.psv file under --result-dir.

How do I benchmark DuckDB vs KDB-X on NYSE TAQ data?▼

Use queryEngines.sh with --engines kdb,duckdb to compare KDB-X and DuckDB on the same queries. KDB-X reads the kdb+ database while DuckDB reads Hive-partitioned Parquet, so generate both formats in Step 3 before running.

Can I re-run TAQ benchmark queries without re-downloading data?▼

Yes, if ${NYSEBENCHMARKDIR}/${SIZE}/kdb and parquet/rowgroup already exist, skip the download and generation steps and go straight to the benchmark script. Confirm SIZE and DATADATE match the existing data, then re-run with new thread counts or an --idx query filter.

What data size should I use for the NYSE TAQ benchmark?▼

Use tiny for pipeline testing or KDB-X Community Edition (about 1 GB memory), small for quick familiarization, and medium or larger for realistic results. Note that tiny and small are not representative of real client data volumes.

Why does the TAQ benchmark fail with 'No result PSV files found'?▼

This error means every engine run failed, usually because the database is missing or --db-dir points to the wrong path. Verify Step 3 produced ${SIZE}/kdb and/or ${SIZE}/parquet/rowgroup and pass the per-size directory, not the format subdirectory.

Does the TAQ benchmark work with KDB-X Community Edition?▼

Yes, but Community Edition has a memory cap, so only the tiny size fits reliably; medium and larger sizes will run out of memory. Use small or tiny, or switch to a non-Community KDB-X license for bigger datasets.

Related Skills

Explore More Data & Analytics Skills→
NVIDIANVIDIA
3.2k

i4h-workflow-setup

Verifies host requirements and runs setup.sh to bootstrap the Isaac for Healthcare agentic workflow.

Official
Intermediate
Proxy2021Proxy2021
5

performance_dashboard

Consolidate KPI data from multiple sources into a dashboard with trend charts and benchmarks.

Community
Advanced
sirius-dbsirius-db
1.0k

tpcds-benchmark

Execute TPC-DS benchmarks across Legacy Sirius, Super Sirius, and DuckDB CPU engines.

Official
Advanced
clawdataclawdata
24

duckdb

Query local DuckDB warehouses with SQL and ingest CSV, Parquet, and JSON files.

Official
Intermediate