skills.rest
Leiisawesome
LeiisawesomeCOMMUNITY
Explore all skills by this author →
GitHub

data-engineering

Ingest, validate, and store NBBO and trade data from Massive into an immutable event log.

Updated Mar 2, 2026
Search Tags:#validation#data-integrity#market-data#event-log#data-ingestion#replay#nbbo
One-click install
npx skills add https://github.com/Leiisawesome/feelies --skill data-engineering-leiisawesome
Or copy as Structured Prompt for Agent▼
Please help me install this Agent Skill.
Skill: data-engineering
Source: https://github.com/Leiisawesome/feelies/tree/main/.cursor/skills/data-engineering
Command: npx skills add https://github.com/Leiisawesome/feelies --skill data-engineering-leiisawesome

SYSTEM DOCUMENTATION & REQUIREMENTS

What problem does it solve?

The Data Engineering skill defines standards for high-fidelity ingestion, validation, and storage of L1 NBBO and trade data from Massive. It guides backfill, gap-detection, deduplication, and recovery protocols to ensure data integrity across pipelines.

Core Features & Use Cases

  • Establish immutable raw logs and typed downstream schemas to enforce contract boundaries.
  • Enable deterministic replay and provenance through per-event metadata and DataHealth state tracking.
  • Support historical backfill, live streaming, and replay paths with a unified normalizer boundary.

Quick Start

Configure Massive REST/WS sources and start the MarketDataNormalizer to begin processing NBBO and Trade events.

Frequently Asked Questions about data-engineering

High-intent search queries and answers about installing and using this skill.

FAQPage Schema
How do I build a market data pipeline for deterministic replay and provenance?▼

You build a market data pipeline by ingesting L1 NBBO and Trades into an immutable event log, applying a normalization layer and per-event metadata to guarantee deterministic replay and full provenance.

How does an immutable event log ensure data integrity for historical backfill?▼

An immutable event log ensures data integrity by establishing raw logs and typed downstream schemas, enforcing contract boundaries while running gap-detection and deduplication protocols during historical backfill.

What's the best way to validate and normalize live NBBO and trade data?▼

The best way to validate live NBBO and trade data is configuring Massive REST and WebSocket endpoints, then starting a MarketDataNormalizer to process events through a unified normalizer boundary.

How do I handle gap-detection and recovery protocols for market data ingestion?▼

Gap-detection and recovery protocols handle market data ingestion by tracking DataHealth state and applying per-event metadata to identify missing sequences and automatically recover data across streaming paths.

Does this data ingestion approach work with both real-time streaming and historical backfill?▼

Yes, this data ingestion approach works with both real-time streaming and historical backfill by applying a unified normalizer boundary to process live events and replay paths consistently.

Why does deterministic replay require typed event models and a normalization layer?▼

Deterministic replay requires typed event models and a normalization layer to enforce contract boundaries, track DataHealth state, and guarantee that replayed market data maintains full provenance and data health.

Related Skills

Explore More Data & Analytics Skills→
erwinknerwinkn

massive

Access Massive.com REST endpoints, WebSocket streams, and market data schemas.

Community
Intermediate
ekremmkasapekremmkasap

backtesting-frameworks

Implement and validate backtesting frameworks for quantitative trading strategies.

Community
Advanced
doanchienthangdevdoanchienthangdev
4

Stream Processing

Process unbounded data streams with Kafka Streams, Apache Flink, and ksqlDB.

Community
Advanced
leary-pokenleary-poken
1

databricks-zerobus-ingest

Ingests data into Delta tables via Zerobus Ingest using gRPC.

Community
Advanced