rust-sqlite-ingestor-low-ram

What problem does it solve?

This Skill addresses the challenge of ingesting massive JSONL log files into SQLite databases efficiently, particularly under strict memory constraints during live operation.

Core Features & Use Cases

• High-Throughput Ingestion: Optimized for ingesting huge JSONL logs at high speeds in historical mode.
• Low RAM Live Mode: Enforces a strict RSS budget in live mode using advanced parsing and SQLite tuning.
• Efficient Parsing: Utilizes byte-scan parsing instead of full JSON parsing to minimize overhead.
• Mode-Specific Tuning: Employs different SQLite PRAGMAs for historical (speed) and live (low-RAM) modes.
• Reliable Transitions: Supports robust historical-to-live transitions, including process restarts via exec().
• Use Case: Ingesting real-time event streams into a SQLite database while maintaining a minimal memory footprint for long-running applications.

Quick Start

Use the rust-sqlite-ingestor-low-ram skill to ingest the file 'events.jsonl' into 'events.db' in live mode.