database-migrations-migration-observability

Instrument migrations with Prometheus histograms for duration, counters for documents processed, and error counters labeled by version and error type. Expose these metrics through a registry and scrape them during migration execution to track progress in real time.

Deploy a Debezium connector via the Kafka Connect REST API with source database credentials and the pgoutput plugin for PostgreSQL. Consume change events from Kafka topics, then apply them to the target while tracking consumer lag and replication lag metrics.

Yes, use Prometheus gauges to record replication lag in seconds per source and target table pair. Compare CDC event timestamps against processing times to compute lag, and alert when it exceeds configured thresholds.

Yes, the MongoDB migration wrapper wraps each migration in a session transaction while recording duration, document counts, and errors to Prometheus. Failed migrations increment error counters and log details through Winston for troubleshooting.

Compare current throughput against expected rows per second and flag low-throughput warnings when it drops below 50 percent. Trigger critical alerts when the error rate exceeds one percent, then route notifications through Slack or email.