databricks-spark-structured-streaming

Use spark.readStream with the kafka format, parse the JSON value column with from_json, and write with writeStream in delta format using append mode. Set a persistent checkpointLocation on a Unity Catalog volume and a processingTime trigger such as 30 seconds.

Real-Time Mode is a Databricks execution mode that processes records continuously instead of micro-batches, achieving sub-second end-to-end latency. It requires DBR 16.4 LTS or later, Classic compute, fixed-size clusters, Photon off, and outputMode update with a realTime trigger.

Yes, use foreachBatch to write one stream to multiple Delta tables or Kafka topics with a single checkpoint. Each write must set txnVersion to the batch_id and a unique txnAppId so retries stay idempotent, since foreachBatch writes are not atomic across sinks.

Checkpoint corruption usually comes from file system issues, manual deletion, or sharing one checkpoint between streams. Delete the checkpoint folder, restart with startingOffsets set to earliest, and rely on idempotent Delta writes to deduplicate reprocessed data.

No, streaming workloads require fixed-size clusters because autoscaling disrupts stateful processing and Real-Time Mode explicitly disables it. Size the cluster so total worker vCPUs meet or exceed the sum of partitions across every pipeline stage.

On DBR 18.3 or later use the native format postgresql sink with outputMode update, which handles batching, retries, and workspace-managed authentication. On older runtimes use a manual foreach sink with INSERT ON CONFLICT upserts and password authentication from a secret scope.