dataflow-staging-experiment

Deploy ephemeral staging Dataflow jobs that each create their own subscription on the prod Pub/Sub topic, receiving a full copy of traffic and writing to a staging Spanner instance. Use partition flags to slice entities and disjoint shard bands so concurrent arms never write the same rows.

Set a per-arm transaction tag and request tags, then query SPANNER_SYS TXN_STATS for abort rate and commit latency, and QUERY_STATS for per-statement CPU. Use the latest single TOP_10MINUTE interval and report lock-wait at table level only, since LOCK_STATS does not key by tag.

Yes. The deploy-dataflow-staging GitHub Actions workflow runs the deploy script under a workload identity, triggered via workflow_dispatch or repository_dispatch. Bots with Contents:write but not Actions:write should use the repository_dispatch API path with client_payload fields.

Early data is cold-start noise, so discard the first several minutes after launch. Also avoid summing across overlapping TOP_10MINUTE buckets, and if you relaunched an arm with the same tag, filter intervals to after the relaunch or old stats blend into the new tag.

Partition index selects which entities an arm processes via consistent hashing, while shard band only controls which rows those entities' writes rotate across. The band does not filter traffic, so arms on the same partition with disjoint bands see identical traffic without row contention.

Pass a seek timestamp within the topic's 6-hour retention window when deploying the staging arm. The ephemeral subscription can seek to before its own creation, replaying hours of prod traffic as an immediate backlog for throughput or serial comparison tests.