Event reporting at 15M+ events a day
Replaced a legacy reporter with a queue that holds every event until the parent group acknowledges it, sends it exactly once, and scales from an afternoon trickle to a full day's backlog within minutes.
Context
A national operator reports every operational event to the parent group it belongs to. Each night the group measures the share that arrived on time, and the result feeds corporate KPIs that people are paid on. The legacy reporter wrote files in a format from another decade, kept no record of what had happened to an individual event, and could not take on the new JSON schema, the new API and the new event types the group was introducing. We replaced it.
Constraints
- Every event sent, and sent once. An event is held until the group acknowledges it; nothing is dropped and nothing is sent twice.
- Load arrives in waves: most of a day’s events before noon, a trickle after. After an outage, a whole day’s worth lands within minutes and still has to clear before the nightly cut-off.
- The group’s endpoint can answer with a 500. When it does, the burden of proof is on the sender, so every attempt had to be on record.
- The legacy producers could not be rewritten. The only change allowed upstream was where their files went.
What we built
The legacy system kept writing its files; the one change was dropping them into a Cloud Storage bucket. A Cloud Function reads each file in chunks, maps the records to the new schema and writes them to Spanner, logging the row count at the head of the file against the count saved, so a loss is caught at ingest rather than discovered in next month’s KPI.
Spanner holds the queue. Under the morning wave it developed hotspots, so the queue is sharded and each send worker owns a shard. Workers are Cloud Functions fanned out through Pub/Sub, and their number follows queue depth: a handful through the afternoon, hundreds after an outage. Every action on an event is written next to it, each attempt, each response from the group, each retry, so the history of any single event can be read in one query. On acknowledgement the event moves to BigQuery as the archive and leaves Spanner.
Queue depth against send rate is the metric that matters, and it pages: when the queue grows faster than the workers drain it, the engineer on call gets an automated phone call, well before the cut-off is at risk.
Outcome
15M+ outbound and up to 3M inbound events a day, each one accounted for. For the first time the business could see where and why events were going missing, and most of it turned out to be upstream: customers not supplying the data the group required, so the events were rejected on arrival. That evidence moved the KPI conversation from the pipeline to the source of the data. Designed and led by Jakub, built with a team of three in a couple of months, and still in daily use.
Stack
Google Cloud: Spanner, Cloud Functions, Pub/Sub, Cloud Storage, BigQuery, Cloud Logging and Cloud Monitoring.