Telemetry for half a million devices, in real time

We broke the monolith into services drawn along the natural seams of the workload, each a container on ECS/EKS that scales to its own load. An ingestion service terminates connections from the half-million-device fleet and does nothing else — validate, normalise, publish. It writes onto a Kafka event bus (MSK), which decouples the firehose of inbound telemetry from everything downstream and absorbs spikes as buffered backlog rather than dropped messages. A processing-and-alerting service consumes that stream, computes rolling aggregates, and raises threshold alerts; a separate query service owns the read path and serves operator dashboards from pre-rolled summaries. Each writes to a time-series store sized for its own access pattern. Because ingestion and query no longer share a machine, a telemetry surge scales the ingestion fleet horizontally without ever touching dashboard latency — and every service is deployed, versioned, and scaled independently.

The platform now absorbs over a million messages a minute across five hundred thousand devices, and each service rides its own autoscaler — the ingestion fleet expands with the surge while the query service holds steady. Operator dashboards render in under a second during peak load, because the read path no longer competes with ingestion for capacity. And cost per device fell ninety-one percent: instead of one over-provisioned cluster paying for the sum of every peak, each right-sized service scales up only when its own load demands it and back down the moment it passes.