High-throughput data processing
Throughput is a pipeline problem: batching, backpressure, idempotency, and a store that can accept the write pattern you actually have.
Personal experience
High-throughput paths in recent platform work are Python workers behind Redis Streams, with Laravel owning the domain write. Earlier backend work at TRIKO used Redis, Memcached, Supervisor, and Google Cloud for hot paths. Specific throughput numbers are not published (TODO_METRIC).
Technical perspective
Measure the unit of work. Rows per second, events per second, and HTTP requests per second are different machines. A pipeline that shines on sequential inserts will collapse on random primary-key updates.
Batch at the edges. Single-row writes over the network are how CPUs wait. Backpressure must travel upstream; unbounded queues are a way to turn a slow database into a memory outage.
Idempotency is not optional once you retry. At-least-once delivery is the default in most brokers. Design the write as “apply this fact if not already applied,” not “increment this counter again.”
Architecture
Ingest, buffer, workers, store
Ingest
Queue / buffer
Workers
Cache
Database
Technical perspective. Caches and supervisors are part of the path, not decorations.