Case study
Nothing lost: a trading chat run like an SRE team
Traders chat in real time, and the law says every message must be archived. So the system makes two promises: fast, and nothing lost. I built a miniature of that platform on Kubernetes, then spent as much time trying to break it.
- p99 latency
- 11 ms
- SLO: < 200 ms
- Time to detect
- 161 s
- bad release
- Time to recover
- 79 s
- one rollback
- Messages lost
- 0
- in every scenario
Build it: fast, and nothing lost
- A message is written to a durable log (a Redis Stream on a persistent volume) before the trader gets “202 Accepted”.
- Three API replicas, spread over two nodes, push it to every connected trader over WebSockets.
- An archiver reads the same log and acknowledges a message only once it is on disk: a crash can duplicate, never lose.
Ship it: every commit is deployed before it is accepted
The pipeline does more than compile: it deploys the commit to a throwaway Kubernetes cluster, simulates 20 traders, and rejects the release if the SLO is missed or a single message isn’t archived.
- Secrets
- Tests
- Build
- CVE scan
- Deploy + load
- SLO verdict
In production, Argo CD keeps the cluster equal to Git: a manual change is reverted in under 5 s, and a rollback becomes a reviewed, traceable “git revert”.
Break it on purpose
- Kill an API pod under load Traffic routes around it 0 errors · 2,399 / 2,399 archived
- Stop the archiver Critical alert, then catch-up 1,249 messages drained in < 15 s
- Ship a release 300 ms slower Alert, one-command rollback 161 s to detect · 79 s to recover
- Ship a release that runs out of memory The rollout blocks itself no user impact
- Ask an AI assistant if all is well It reads alerts and runbooks, changes nothing every call audited
What breaking it taught me
Seven real problems a green test suite didn’t see:
- A Redis restart left a dead connection: reconnect and retry.
- The compliance metric cried wolf on every restart: measure at the source.
- An upgrade silently undid a manual fix: if it isn’t in Git, it isn’t done.
- An alert kept paging 5 minutes after the fix: shorter window.
- Rollouts cut in-flight requests: a delay before shutdown, and 0 errors.
- Monitoring crashed during the incident it was showing: sized from observation.
- A private key slipped into a commit: caught before the push, then detected automatically in pre-commit and CI.
Built with an AI coding assistant (Claude Code), verifying every decision by running it: every number on this page was measured, not estimated.