Scaling Security Insights: how we achieved a 10x increase in global scanning capacity
Cloudflare's Security Insights system has achieved a 10x throughput increase, processing over 120 scans per second by optimizing Kafka consumers, Postgres queries, and APIs without additional hardware.
MAIN POINTS
- Cloudflare Security Insights processes over 120 scans per second.
- Achieved a 10x increase in throughput.
- Optimization involved Kafka consumers, Postgres queries, and APIs.
- Scaling was accomplished without adding new hardware.
TAKEAWAYS
- Efficient system optimization can significantly increase processing capacity.
- Hardware additions are not always necessary for scaling.
- Cloudflare's approach demonstrates effective resource management.
- Frequent insights are now available to all Cloudflare customers.