LSM-backed DBs: fix fast writes and deceptive reads with targeted telemetry

Executive Summary

Many SREs and core database engineers at enterprises running LSM-based stores struggle with fast-write regimes that trigger compactions and “deceptive reads” — read latency spikes that originate from internal compaction, bloom-filter inefficiency, or cache thrashing rather than query patterns. These problems are widespread among the roughly 50,000 enterprises that allocate about $200K each per year to database and observability budgets, producing a $10B addressable market and frequent costly firefights and overprovisioning. You could build a focused telemetry and diagnostics platform that understands LSM internals: lightweight agents and OpenTelemetry-compatible collectors that emit per-memtable, SSTable, compaction, bloom-filter and I/O backpressure signals plus correlated distributed traces that tie writes and compactions to user-visible latency. Layered on top would be signature-based root-cause detection, compaction heatmaps, actionable runbooks and tuned sampling to keep runtime overhead low (targeting <1% CPU and <5% memory), with connectors into SIEM/APM and managed DB APIs for incremental adoption. This market is attractive now because LSM engines (RocksDB, Cassandra, Scylla, TiKV) are becoming dominant in cloud-native stacks, creating a concentrated set of failure modes, while teams increasingly prefer platform-native observability (OpenTelemetry) and managed services that make plug-and-play telemetry viable. The product could stand out through deep protocol-level signals and compact, high-signal telemetry that meaningfully shortens diagnosis time, but be realistic about challenges: variability across LSM implementations, limited visibility in closed managed offerings, and competition from generalist APM vendors—so go in with strong case studies and measurable MTTR reductions to win enterprise buy-in.