kbavishi90
3 months ago
Curious about how you handle metrics with ephemeral labels? We have a bunch of serverless workloads emitting a lot of short-lived metrics, and this causes big cardinality spikes because the labels are ephemeral.
Do you support storing such high cardinality metrics in disk alone and not in the in-memory index?
kirankgollu
3 months ago
Our approach is that you need not worry about this. Oodle will be able to handle any scale including the ephemeral metrics.
Secondly, we only look at the unique time series in an hour when computing the billing. This should help us handle ephemeral metrics a lot better compared to many disk based data stores.
Finally, we do support high cardinality since the underlying datastore is columnar. Existing customers are sending on the order of few million cardinality per metric.
re: storing high cardinality metrics in disk vs in-memory index, vijay will comment shortly.
mvijaykarthik
3 months ago
1. We store high cardinality metrics in object storage and only keep most recent data in memory. We use serverless functions (lambdas) for querying these objects which helps us scale.
2. We do not have a global index - each hour of metrics have their own index. This helps with ephemeral labels as size of the index remains small