You'd generally add an index to your logs in Clickhouse to do searching (via ngram or token bloom filters typically: https://clickhouse.com/docs/en/optimize/skipping-indexes#blo...). There's other ways of indexing as well but that's generally the best for full text search. We use token bloom filter indexes today and find them quite effective (it can skip whole chunks of logs due to the bloom filter being able to say that a word did not appear in the chunk of logs).

Indeed Loki is incredibly slow - Clickhouse is deployed for logging at scale (ex. trip.com is running a 50pb logging solution that allowed them to 4x their old ES cluster volume while also running queries 4-30x faster)

If you want fast searching for some unique word or phrase across terabytes of logs (aka "needle in the haystack" type of search), then take a look at VictoriaLogs [1] (I'm the core developer). It uses bloom filters for quick skipping of data blocks, which do not contain the given word or phrase. Contrary to other open-source solutions for log storage and analysis, VictoriaLogs works efficiently with any types of logs containing any sets of fields, without the need in any configuration and tuning.

[1] https://docs.victoriametrics.com/victorialogs/

Hrm while we aren't a 1:1 Kibana replacement today (we're not apples to apples since Kibana is locked into Elastic, whereas we're on Clickhouse) - I don't think we're too far off with our UI-based filters, Lucene filter language, and timestamp filtering/sorting/live tail.

There is a setup modal (which is intended to be set up once by the data owner, similar to maybe how you'd set up some indexes in Elastic) - and afterwards the experience is similar IMO. If you're open to sharing more I'd love to learn more mike@hyperdx.io or if you want to open an issue/join our discord.

Thank you, really appreciate feedback like that! :D

thank you! means a lot coming from you ;)

Speaking of the system tables - it's awesome how much telemetry is saved in there that helps us build a really powerful preset clickhouse monitoring dashboard (heavily inspired from the built-in clickhouse one of course). We figured that alone is quite useful for teams running any Clickhouse instance and want better insights into what's going on.

Yup! Since we're built on Otel, we suggest you can use the HTTP OTLP endpoint, which is yourdomain:4318/v1/logs, we have a quick blog post on that on how to curl a payload over: https://www.hyperdx.io/blog/testing-sending-opentelemetry-ev...

But you can also look at example payloads in the otel repo: https://github.com/open-telemetry/opentelemetry-proto/blob/m...

Great call out on CF workers, will make sure we get that updated :)

Totally fair! Here's a few on the top of my head, they're mostly about Elastic really but of course Kibana is only really useful on ES:

1. At my last job we were running some of the largest elastic clusters of our hyperscaler's cloud - elastic is slow, expensive and finicky to operate at decent scale. We've found the exact opposite with Clickhouse, it's fast, easy to operate, and supports things like S3-backed storage directly in their open source product. As an example, Uber switched from Elastic -> Clickhouse and halved their infra footprint while increasing volume.

2. Elastic is tricky to manage, field type conflicts come up super common at scale and are annoying to deconflict. Clickhouse is a lot more flexible in its schema to avoid those problems (and give you knobs to fine tune performance at a more granular level with their indexes/schemas)

3. We allow for both SQL and Lucene, both are relatively "standard" languages that engineers are likely already familiar in one way or another. Compared to elastic moving to ES|QL, another vendor-specific language that will be difficult to onboard to. The last thing you want during an incident is trying to recall vendor-specific languages for querying that critical data!

tl;dr - we try to make it easy to "do observability" on what we think is the best DB for observability today (Clickhouse), analogous to what Kibana did for ES.

This world needs a new Kibana that is lightweight and not written in java/typescript.

Thank you Max! It's awesome to hear that :)

Our v1 is completely built on Clickhouse! So v2 is making it more widely compatible to Clickhouse installations that aren't tied to our ingestion pipeline and schema. So if you're already on Clickhouse for obseravbility today, or have a preferred way of shipping data in, you can use us on top of Clickhouse now without throwing away your existing work.

We're essentially making our existing product a lot easier to deploy into more complex observability setups based on Clickhouse - while also shipping a few new capabilities like SQL and event deltas while we're at it!

If you're using the Grafana Clickhouse plugin - a number of things we do differently to them today:

- We support Lucene-based search, which means it's a lot easier to find the events you're looking for without needing to break into verbose SQL search. (Column/property/full text search are all super easy).

- We're optimized exclusively for Clickhouse, which means we do a number of things to optimize the queries we run and that returns you a nice performance boost (we see a 2x perf boost, but this will vary a ton of data and queries). For example we allow you to do a search on a subset of columns (so the search is performant), and then click in and expand an entire row of interest on-demand (so we only do a SELECT * for a single row). This is also a much nicer DX than needing to specify every column you might need. We have a few other optimizations as well.

- We have (what we think is) a nice chart builder - so you don't need to mess with template variables and macros to build a chart, but still lets you escape hatch out into SQL for the important bits.

- We think our event deltas feature for analyzing traces is pretty neat - afaik this isn't something you get in Grafana. - You can live tail events, I don't think Grafana for Clickhouse has that (I'm a big fan of tail -f)

Overall we focus on trying to bring an easy-to-use high-cardinality observability experience to Clickhouse, whereas Grafana seems to focus more on a highly SQL-dependent dashboard building experience (which has its own advantages of course).

edit: fixed line breaks!

Kibana is once again Open Source, as of 2 months ago. https://github.com/elastic/kibana/blob/main/LICENSE.txt

What do we say in such cases? It was good while it lasted!

Once that happens - eventually some new kids would appear on the block.

Such is the life.

Seems particularly true for tools that have operational implications. It's very easy to justify why something should be for-pay when it's indispensible day in and day out.

Ahh yeah fwiw it wasn't _intended_ to be a dig at the open source status of Kibana - but rather we're open source and building on top of Clickhouse.

On the commercial OSS side of things - I suspect the trend there is more nuanced than all OSS companies being suspect to the same problem, but rather companies that generally solve a "behind the API" problem are more susceptible to problems of cloud vendors taking their code and competing with them commercially (ex. if you're a DB like Redis, Mongo, Elastic - or a CLI like Terraform). We're building an end-user experience (more like Gitlab) - where experience differentiation matters a lot more than simple infrastructure hosting, something AWS is not particularly well suited at competing on!

It's been 3 years of Gitlab post-IPO and they're still MIT, and that's the boat we're on as well :)

I didn't downvote you, but people who shill their wares without providing a link are just wasting everyone's time