That was difficult to read, smelt very AI assisted though the message was worthwhile, it could've been shorter and more to the point.

A few things I've been thinking about recently:

- we have authentication everywhere in our stack, so I've started including the user id on every log line. This makes getting a holistic view of what a user experienced much easier.

- logging an error as a separate log line to the request log is a pain. You can filter for the trace, but it makes it hard to surface "show me all the logs for 5xx requests and the error associated" - it's doable, but it's more difficult than filtering on the status code of the request log

- it's not enough to just start including that context, you have to educate your coworkers that it's now present. I've seen people making life hard for themselves because they didn't realize we'd added this context

A post on this topic feels incomplete without a shout-out to Charity Majors - she has been preaching this for a decade, branded the term "wide events" and "observability", and built honeycomb.io around this concept.

Also worth pointing out that you can implement this method with a lot of tools these days. Both structured Logs or Traces lend itself to capture wide events. Just make sure to use a tool that supports general query patterns and has rich visualizations (time-series, histograms).

While I agree with some of it, I feel like there's a big gotcha here that isn't addressed. Having 1 single wide event, at the end of a request, means that if something unexpected happens in the middle (stack overflow, some bug that throws an error that bypasses your logging system, lambda times out etc...) you don't get any visibility into what happens.

You also most likely lose out on a lot of logging frameworks your language has that your dependencies might use.

I would say this is a good layer to put on top of your regular logs. Make sure you have a request/session wide id and aggregate all those in your clickhouse or whatever into a single "log".

The presentation is fantastic and I loved the interactive examples!

Too bad that all of this effort is spent arguing something which can be summarised as "add structured tags to your logs"

Generally speaking my biggest gripe with wide logs (and other "innovative" solutions to logging) is that whatever perceived benefit you argue for doesn't justify the increased complexity and loss of readability.

We're throwing away `grep "uid=user-123" application.log` to get what? The shipping method of the user attached to every log? Doesn't feel an improvement to me...

P.S. The checkboxes in the wide event builder don't work for me (brave - android)

Horrid advice at the end about logging every error, exception, slow request, etc if you are sampling healthy requests.

Taking slow requests as an example, a dependency gets slower and now your log volume suddenly goes up 100x. Can your service handle that? Are you causing a cascading outage due to increased log volumes?

Recovery is easier if your service is doing the same or less work in a degraded state. Increasing logging by 20-100x when degraded is not that.

> Logs were designed for a different era. An era of monoliths, single servers, and problems you could reproduce locally.

I worked with enterprise message bus loggers in semiconductor manufacturing context wherein we had thousands of participants on the message bus. It generated something like 300-400 megabytes per hour. Despite the insane volume we made this work really well using just grep and other basic CLI tools.

The logs were mere time series of events. Figuring out the detail about specific events (e.g. a list of all the tools a lot visited) required writing queries into the Oracle monster. You could derive history from the event logs if you had enough patience & disk space, but that would have been very silly given the alternative option. We used them predominantly to establish a casual chain between events when the details are still preliminary. Identifying suspects and such. Actually resolving really complicated business usually requires more than a perfectly detailed log file.

Because of the nature of how software is built and deployed nowadays, it’s generally not possible to write single log entries that tell the “whole story” of “what happened”.

I could write about this for hours, but instead I’ll just discuss two concepts that you need in modern logging: vertical correlation and horizontal correlation.

Within a system, requests tend to go “up” and “down” stacks of software. It is very useful in these scenarios to have “vertical correlation” fields shared between adjacent layers, so that activity in one layer can be unambiguously attributed to activity in the adjacent layers. But sharing such a correlation value requires passing the value between layers, which might be a breaking api change. Occasionally it’s possible to construct a correlation value at each adjacent layer by transforming existing parameters in exactly the same way on the calling side and called side.

Additionally, software on one system converses with software on other systems; in those cases you need to have pairwise correlation values between adjacent peer layers. Again, same limitations apply to carrying such a correlation value via the API or protocol.

Really foresighted devs can anticipate these requirements and generate unique transaction ids that can be shared between machines and up and down the stack.

I hope registering an entire domain name for a blog post doesn't become a trend. I like linking to things that are likely to last a long time - a personal blog is one thing, but expecting people to keep paying the renewal fee every year for a single article feels less likely to me.

A good alternative here is subdomains, since those don't have an additional annual fee. https://logging-sucks.boristane.com/ could work well here.

> Logs were designed for a different era. An era of monoliths, single servers, and problems you could reproduce locally. Today, a single user request might touch 15 services, 3 databases, 2 caches, and a message queue. Your logs are still acting like it's 2005.

Logs are fine. The job of local logs is to record the talk of a local process. They are doing this fine. Local logs were never meant to give you a picture of what's going on some other server. For such context, you need a transaction tracing that can stitch the story together across all processes involved.

Usually, looking at the logs at right place should lead you to the root cause.

I agree with this statement: "Instead of logging what your code is doing, log what happened to this request." but the impression I can't shake is that this person lacks experience, or more likely has a lot of experience doing the same thing over and over.

"Bug parts" (as in "acceptable number of bug parts per candy bar") logging should include the precursors of processing metrics. I think what he calls "wide events" I call bug parts logging in order to emphasize that it also may include signals pertaining to which code paths were taken, how many times, and how long it took.

Logging is not metrics is not auditing. In particular processing can continue if logging (temporarily) fails but not if auditing has failed. I prefer the terminology "observables" to "logging" and "evaluatives" to "metrics".

In mature SCADA systems there is the well-worn notion of a "historian". Read up on it.

A fluid level sensor on CANbus sending events 10x a second isn't telling me whether or not I have enough fuel to get to my destination (a significant question); however, that granularity might be helpful for diagnosing a stuck sensor (or bad connection). It would be impossibly fatiguing and hopelessly distracting to try to answer the significan question from this firehose of low-information events. Even a de-noised fuel gauge doesn't directly diagnose my desired evaluative (will I get there or not?).

Does my fuel gauge need to also serve as the debugging interface for the sensor? No, it does not. Likewise, send metrics / evaluatives to the cloud not logging / observables; when something goes sideways the real work is getting off your ass and taking a look. Take the time to think about what that looks like: maybe that's the best takeaway.

> Logs were designed for a different era. An era of monoliths, single servers, and problems you could reproduce locally. Today, a single user request might touch 15 services, 3 databases, 2 caches, and a message queue. Your logs are still acting like it's 2005.

If a user request is hitting that many things, in my view, that is a deeply broken architecture.

> Logging Sucks

But does it? Or is it bad logging, or excessive logging, or unsearchable logs?

A client of mine uses SnapLogic, which is a middleware / ETL that's supposed run pipelines in batch mode to pass data around between systems. It generates an enormous amount of logs that are so difficult to access, search and read that they may as well don't exist.

We're replacing all of that with simple Python scripts that do the same thing and generate normal simple logs with simple errors when something's truly wrong or the data is in the wrong format.

Terse logging is what you want, not an exhaustive (and exhausting) torrent of irrelevant information.

This was a brilliant write up, and loved the interactivity.

I do think "logs are broken" is a bit overstated. The real problem is unstructured events + weak conventions + poor correlation.

Brilliant write up regardless

One thing this is missing: Standardization and probably the ECS' idea of "related" fields.

A common problem in a log aggregation is the question if you query for user.id, user_id, userID, buyer.user.id, buyer.id, buyer_user_id, buyer_id, ... Every log aggregation ends up being plagued by this. You need standard field names there, or it becomes a horrible mess.

And for a centralized aggregation, I like ECS' idea of "related". If you have a buyer and a seller, both with user IDs, you'd have a `related.user.id` with both id's in there. This makes it very simple to say "hey, give me everything related to request X" or "give me everything involving user Y in this time frame" (as long as this is kept up to date, naturally)

Google solved most of these problems around 2005, with tools like LOG_EVERY_N (now part of absl [1]), Dapper [2], and several other tools that aren't public yet. You can trace an individual request through every internal system, view the request/response protobufs, every log that the server emitted, timing details, etc. More to the point, you can share this trace, which means that it's possible for one person to discover the bug, reproduce it, and then have another person in a completely different office/timezone/country debug it, even if the latter cannot reproduce the bug themselves. This has proved hugely useful; just last week I was tasked with reproducing a bug on sparsely-available prerelease hardware so that a distant team could diagnose what went wrong.

The key insight that this article hints at but doesn't quite get too: you should treat your logs as a product whose customers are the rest of the devs in your company. The way you log things is intimately connected with what you want to do with them, and you need to build systems to generate useful insights from the log statements. In some cases it literally is part of the product: many of the machine learning systems that generate recommendations, search results, spam filtering, abuse detection, traffic direction, etc. are all based on the logs for the product, and you need to consider them as first-class citizens that you absolutely cannot break while adding new features. Logs are not just for debugging.

[1] https://absl.readthedocs.io/en/latest/absl.logging.html

[2] https://research.google/pubs/dapper-a-large-scale-distribute...

Just out of curiosity, how have you seen risk/compliance, regulatory, and audit departments at organizations deal with the disconnect between security and privacy for something like mainframe logging (e.g., JES2, JES3), which is typically inherently governed, and modern distributed logging, which is typically inherently permissive? Both are vastly different approaches, but each is somehow considered 'compliant.' Btw, employees at a company I was at were once investigated for insider trading simply because it was discovered the company used pooled logs that were accessible by production support programmers (the company decided to override the default mainframe security), which was deemed a possible source of insider trading information that could be tapped into by those who had log access (programmers were eventually cleared if it was discovered their small personal trades were immaterial and just coincidental with the company's trading, but the investigation led to uncomfortable confrontations for some!).

Logs were designed for a different era. An era of monoliths, single servers, and problems you could reproduce locally. Today, a single user request might touch 15 services, 3 databases, 2 caches, and a message queue.

If a user request is hitting that many things, in my view, that is a deeply broken architecture.

I'm building an analytics SaaS and we made the conscious decision to keep it simple: Next.js API routes + Supabase + minimal external services. A single page view hits maybe 3 components max (CDN -> App -> Database).

That said, I agree completely on structured logging with rich context. We include user_id, session_id, and event_type on every log line. Makes debugging infinitely easier.

The "wide events" concept is solid, but the real win is just having consistent, searchable structure. You don't need a revolutionary new paradigm - just stop logging random strings and use JSON with a schema.

AI slop blogvert. The first example is disingenuous btw. Everyone these days uses requestIDs to be able to query all log lines emanated by a single request, usually set by the first backend service to receive the request and then propagated using headers (and also set in the server response).

There isn't anything radical about his proposed solutions either. Most log storage can be set with a rule where all warning logs or above can be retained, but only a sample of info and debug logs.

The "key insight" is also flawed. The reason why we log at every step is because sometimes your request never completes and it could be for 1000 reasons but you really need to know how far it got in your system. Logging only a summary at the end is happy path thinking.

Persisting a data schema that represents business events is a great idea. That’s more about Event Sourcing though and doing that can answer a ton of questions about the system without doing it in log messages.

Wide events as a strategy is expensive, even with sampling, and doesn’t address the fundamental problem - why do we log messages?

I was hoping the article would enumerate why we log messages. Nailing down those scenarios first will lead to a happy life.

Why do we log? - proof of life - is the system running? - what is the state (in memory) when an error occurred? - when did an error occur? - do I need to get up at 2 am and fix something? - what do I need to fix?

I feel like every team operating a system has their own reasons for logging.

Our logging guidance is: "Don't write comments, write logs" and that serves us pretty well. The point being, don't write code "clever code", write obvious code, and try to make it similar to everything else thats been done, regardless if you agree with it.

I've recently come off a team that was racking up a huge Splunk bill with ~70 log events for each request on a high traffic service, and this is all very resonant (except the bit about sampling, I never gave that much thought - reducing our Splunk bill 70x was ambitious enough for me!).

Hadn't heard the "wide event" name, but I had settled on the same idea myself in that time (called them "top-level events" - i.e. we would gather information from the duration of the request and only log it at the "top" of the stack at the end), and evangelised them internally mostly on the basis it gave you fantastic correlation ability.

In theory if you've got a trace id in Splunk you can do correlated queries anyway, but we were working in Spring and forever having issues with losing our MDC after doing cross-thread dispatch and forgetting to copy the MDC thread global across. This wasn't obvious from the top-level, and usually only during an incident would you realise you weren't seeing all the loglines you expected for a given trace. So absent a better solution there, tracking debug info more explicitly was appealing.

Also used these top-level events to store sub-durations (e.g. for calling downstream services, invoking a model etc), and with Splunk if you record not just the length of a sub-process but its absolute start, you can reconstruct a hacky waterfall chart of where time was spent in your query.

"Logs were designed for a different era. An era of monoliths, single servers, and problems you could reproduce locally."

But the next era will be like the previous one. Today monolith is enough for most of apps.

> Here's the mental model shift that changes everything: Instead of logging what your code is doing, log what happened to this request.

Yeah that doesn't magically fix everything. Logging is still an arbitrary, clunky, unintuitive process that requires intentional design and extra systems to be useful.

The "Wide Event log" example is 949 bytes, which isn't unmanageably large, but it is 3x larger than most log messages which are about 300 bytes. And in that blob of data might be key insights, but it is left up to an extra engineering process to discover what might be unusual in that blob. It lacks things like code line numbers, stack trace, and context given by the program about its particular functions (rather than assumptions based on a few pieces of metadata). And it's excessively verbose, as it has a trace and request ID and service name, but duplicates information already available to tracing systems based on those 3 metrics.

> Wide events are a philosophy: one comprehensive event per request, with all context attached.

That's simply impossible. You cannot have all context from viewing a single point in the network, regardless of how hard you try to record or pass on information. That's the whole point of tracing: you correlate the context of different network points, specifically because that's the only way to discover the missing details.

> Modern columnar databases (ClickHouse, BigQuery, etc.) are specifically designed for high-cardinality, high-dimensionality data. The tooling has caught up. Your practices should too.

You should not depend on a space shuttle to get to the grocery store. Logging is intended to be an abstracted component which can be built on by other systems. Your app should work just as well running from Docker on your laptop as it does in the cloud.

Content marketing and lead generation is getting more sneaky

I prefer narrow entries with an "event ID" attached; you can trivially filter to get all of your items with the same event ID, but it works better with all the tools designed for narrow entries.

So instead of one entry with e.g. 27 elements, you have maybe a dozen entries, all with the same event id and 2-4 elements (including the event ID) each. This also lets you log incrementally; if your server crashes before you log the wide entry you have zero data.

It also lets you adjust the granularity. E.g. for a web service you might have both a session and a request ID.

"Today, a single user request might touch 15 services, 3 databases, 2 caches, and a message queue."

This right here is the fundamental problem because the way it's done is highly inefficient, complex and I believe only exists so cloud providers can sell their expensive offerings.

A monolith is fine 90% of the time.

> Logs were designed for a different era. An era of monoliths, single servers, and problems you could reproduce locally. Today, a single user request might touch 15 services, 3 databases, 2 caches, and a message queue. Your logs are still acting like it's 2005.

Perhaps it's time to take back the good things from 2005.

The problem statement in this article sounds weird. I thought in 2025 everyone logs at least thread id and context id (user id, request id etc), and in microservice architecture at least transaction or saga id. You don’t need structured logging, because grep by this id is sufficient for incident investigation. And for analytics and metrics databases of events and requests make more sense.

Good write up.

Gonna go on a tangent here. Why the single purpose domain? Especially since the author has a blog. My blog is full of links to single post domains that are no longer.

Structured Logging is not just JSON. It's the use of templates with context. It solves 90% of what this article complains about if you just log the template along with the variables and the message separately. Along with logging the right stuff. IE `"User {username} created order {orderid}"`

The article, AI or not, is extremely naive. It doesn't mention any premise or any problem to solve. Proposes a solution and just goes with it. What if your monster of a event is lost when your service crashes or is lost by the logging library/service/etc? What if you're interested in measuring, post factum, how long each step takes? What if you want to trace a log through several (micro-)services and maybe between a mobile app and some batch job executor that runs once a day?

"Logging sucks" when you don't understand the problem you're trying to solve.

I agree that logging suck bollocks, especially when most of the time you really want metrics.

Opentel is great, but sadly A lot of the stuff that I am using doesn't support it.

The thing that made it much more bearable, even easy is loki and a decent log parser.

I know a lot of kids like using SQL to interact with things, loki's explore interface beats the living shit out of SQL (In my opinion) its really simple to just isolate and slice logs interactivly. You can build your query really simply.

It beats splunk/sumo/scuba(facebook's log system) hands down in terms of searchability.

Kinda get what he’s saying: provide more metadata with structured logging as opposed to lots of string only logs. Ok, modern logging frameworks steer you towards that anyway. But as a counterpoint: often it can be hard to safely enrich logging like that. In the example they include subscription age, user info, etc. More than once I’ve seen logging code lookup metadata or assume it existed, only to cause perf issues or outright errors as expected data didn’t exist. Similar with sampling, it can be frustrating when the thing you need gets sampled out. In the end “it depends” on scenario, but I still find myself not logging enough or else logging too much

I've recently added error tracking to my self-hosted analytics app (UXWizz), and the way I did it is simply add extra events to each user/session. Once you have the concept of a session or user, you can simply attach errors or logs as Events stored for that user. This solves the main problem mentioned in the article, where you don't know what happened, plus being an Event stored in a MySQL database, you can still query it.

Why not simply use Events for logging, instead of plain strings?

Logging is one tool of many, you need logs, metrics and distributed tracing at the very least for any significant piece of modern infra.

If you really want to get serious, you also want some kind of continuous profiling (like pyroscope) or at the very least some periodic thread dump collector (for serious degradation diagnostics once every couple of minutes is enough).

But logging is still a great tool.

I see more and more blog posts that contain interactive elements. Despite the general enshittification of the average blog and the internet, this feels like a 'modern' touch that actually adds something valuable to the sufficient ad-free no-popups old blog style.

You might also need different systems for low-cardinality, low-latency production monitoring (where you want to throw alerts quickly and high cardinality fields would just get in the way), and medium to long term logging with wide events.

Also if you're going to log wide events, for the sake of the person querying them after you, please don't let your schema be an ad hoc JSON dict of dicts, put some thought into the schema structure (and better have a logging system that enforces the schema).

This seems like a classic time vs space trade off.

Instead of reconstructing a "wide event" from multiple log lines with the same request id, the suggestion seems to be logging wide events repeatedly to simplify reconstruction from request ids.

I personally don't see the advantage, and in either scenario, if you're not logging what's needed your screwed.

> Today, a single user request might touch 15 services, 3 databases, 2 caches, and a message queue.

And this is why _the internet_ today sucks.

From what I gather: This is referring to Web sites or other HTTP applications which are internally implemented as a collection of separate applications/ micro-services?

You get: "Show me all checkout failures for premium users in the last hour where the new checkout flow was enabled, grouped by error code."

I can do this with axiom already and I’ve never worried about how to log.

> No grep-ing.

How is grep a bad thing? I find myself using it all the time.

I’m not into graphical user interfaces. They overwhelm me. By the time I’ve clicked myself through the GUI or written some horrible proprietary $COMPANY Query Language string, I might have already figured out the bug using tried and tested CLI tools.

I broadly agree with the article.

The described pattern is standard in Meta. This, along with the infrastructure and tooling to support it, was the single largest "devx quality of life improvement" in my experience moving to big tech.

Use events instead of repetitious logging calls.

https://dave.autonoma.ca/blog/2022/01/08/logging-code-smell/

> An era of monoliths, single servers, and problems you could reproduce locally.

Actually this is the problem. It’s extremely difficult to debug when you do not preserve basic properties that allow you to read code and reason about events.

I've generally found that structured logs that include a correlation ID make it quite easy to narrow down the general area or exact cause of problems. Usually (in enterprise orgs) via Splunk or Datadog.

Where I've had problems it's usually been one of:

There wasn't anything logged in the error block. A comment saying "never happens" is often discovered later :)

Too much was logged and someone mandated dialing the logging down to save costs. Sigh.

A new thread was started and the thread-local details including the correlation ID got lost, then the error occurred downstream of that. I'd like better solutions for that one.

Edit: Incidentally a correlation ID is not (necessarily) the same thing as a request ID. An API often needs to allow for the caller making multiple calls to achieve an objective; 5 request IDs might be tied to a single correlation ID.

this is the best lead generation form i've ever seen

Correction, logging used to suck - now it's fixed https://pydantic.dev/logfire :-)

Slapping on OpenTelemetry actually will solve your problem.

Point #1 isn't true, auto instrumentation exists and is really good. When I integrate OTel I add my own auto instrumentors wherever possible to automatically add lots of context. Which gets into point #2.

Point #2 also isn't true. It can add business context in a hierarchal manner and ship wide events. You shouldn't have to tell every span all the information again. Just where it appears naturally the first time.

Point #3 also also isn't true because OTel libs make it really annoying to just write a log message and very strongly pushes you into a hierarchy of nested context managers.

Like the author's ideal setup is basically using OTel with Honeycomb. You get the querying and everything. And unlike rawdogging wide events all your traces are connected, can span multiple services and do timing for you.

the best implementation of structured logging I've seen is dotnet build's binlogs (https://msbuildlog.com), I would love to see it evolve into a general purpose logging solution

AI writing sucks even more, get rekt

Sounds like he’s just asking for an old school Inman style transaction log.

Tangential, but I wonder if the given example might be straying a step too far? Normally we want to keep sensitive data out of logs, but the example includes a user.lifetime_value_cents field. I'd want to have a chat with the rest of the business before sticking something like that in logs.

Overly dismissive of OTLP without proper substance to the criticism.

Maybe better written and simplified to: “microservices suck”.

> Your logs are lying to you. Not maliciously. They're just not equipped to tell the truth.

The best way to equip logs to tell the truth is to have other parts of the system consume them as their source of truth.

Firstly: "what the system does" and "what the logs say" can't be two different things.

Secondly: developers can't put less info into the logs than they should, because their feature simply won't work without it.

The substance of this post is outstanding.

The framing is not, though. Why does it have to sound so dramatic and provocative? It’s insulting to its audience. Grumpiness, in the long term, is a career-limiting attitude.

Some excellent points raised in this article.

distributed event id and you are all set

This article is attacking a strawman. It makes up terrible logs and then says they are bad. Even if this was a single monolith the logs still don't include even something like a thread id, to avoid mixing different requests together.

Splunk is expensive but it makes searching logs so much faster and more effective. I think of it as SQL for unstructured data.

Logfiles are a user interface.