I think basically it is like

https://en.wikipedia.org/wiki/Windows_Display_Driver_Model

except Microsoft had that up and running in 2006 but politics has kept Linux from doing the same. Specifically, WDDM and Wayland simplify the graphics interface to an API that lets you blast out a rectangle of pixels so unlike the old HAL model you weren't specifying an API where it can draw primitives like lines and circles, the API allows user-space applications to run software on the GPU to make pixel rectangles that either get blasted to the screen or composited.

Microsoft is really good at maintaining backwards compatible with the UI (I can still run Office '97 on my PC) and they figured out how to make it so old applications don't know what happened.

Incompetence at GUI work is a hallmark of the politics of Linux, just as power management "just not working" is a hallmark of the politics of the Intel-Microsoft-Dell alliance.

> Can anyone explain to me what the fundamental differences between X11 and Wayland in this respect?

> What newer better features is it meant to provide, or what problems is it meant to solve or address in relation to X11 and why should it take so long?

IMHO, Wayland is meant to address a few issues that can't really be addressed in X11.

Integrating the 'display server' with the window manager eliminates some unfixable syncrhonization issues in the X11 design, mostly around creation of new windows and how they're placed (or something like that, I don't quite remember), but the downside is that window managers need to integrate a compositor and have more behavior for that.

The other big thing is that X11 is a distributed system with no security between parts. Any connected X client can read or generate keyboard events at any time, or read any part of the screen. This is not ideal for lots of reasons, but the Wayland way requires different approaches to make things like screenshots, screen capture, and global hotkeys work. Also, because each window manager/compositor has its own display server (although many use the wlroots library for the base), consensus building is needed or a program that wants a hotkey needs to know how to do it in X and in all the popular Wayland compositors. Consensus building is hard.

From what I understand, Wayland required new driver architecture, and drivers for Nvidia were slow to arrive.

X11 was built around networking and has always allowed for a display server located near your monitor and clients distributed elsewhere in a network. This used to work well, but requires proper client architecture to handle the asynchronous nature of networks for best results. There's a fraction of users who make use of this (including me) in their regular use case for X. Waypipe was slow to arrive, and isn't as convenient as ssh -X, but should help provide for this use case.

There's probably some other stuff. But basically, Wayland is not a drop in replacement for X11 and X11 works for a lot of people, so they don't see a big reason to switch and disrupt their way of life.

It continues because most of the people who were working on Xorg prefer to work on Wayland. There's a lot of people relying on the work of a few people who do the deep work on display servers, so you know... either you use what they're making, or you use the older things they made, or you go to CLIs. :)

X11 is just a network protocol that specifies how a client (e.g., your browser) can send messages to a server (e.g., /usr/bin/Xorg) so the server will draw stuff on the screen. A complicated part of this process is the decision of where each client windows will be shown on the screen, so in X11 we have a special client that is the "window manager" (WM), which does stuff like drawing maximize/minimize buttons everywhere, coordinating which apps are being shown, and, these days, also compositing (which is a more complicated version of what I just said, except the apps don't even know where they are on the screen or if they are being displayed). The fact that the WM/compositor is just a client and not a server creates all sorts of problems, especially in the security area, but also synchronization. The protocol was created in the early 1980's so there are all sorts of insecure design decisions and other decisions that made a lot more sense at that time.

Since then, the X11 protocol got extended and extended, but the "core protocol" is not allowed to change (by design). There are a zillion instructions on how applications can tell the server how to draw stuff, but basically none of these are used anymore.

These days, when it comes to drawing, what the clients want to do is very different than the traditional X11 network protocol model. Clients want to create themselves the buffers, draw on them (using GL/Vulkan), and just be able to tell the compositor: "hey, I finish drawing on my buffer, please display it on the screen", without even having to send pixels over the network. This is accomplished by the DRI and Composite extensions (and a lot of glue everywhere) and these clients are able to not make use of a vast part of the X11 protocol.

So based on all that, Wayland's idea is to promote that new model to be the base level. Wayland is also a protocol, but there are no protocol calls such as "draw this line from x:20,y:20 to x:40,y:20". The operations match the new model: "hey, I just finished drawing on my window that has handle 0x30984, can you please display it for me?" (well, actually, it's more like: "when syncobj handle 234 gets signaled, the buffer is yours to display"). The big difference here is that instead of having a server, a client and a WM/compositor, there is just a client and a server/WM/compositor. This simplifies a lot of things, allows more security, but creates the problem that, well, Gnome wants its compositor to behave in some way, but KDE wants its compositor to behave in another way, so each one will have to be a whole Server and not just an application anymore. And that causes the fragmentation that has been haunting us a lot.

On top of all that, Input handling and Accessibility was sorta underestimated on the Wayland side, and things have been evolving rather slowly. A lot of stuff that is possible to accomplish these days with X11 is still not covered by Wayland. A lot of super-specialized X11 apps will never be ported to Wayland, and there is probably not even Wayland protocol in existence to support that. This is all, of course, a solvable problem: throw enough money at these issues and they will all disappear (but you may want to dedicate part of this money as bribes, because gathering consensus for some stuff without bribery may be impossible so your extension proposals may get stuck in limbo). For example: even a filthy rich behemoth such as Valve has been having problems merging extensions that will allow its games work better on Linux. Good luck with anything you may need.

I am also deeply disappointed that Wayland has been taking to much to develop, but I don't have anybody specifically to blame. I believe part of the issue is that more money should be thrown by I-dont-know-who at it. The Compositor fragmentation is also a painful issue, but going back to insecure X11 protocols doesn't seem like a good alternative. Anyway: I just don't know how to make things better. Let's just try not to blame the developers that have been doing herculean efforts to keep all these cards stacked on top of each other. Without them, perhaps we'd still be doing insecure inefficient 80's-style window drawing and wrapping, moving pixels all over network sockets.

I'd love to be corrected on my views by anybody on this forum.