The success and failure of Ninja (2020)

211 pointsposted 15 hours ago
by quincepie

43 Comments

willvarfar

14 hours ago

> we talk about programming like it is about writing code, but the code ends up being less important than the architecture, and the architecture ends up being less important than social issues.

A thousand times this! This puts into words something that's been lurking in the back of my mind for a very long time.

nuclearnice3

14 hours ago

Strongly agree. Peopleware 1987 [1]

> The first chapter of the book claims, "The major problems of our work are not so much technological as sociological in nature". The book approaches sociological or 'political' problems such as group chemistry and team jelling, "flow time" and quiet in the work environment, and the high cost of turnover

[1] https://en.wikipedia.org/wiki/Peopleware:_Productive_Project...

no_wizard

13 hours ago

I’ve been drumming this for so long now, even before I heard of (let alone read) this book.

I feel that the development of psychology and sociology has been lost on the workplace and it isn’t well applied. Executives want everyone to be widgets except themselves, even when study after study shows that for companies to perform optimally their workers must feel well compensated, well valued, balanced freedom in the workplace, chances for advancement etc.

In many respects you could apply psychology and sociology to how products should / could behave etc. as well, which I’m sure due to the monetary component some companies have taken seriously at least in some periods of their lifecycle, like Apple under Steve Jobs in his comeback

BOOSTERHIDROGEN

8 hours ago

What if the company has significant constraints on its financial health?

lmm

7 hours ago

Then it's all the more important to avoid unnecessary employee turnover.

pydry

3 hours ago

>Executives want everyone to be widgets except themselves

Of course. This maximizes their relative power within the company.

Some executives are focused on the health of a company as a whole but not many. To most of them the pie can be assumed to be a fixed size and their job is to take as much of it as possible.

mihaaly

14 hours ago

Considering that programing and tools used for it are not for computers but humans, and that apart from most trivial things more than one people is necessary to make something that work on/with computer(s), it is no surprise that SE is much more social science than many would like to admit or feel comfortable with, over-emphasizing its natural science part to the level of failure eventually (on the product level aimed at addressing needs of the people). Probably because social sciences are very fluid and much less reliable than natuaral sciences, so we have an inner tendency avoiding the social bit, or handling it on a very primitive level? I do not know, this is a feeling. So much focus on atomic details of technology yet the group effort of the product is still rubbish too many times.

transpute

13 hours ago

https://en.wikipedia.org/wiki/Conway's_law

> Organizations which design systems (in the broad sense used here) are constrained to produce designs which are copies of the communication structures of these organizations. — Melvin E. Conway, How Do Committees Invent?

transpute

9 hours ago

Source code repos could have USER.md and DEVELOPER.md files to record social context.

lou1306

2 hours ago

But again, that is at best infrastructure documentation, not code. Unless you dilute the term "code" until it loses nearly all utility.

Swizec

14 hours ago

In my experience roughly 80% of technical issues are because 2 people (or teams) didn’t want to just sit down together and talk it out.

dang

15 hours ago

Discussed at the time:

The Success and Failure of Ninja - https://news.ycombinator.com/item?id=23157783 - May 2020 (38 comments)

(Reposts are fine after a year or so! links to past threads are just to satisfy extra-curious readers)

high_priest

11 hours ago

> I also believe that programmers feel latency and it affects their mood even if they don't notice it. (Google has recently done some research in this area that kinda confirmed my belief, here's hoping they'll publish it publicly!)

Anyone knows if it happened? Has the google research on latency been published?

edflsafoiewq

7 hours ago

Most interesting point to me

> You must often compromise between correctness and convenience or performance and you should be intentional when you choose a point along that continuum. I find some programmers are inflexible when considering this dynamic, where it's somehow obvious that one of those concerns dominates, but in my experience the interplay is pretty subtle; for example, a tool that trades off correctness for convenience might overall produce a more correct ecosystem than a more correct but less convenient alternative, if programmers end up avoiding the latter.

mgaunard

14 hours ago

I switched to samurai for the few things I have that still used ninja; it's an improvement in every possible way.

But regardless, I think those kinds of build systems are just wrong. What I want from a build system is to hash the content of all the transitive inputs and look up if it exists or not in a registry.

bonzini

9 minutes ago

Is Samurai still alive? I have sent a pull request to improve signal handling but it has been sitting ignored for over half a year.

dikei

29 minutes ago

Yes, basically any build system that supports distributed caching use digest instead of timestamp when checking modification: Bazel, Pants, Buck, etc.

They're all hugely complex though.

For local build only, I think SCons and Waf both use hash for changes detection.

Sesse__

13 hours ago

You might be interested in n2, from the author of ninja.

chubot

8 hours ago

What’s better about Samurai? I thought it was a compatible subset of ninja

Also, “not the thing I wanted” doesn’t mean “wrong”, simply because there are other people in the world with different preferences

mgaunard

2 hours ago

One thing in particular that's always been a problem with Ninja is the output. It does too much buffering, removes colors without being able to force them back, and in general leads to an experience where for me it's not usable since I want to pipe its output to a pager. When I used ninja I needed to maintain builds with all sorts of patches to fix it. With samurai it just did the right thing out of the box.

phyrex

4 hours ago

That's how metas buck2 works

TOGoS

13 hours ago

I think that was the idea behind NetKernel.

I've built something similar, a Deno library called "TDAR"[1], and it works well, but it takes some work to wrap up all the command-line tools that expect to work in some mutable filesystem so that you can pretend you're calling pure functions.

[1] I haven't got around to pulling it out of the parent project[2], but I talked about it in this youtube video: https://youtu.be/sty29o8sUKI

[2] If you're interested in this kind of thing you could poke me to open up the source for that thing. togos zero zero at gee mail dot comb

dima55

13 hours ago

That's called "ccache"

mgaunard

13 hours ago

ccache is just a hack to make traditional build systems less stupid.

Good build systems have native support for these things.

pjmlp

13 hours ago

Given that ninja is required for C++20 modules when using CMake, it is going to stay around for quite a bit.

grobibi

14 hours ago

I thought this was going to be about people buying less air fryers.

airstrike

13 hours ago

I thought of the smoothie blenders first too, but I can't see how they would ever have failed given how great they are. My life has changed since buying the first such blender about 4 months ago

firesteelrain

8 hours ago

Oh don’t call the Ninja a blender - there is a giant thread on one of the main FB groups. OP is getting ripped

Spivak

6 hours ago

Is there some fun tea here? Ninja themselves describe them as blenders, has the community mythologized them into something else?

firesteelrain

2 hours ago

More like an ice shaver that adds air is what the community likes to call it

Because blenders don’t turn things into an ice cream texture

ultrafez

an hour ago

We're conflating the Ninja Creami with Ninja's smoothie makers and blenders - they are separate product lines

Krastan

12 hours ago

I thought this was going to be about the Fortnite streamer

forrestthewoods

an hour ago

Ninja is pretty popular with gamedevs.

I was amused by this line:

> But Windows is still a huge platform in terms of developers, and those developers are starved for tools.

As a primarily Windows dev I feel that it is poor Linux devs who are starved for tools! Living life without a good debugger (Visual Studio) or profiler (Superluminal) is so tragic. ;(

It does feel like in recent years the gap between the two platforms is increasingly minimal. I definitely like all the Rust utilities that generally work crossplatform for example.

zX41ZdbW

12 hours ago

> Relatedly, please forgive me for the embarrassing name.

The name is great!

PS. It's possible to make it even faster if we implement this: https://github.com/ninja-build/ninja/issues/2157 But you explained in the article that the tool intentionally lacks state, even tiny hints from previous runs.

einpoklum

14 hours ago

### Statistics ###

ninja has ~26 kloc, ~3,100 commits, and only a quarter of them by the original author (although by loc changed their weight is higher). Interesting!

https://github.com/ninja-build/ninja/graphs/contributors

### Bunch of other comments ###

> users of ninja ... all Meson projects, which appears to increasingly be the build system used in the free software world;

So, AFAICT, that hasn't turned out to be the case.

> the code ends up being less important than the architecture, and the architecture ends up being less important than social issues.

Well... sometimes. Other times, the fact that there's good code that does something goes a very long way, and people live with the architectural faults. And as for the social issues - they rarely stand in opposition to the code itself.

> Some pieces of Ninja took struggle to get to and then are obvious in retrospect. I think this is true of much of math

Yup. And the some of the rest of math becomes obvious when some re-derives it using alternative and more convenient/powerful techniques.

> I think the reason so few succeed at this is that it's just too tempting to mix the layers.

As an author of a library that also focuses on being a "layer" of sorts (https://github.com/eyalroz/cuda-api-wrappers/), I struggle with this temptation a lot! Especially when, like the author says, the boundaries of the layers are not as clear as one might imagine.

> I strongly believe that iteration time has a huge impact on programmer satisfaction

I'm pretty certain that the vast majority developers perform 10x more incremental builds than full builds. So, not just satisfaction - it's just most of what we do. It's also those builds which we wait-out rather than possible go look for some distraction:

https://xkcd.com/303/

OTOH, the article doesn't mention interaction with build artifact caching schemes, which lessen the difference between building from scratch and building incrementally.

> Peter Collingbourne found Ninja and did the work to plug it into the much more popular CMake ... If anyone is responsible for making Ninja succeed out there in the real world, Peter is due the credit.

It is so gratifying when a person you didn't know makes your software project that much more impactful! Makes you really feel optimistic again about humanity and socialism and stuff.

a_t48

14 hours ago

Im going to have to give your CUDA wrapper a look later. :)

einpoklum

12 hours ago

I should say that unlike the author of ninja though, I am _very_ interested in user complaints and criticism, even if its not fully articulated and respectful. I _need_ contradiction and opposition to go beyond the bounds of my own conceptions as a almost-always-sole developer and sole maintainer of the library. I may not accept/agree with everything, but I'll at least know to take the concerns into consideration. And I've already refactored quite a bit over the years based on use cases user have pointed out to me.

a_t48

11 hours ago

Same :) I started down the rabbit hole of abstracting CUDA for our robotics framework, but it’s not really something I want to maintain right now.

santoshalper

14 hours ago

Man, I was so afraid this was going to be about Fortnite. Turns out it was a fantastic read. I feel really sad but unsurprised about his description of what it's like to be an Open Source maintainer.