One problem with this is that it presupposes that the interviewer is skilled enough to spot problems, ask the right probing questions, etc. Casey is definitely skilled enough. I can’t say that’s universal.

Another problem, which TFA hints at, is bias. System Design interviews are often terrible for this reason. If you present me with some scenario that I know is trivial to handle with a single server, I’m going to want to discuss that, but you are probably expecting me to talk about a message queue, a caching layer, etc. Both are valid depending on the situation, but if you’ve only ever known one type, you may dismiss others out of hand.

I can probably go into great detail if we're talking about a project I worked on in the past 3-6 months. But beyond that, the details of past work start to become fuzzier. So if I encounter an interviewer like this, I'd better hope that my most recent project fits the bill, because I'd probably stumble through it for earlier projects.

That’s absolutely right, but there’s another issue with the LeetCode-style interview that hasn’t been getting much attention lately, including in this article. My company is hiring right now, and we’ve shifted all of our initial interviews to Zoom, where we include a brief coding task. However, it’s becoming more and more clear that many applicants are using LLMs to produce their code. It’s reached the point where it feels almost impossible to assess whether someone can actually code on their own in these remote settings. On the other hand, it’s much harder to lean on an LLM in a more open-ended, conversational interview without it feeling unnatural. That, I think, is one of the biggest flaws in the current remote coding interview setup.

I like this approach. The only problem (or at least one of the problems) is that many people do not have suitable personal projects, as in they might be 10 years old. If one of the goals is to filter out people that don't work on projects in their free time, then this works, but I know many good programmers who don't, or at least they don't anymore due to having kids and other responsibilities outside of work.

As this article touches on, I'm big into pair programming interviews. I was part of the pair programming step at a past company and we'd always use StringCalc [0]. It's starts out super easy and gets gently progressively harder. The goal isn't to finish but to just to see how people think. We would do pretty legit pairing where we'd help if they got stuck on anything or thought we had a better solution. This shows so much about how someone thinks, collaborates, and responds to feedback. Often within 10 mins I could tell how it was going to go. We always had to finish, though, just in case.

Of course it depends on what type of app you have and how your company works and so on and so forth.

[0] https://github.com/MokshankSoni/StringCalc_TDD

Some engineers do not have personal projects to show. Only work projects that can’t be shown.

And junior developers who have personal or university projects it’s sometimes too shallow for this check described in this article.

There are three competing factors at play: minimising false hires, minimising false no-hire, salary.

Presumably I couldn't hire Casey for a team lead position paying £120k per year. Equally, I don't want to miss out on talent by trying to catch every edge case and making an interview process 3 weeks long.

I hired a very technically strong candidate once, he loved optimising games as a hobby. Unfortunately we were a SaaS startup and he seemed to be allergic to using prebuilt components (think SQS) because "we could build them more efficiently". It's impossible to catch every foible like this in an interview scenario.

For these reasons, ultimately there will be bad hires. The biggest mistake is leaders not being willing to fire people. Sometimes this is for fear of reducing head count, because it makes them feel like an arsehole or they aren't themselves capable/invested enough to care. It's painful but I've found it always better to fire fast.

1) If they are competent programmers.

2) Whether they will be productive at your company/project.

this is exactly what I do

also, if an interviewee doesn't know something that I think they should, I help them learn it

enough interviewers make the process unpleasant

I try to do the opposite

> Chooses a project the candidate has worked on.

> Asks questions with the goal of having the candidate teach him about his project.

I really like this. Asking candidates about their PhD thesis has not gone as well as I hoped, there's usually an increasing level of panic as they realise I read it before the interview. Asking about patches they've written to open source has the same effect.

Asking about something I can't verify changes the stress profile on it a lot. Going to change to this strategy. Thank you HN :)

As a 60+ year old guy who has been at the same employer for the past 20 years, I thankfully evaded all that leetcode crap. My interview style has always been of the sort described -- ask in depth questions about something they have worked on.

The telltale sign of a padded resume is when the person acts like a cork -- you try to push down deeper and they always pop back up to surface level answers. Some give up the game quickly and admit that they were on the team that did the thing in question, despite their resume saying they did the thing. Other people are very practiced and unflappable and just bob back up over and over without shame.

For new engineers with say three years of experience or less, I can forgive the overstating their experience and I'll shift to asking them to describe their part of that project. But for the people with a few years on their resume, then that behavior is definitely a deal breaker.

Another line of questioning I take for people with experience are questions like: what are the tradeoffs of directed testing vs random testing? When do you have enough confidence in your testing that you say the product is ready to ship? What are some examples of where bugs made it out the door, and what was your process after that happened? What are some reasons why every part of a design might pass their unit tests, but the total design has failures?

These should be easy to discuss for anyone with experience, but there are a surprising number of people who fall flat. Similarly, I am shocked at how often I ask someone to write some code in any language they want to, even pseudocode, for a simple problem and are unable to do it. Eg, given a stream of numbers, maintain the largest two numbers seen so far.

When doing technical interviewing I look for 3 things:

1) Can this person do stuff.

2) Can they learn stuff.

3) Are they interested in learning and doing the things we need them to.

The authors approach of going deep in one area is good for determining #1. You weed out all the people who "were on the team that..." or similar work-adjacent activities and find those that actually did the work / wrote the code / designed the thing / did the testing. Anyone with several years should have worked on more than one thing. Variety of things they actually did indicates an ability to learn which is #2 above. Finally you'll have to gauge how well they'll take to whatever it is you need them to work on, and that's a toss-up. This is where going to a big-name school can be useful since a big part of college is pushing through shit you don't care about for courses that aren't your thing. Otherwise a real interest in your project/product should be enough to motivate the learning and doing if they're capable.

The rest of the "team fit" stuff... You can figure out what they're like on the side while doing an hour long technical interview.

And that's my take on interviewing. I find it quite effective at finding good engineers.

> If they can answer your questions, and their answers come from solid foundations

I am currently a cloud architect, but generally speaking when I've interviewed candidates, I prefer this approach.

If you can't answer detailed questions about something you've allegedly done, you're cooked. In real life, you're not going into a situation where you can't use reference information to help you remember all the details.

> [Linear] mentioned that this approach is working very well for them, and they have achieved, at the time of writing, 96% retention.

Is this evidence that their hiring process is sound, and is it more a consequence of Linear being a rocketship? Perhaps if their retention number includes when a bad hire is let go, this is more believable that they are meeting their standards.

I’ve only worked at small startups, but usually “retention” means that no one has left for somewhere better.

It's hard to give people a copy of your code and see if they can navigate it. I find a reasonable middle ground is to create a template of the hardest bits of code. Get them to build something with it, hack it, and so on. See where they ask questions.

An algorithm takes hours to pick up. A language may take weeks. A paradigm takes months. If you use, say, functional programming everywhere, you should probably check that the applicant knows how to write functional. It's not about map(), it's about whether you code with globals and side effects. If it's reactive programming, does it properly observe and collect? If you're doing declarative UI, how do you abstract the pieces, where is the button click handled?

Some things are learnable, but this is where you see people get defensive. If you're moving in from a similar paradigm but different language, you'll find a way and you'll even contribute your own insights. I think people tend to butt heads here and it's a good headbutting simulator if you do some live coding with the team you're working with.

Most people should not be hired if he gets the needed detail. Most people don't have significant open source experience. That means for most people the projects are things they can talk about at a high level, but this interview is digging into things that any court would consider trade secrets. Even if you don't have a NDA, you still have a moral, ethical (and likely legal but check your local laws) obligation to not discuss the details of someone else's code in that level.

For others who don’t know who Casey Muratori is:

https://caseymuratori.com/about

https://en.wikipedia.org/wiki/Casey_Muratori

He's also that guy in many of ThePrimeagen's YouTube videos.

And wow, that was a great rabbit hole to go down. Interesting guy.

What about NDA's? I couldn't in good conscience drill down with someone from some other company into the projects I am currently working on.

This is exactly how I interview, and it is the most effective way I know to separate the best from the rest. To pass it the candidate must be able to recall in great detail a past project's problem, investigation, and solution. That's hard to fake from memory if you don't have direct involvement and understanding.

I'll also bounce around and go into depth on particular technologies or protocols, etc which may be relevant as well as other projects' problems and solutions. I also sprinkle in my current interests whatever they may be and have them make assessments, not to see if they agree but to see how they go about analyzing and weighing different factors.

Leetcode tests programming; interviewing about previous projects tests software engineering, or at the very least the person's understanding of it.

My problem with that is that when I get interviewed about a project, I will talk about the whole project, not just my personal contributions to it which I believe is the primary purpose of a job interview. Of course, knowing about the whole of the project is important too, but if I just have an overview and little to no contributions, I wouldn't be a valuable asset to the company I'm interviewing at.

(Not to dismiss my own contributions of course, I'm a competent engineer and can do anything - it's more a matter of what I'm enthusiastic and energised about. I wouldn't be energised by clicking around in the AWS console, but I know of it, for example).

I dunno who Casey Muratori is but I’ve followed the same principle for years! It’s a much better experience for the candidate and gives better signal provided the questioner can actually conduct a good interview and ask probing questions.

https://www.youtube.com/watch?v=gZ2V5VtwrCw&t=1620s

> Try to get as narrow as possible, even to implementation details.

I think this is a great way to tell if someone knows their craft but it could also select for people with 1) really good memory 2) really good bs. I have made lots of technical decisions that I stand by but I have a hard time remembering what I made for breakfast. I kind of have to have the code in front of me to remember those kinds of details.

This is part of the reason that I try and keep at least one personal project going at all times. I want to be able to competently speak about something technical that I am working on so so that when an interviewer asks about it I have it fresh in my mind the “gotchas” that came up when working on it.

> Whether they will be productive at your company/project.

That got a highlight in the article, but not a lot of text to explain it.

> they have to fix a bug in an unknown codebase

Creating those scenarios costs a lot of time and resources.

---

I think the LeetCode stuff is just fine. There are some issues with it:

- People can cheat by being LeetCoders that specialize on solving LeetCode, not being actual day-to-day programmers. - You might end up with a monoculture team.

Overall, though, it seems to be the safest approach.

> To play devil's advocate: difficult coding questions are a way to minimize false positives

Apparently, Fizz-Buzz is still a "difficult" coding question, considering the number of candidates that still squirm at the prospect of answering it.

I wish the HN title was "How to effectively conduct programming interviews", like it is in the article.

I find Casey Muratori completely insufferable and this title riled me up, but the content is actually pretty good. My perception of him is that he is a good engineer, but generally overconfident and unwilling to approach other peoples' viewpoints with an open mind. The current title played right into my biases.

(At the time of writing, the HN title is "Casey Muratori: I can always tell a good programmer in an interview")

> With this approach, he claims he can always understand if someone is a competent programmer, and he has never seen it fail.

I'm curious, how can he be so sure? How does he know that he has never failed a competent programmer or that he has never passed a poor programmer?

Just because he hasn't seen it fail, it doesn't mean his process hasn't failed.

I feel like I am bad at these kinds of interviews. Maybe I just haven’t built big enough systems yet (fairly new to the senior role)

More important that if it is a good programmer or not, is to be able to spot if the guy can get the job DONE.

That was not the title of the article

Interviews have been research extensively. Yet every article I've seen gives me the strong impression that nobody is even aware it exists, much less has tried to look for it (or even better yet found it!). Everyone is "this is what I think works", but nobody gives me any reason to think their system works - they haven't verified it in anyway and so it might just be luck that they have hired good people.

Most people are reasonably good, so luck doesn't seem that unlikely - someone should draw resumes at random from their pile and make an offer to whoever wins - I'm curious how selecting for people who are lucky (or who God approves of if you want to go there) compares to your process. If you cannot show me data on why your process is better than that (or something else) I have to assume you don't really know.

PLEASE, when someone talks about how to interview can they at least put forth the effort to cite real research. If you cite someone else you can tell me you think it is invalid for whatever reason, but at least show me you care enough to read it before you tell me what I should do.

Personally I don't think this topic exciting enough to dig into (scientific papers tend to be hard to read, I want "an executive summary"). But when I interview someone I limit myself to the questions my HR department tells me to ask because they are scientifically validated to be useful.

I've really liked interviews where I either present a personal project I've worked on, or get to interview someone about their own personal projects. It's just more fun.

Their major complaint of the project approach is not getting signal on adaptability to new codebases. That has never been a first concern at any company I've worked at, and frankly if engineers are touching a new codebase every month then I'm getting a bit worried.

> 1. If they are competent programmers. 2. Whether they will be productive at your company/project.

> His opinion is that the second question is much harder to answer, and he doesn't know how to reliably do it, and whether there is a way to answer it.

In my experience, I look for signs that someone is a child trapped in an adult's body. (With a lot of leeway for younger candidates.)

IE, what I'm looking for is someone who isn't going to pick silly arguments, get into pissing matches in code reviews, or argue with facts of the requirements.

---

Years ago I used to use an interview question based around the differences of XML libraries built into .Net. The point was for the candidate to demonstrate that they understood tradeoffs; but one candidate suddenly took the tone of a teenager and yelled, "I don't want to work with XML, I want to work with JSON."

They clearly failed #2. It wasn't because they didn't like XML, it was because they rejected the point of the question.

> "Another benefit of LeetCode style interviews is that they are somehow standardized. Standardized processes help large organizations stay consistent."

I may be overlooking some really important concepts here, but there I bug. It looks like the need is not for a "good programmer" (the article should define what is a good programmer btw) but for a programmer who applies to the standards for whatever it means. It is a bit chilling. And afaik those leetcode interviews tend to fade away.

> "How they navigate unknown codebases." That point seems very short-term sighted. For how long a codebase is considered as being "unknown" ?

Globally, "good" is not defined and the "scope" of the programmer's job isn't even touched, which I think changes the way you hire someone.

Anways, it was a nice read although I don't really know what to conclude. The pair programming concept is, for sure, the best I would like to experience in an interview.

The OC and the submission's title both bury the lede:

Some company named Linear conducts paid work trials with candidates.

https://linear.app/now/why-and-how-we-do-work-trials-at-line...

--

FWIW, I've always done as Muratori advises.

But I accept any kind of prior writing, in lieu of code. (I've interviewed for DBA, QA/Test, tech writing, etc. I believe, but cannot prove, that anyone who can write an essay can also code.)

Also, when possible, I let teams choose their own members, from the qualified candidates in the funnel. Per advice of Luke Hohmann and others. IIRC Facebook does this too.

But The Correct Answer™ is some kind of (paid) probationary period, with no fault divorce. Long enough to validate skills, get past the "honeymoon" period, and verify team fit.

Alas, that requires change in labor laws, universal health care, (very) cheap day care, and other misc reasonable social safety net infra.

Getting overconfident as an interviewer is exactly how you get conned. Everyone makes bad hires sometimes. No process is fool-proof, only fool-resistant.

Honestly I don't really care about a candidate's previous projects. It's fun to talk about if we have the time for sure, especially if it's immediately relevant to the role.

I want to get a glimpse into how the candidate is going to handle our tech stack, our code base, not one of their old code bases.

I still think this is very possible to learn without LeetCode (which I also disdain and never use).

Maybe it's survival bias.

They claim a way to interview is pair programming to solve a bug in a codebase unknown to the interviewee. They also claim to allow AI.

I get what they are aiming for, but I foresee trouble:

For one, this kind of interview is way harder to set up than simply asking the candidate to solve an algorithmic question (which is flawed but way simpler).

Also, it can be hard to fine tune so that it's not unfair to the candidate. Some bugs can only be solved after days of looking at the problem, so you have to iterate over this interview setup to find the right difficulty, unfairly ditching candidates in the process. And it becomes "a project"; a lot of companies cannot afford to spend much time on this.

Finally, if you're pairing how do you refrain from helping too much? You're not the one being interviewed, the candidate is! If you allow AI, how do you tweak the problem so that it's both self-contained and reasonably easy, while keeping it impervious to being one-shot by AI?

Of course, traditional interview techniques share some of these problems, but they are way easier to set up. That's what missing here, there's a cost/benefit analysis for interviewers too.

>...instead, he follows a drill-down approach where he chooses a project the candidate has worked on, asks questions with the goal of having the candidate teach him about his project.

This approach is similar to what I have always said with open source contributions to credible projects since these days, Leetcode puzzles can be solved with AI tools.

[0] https://news.ycombinator.com/item?id=45530672