Instead of the zero-width div, you could set up an event listener for the copy event (using addEventListener() method) which calls .clipboardData.setData() on the ClipboardEvent to change it to your modified code.

That should avoid messing things up for people with screen readers while still trapping the copy+pasters.

There's an easier fix. Have the candidates state if they have a vision disability first and then send them down a different pathway for validation. There aren't that many, so it's not going to be costly or anything.

You also forgot people who turn off CSS and JS, or use browsers that don't support them.

> Show them the problem on a shared screen

I'm pretty sure the intent is to weed people out well before they get to a point where you could share a screen with them. He mentioned a few people "resubmitted the application", so sure this is probably an initial step.

Can't read the problem when the point is to catch those who copy-paste the code.

Sure, it's not foolproof, but a large percentage of folks would just copy&paste rather than taking the screenshot. Now that may start to change.

There was story like this on NPR recently where a professor used this method to weed out students who were using AI to answer an essay question about a book the class was assigned to read. The book mentioned nothing about Marxism, but the prof inserted unseeable text into the question such that when it was copy&pasted into an AI chat it added an extra instruction to make sure to talk about Marxism in relation to this book (which wasn't at all related to Marxism). When he got answers that talked extensively about the book in Marxist terms he knew that they had used AI.

I did the same thing. After seeing the answer I thought “this CTO is a booger eating moron”…

Good candidates are not universally good, they are good for you after hire.

One of the best business analysts I worked with (also a profession, mind you) was almost fired when working under an old, grumpy and clearly underskilled one.

I was hired once without interview into a unicorn, was loved by colleagues but hated the work, the business and the industry, then left rather quickly.

See? There are mismatches and unknown unknowns, not just bad or good developers.

Yeah sourcing developers / collaborating with developers is a huge barrier of entry. More than others factors such as deciding what software to produce for the market

Take your current technical assessment and think about the types of responses or code submissions that really impressed you. What was special about them? What did you see in the response that drew a positive reaction from your team?

Can you re-frame your process or the prompt to only elicit those specific responses?

So instead of a whole exercise of building a React app or a whole backend API, for example, what would really "wow" you if you saw the candidate do it in the submission for a project? Could you re-frame your whole process so you only target those specific responses and elicit specific outputs?

Now that you've taken what was previously a 2 hour coding exercise (for example) and distilled down to 3-4 key questions, you can seek the same outputs in 15-30 minutes instead.

There are several advantages to this:

1) Many times, candidates know the answer but they actually can't figure out what you're looking for when there's a lot of cruft around the question/problem being solved. You can end up losing candidates that know how to solve the problem the way you want, but because of the way the question was posed, the objective is opaque.

2) It saves a lot of time for both sides. Interviewer doesn't have to review a big submission, candidate doesn't have to waste time doing a long project.

3) By condensing the cycle, you can evaluate more candidates and you can hopefully select a top candidate before they get another opportunity. You shorten the cycle time because the candidate doesn't have to find free time to do a project or sit down for long interviews, you don't need to have people review the code submissions, etc.

There are quite a few people who can code but have 0 social skills and can’t talk well or at all. In that sense I have to disagree with you, but I still wouldn’t hire them because they drag teams down.

> into an AI solver

Not just an "AI solver", a Python interpreter will also do it "wrong". The idea is that it's so simple that anyone qualified should be able to do it in their head, so they get the answer without the equals sign (but IMO a qualified applicant might also know it takes 5 seconds to run it in the repl and it'd be better to be correct than to use the fewest tools, or might be using a screen reader).

I'm one of those hit by the reader mode "bug". I really wouldn't want the mode to give me anything beyond raw text+images, as that can be abused.

Other way around.. if you use reader mode you generate the wrong answer ("tricked" like an LLM). At least, it's wrong according to the author/a former CTO.

It also excludes users of Lynx, cURL, possibly people using accessibility tools, those with custom/overriding style sheets.

The trick is that the = has CSS styling with "opacity: 0; font-size: 1px;".

In normal mode the question is different than in reader mode, or when copied.

Thus, if you get the wrong answer, you "cheated" (or used reader mode)

Oh now it works, ok I got it right, I am hired

You're right. I agree.

It depends why they didn't get it "correct" (asked ChatGPT bad, used Python REPL not so bad, used screen reader very not bad) and what "correct" even means for this problem.

There's a bizarro version of this guy who rejects people who do it in their head because they weren't told to not use an interpreter and he values them using the tools available to solve a problem. In his mind, the = is definitely part of the code, you should have double checked.

That's cheating (even if it just assures you that your answer is correct)

yes, they were (theoretically) a python developer, should have mentioned this was an ML role (your guess is right, slice just before the last char)

Just to be clear: the main problem is not that they did not know what `:-1` was - there are many weird syntax additions with every version - understandable.

IMHO the problem is that there's usually a single interviewer that decides go/no go.

We all have biases, so leaving such an important decision (like hiring an EM) to one person is, (again IMHO) ...stupid .

I do think running the code would be a tiny bit faster, even if it's merely seconds either way. Opening a python REPL and pasting that would take around 5 seconds in my case. Running the code in my head would take roughly the same at first, but then if it's in an interview I'd take the time to double check. And then check a few more times because I'd expect some kind of trick here.

Considering there's no (explicit) instruction forbidding or discouraging it, I'd consider the REPL solution to be perfectly valid. In fact some interview tests specifically look for this kind of problem solving.

I get it still, I'd expect some valuable signal from this test. Candidates who execute this code are likely to do so because they really want to avoid running the code in their head, not just because it's more straightforward, and that's probably a bad sign. And pasting that into an LLM instead of a REPL would be a massive red flag.

I just don't think answering "-11" here is a signal strong enough to disqualify candidates on its own.

If you're looking for junior-ish python devs, I'd expect a good chunk of the better ones to have a python repl open and ready just as a matter of habit.

So for them, yes, it would clearly be faster to run the code than to work through it manually.

What you're doing here is selecting for candidates who are less comfortable with using the tools that they'd be expected to use every day in the role you're hiring for. It's likely to provide a negative signal.

I can run the code in my head. I can probably be right. I can be 99.9999% sure I am right.

OR:

I could run the code in the interpreter and be 100% certain.

I know what attitude I would prefer out of my developers.

I did it in my head but got 4 (3 * 3 - 5), so I fail, too. Hopefully I'd be paying closer attention if I was actually applying for a job.

Presumably you'd get it if you short-circuited and didn't evaluate the else part.

I copy/pasted it into my editor to see it larger. I still used my brain. This 'test' is not a true test but a trick, and the trick doesn't have to do with programming ability.

Why would I copy paste the code somewhere?

It's a tiny code snippet that I was going to evaluate visually on the author's website.