"where we might as well save the bandwidth"

I come from a world (yesteryear) where a computer had 1KB of RAM (ZX80). I've used links with modems rocking 1200 bps (1200 bits per second). I recall US Robotics modems getting to speeds of 56K - well that was mostly a fib worse than MS doing QA these days. Ooh I could chat with some bloke from Novell on Compuserve.

In 1994ish I was asked to look into this fancy new world wide web thing on the internet. I was working at a UK military college as an IT bod, I was 24. I had a Windows 3.1 PC. I telnetted into a local VAX, then onto the X25 PAD. I used JANET to get to somewhere in the US (NIST) and from there to Switzerland to where this www thing started off. I was using telnet and WAIS and Gopher and then I was apparently using something called "www".

I described this www thing as "a bit wank", which shows what a visionary I am!

It sort of already is. This behavior is only applied to sessions with a TTY and then the client can disable it, which is a sensible default. This specific use case is tripping it up obviously since the server knows ahead of time that the connection is not important enough to obfuscate and this isn't a typical terminal session, but in almost any other scenario there is no way to make that determination and the client expects its ObscureKeystrokeTiming to be honored.

Yes, but I wouldn't be surprised if the change is rejected. The crypto library is very opinionated, you're also not allowed to configure the order of TLS cipher suites, for example.

It's not just the pointless chaff, the SSH protocol is inherently very chatty, and SFTP even more so. The solution, for a high-performance game, is don't use SSH. Either run it over Wireguard or grab some standard crypto library and encrypt the packets yourself. You'll probably make a few minor mistakes but unless the other player is the NSA it'll be good enough.

For that matter, why does it need to be encrypted at all? What's the threat model?

If there really is a genuine need to encrypt and low latency is critical, consider using a stream cipher mode like AES-CTR to pregenerate keystream at times when the CPU is lightly loaded. Then when you need to encrypt (say) 128 bytes you peel off that many bytes of keystream and encrypt at close to zero cost. Just remember to also MAC the encrypted data, since AES-CTR provides zero integrity protection.

Threats exist in both trusted and untrusted environments though.

This feels like a really niche use case for SSH. Exposing this more broadly could lead to set-it-and-forget-it scenarios and ultimately make someone less secure.

+1... Given how much SSH is used for computer-to-computer communication it seems like there really should be a way to disable this when it isn't necessary.

The guy in charge of Go's security decreed that SSL 1.3 (which he was a contributor to) was so secure that silly programmers should not be able to override what algorithms are allowed or not allowed, because why would they possibly need to do that, because he's such a genius, and even if someone DID find a security vulnerability, well....they can just wait for Google to publicly disclose it and release a patch, compile the new version, update their code to work with that version of Go, rebuild their containers, put stuff through testing, and then release it into production.

Versus...seeing there's a vulnerability, someone adding a one-line change to disable the vulnerable algorithm, compile, image update, test. And a lot less testing because you're not moving to a new version of the language / compiler.

The man has no practical experience in running a production network service, an ego the size of a small moon, and yet was a major contributor to a security protocol now in use by billions of people.

But hey, you can be a handbag designer and end up head of design at Apple soooooooo

Relying on not advertising some feature for it is very janky way to do it.

The proper fix would be adding option server-side to signal client it's not needed and have client side have option to accept or warn about that

It made the front page when it was added.

https://news.ycombinator.com/item?id=37307708

Sadly, much fewer computer systems have telnet nowadays.

Also, port 21 is often blocked.

Even if the chatbot served only as a Rubber Ducky [1], that's already valuable.

I've used Claude for debugging system behavior, and I kind of agree with the author. While Claude isn't always directly helpful (hallucinations remain, or at least outdated information), it helps me 1) spell out my understanding of the system (see [1]) and 2) help me keep momentum by supplying tasks.

[1] https://en.wikipedia.org/wiki/Rubber_duck_debugging

You’re absolutely right!

Claude is much faster at extracting fields from a pcap and processing them with awk than I am!

AIs are exceptional at sensing personalities from text. Claude nailed it here, the author felt so good about the "holy cow" comments that he even included them in the blog post. I'm not just poking this, but saying that the bots are fantastic sycophants.

It's like I keep saying, it probably wasn't a good idea to give our development tools Genuine People Personalities...

I think the blog post itself was partially written with LLMs, which explains some of the odd narrative style.

Unfortunate result of the security theater.. "Someone who has access to run privileged application can run side channel attacks! Let's drop cpu performance 20 percent over the world"

Memories! I was at the hacking conference HAL2001 and listening to Dug Song and Solar Designer, who were talking about their SSH timing analysis: https://download.openwall.net/pub/advisories/OW-003-ssh-traf...

Time flies

Haven't heard of TCP_CORK, very interesting.

For people who don't feel like googling it:

1. You TCP_CORK a socket

2. You put data into it and the kernel buffers it

3. If you uncork the socket, or if the buffer hits MSS, the kernel sends the packet

Basically, the kernel waits until it has a full packet worth of data, or until you say you don't have any more data to send, and then it sends. Sort of an extreme TCP_YESDELAY.

See https://catonmat.net/tcp-cork for where I learned it all from.

Oh wow - I've never heard of TCP_CORK before. Without disabling pings I'd still pay the cost of receiving way more packets, but maybe that'd be tolerable if I didn't have to send so many pongs. This is super handy; excited to play around with it.

I am aware of TCP_NODELAY (funny enough I recently posted about TCP_NODELAY to HN[1] when I was thinking about it for the same game that I wrote about here). But I think the latency hit from disabling it just doesn't work for me.

[1] https://news.ycombinator.com/item?id=46359120

Can you explain how TCP_CORK helps here? The chaff packets are spaced 20ms apart and sent per socket, so I don’t see how TCP_CORK could help unless it coalesced across 20ms intervals? But coalescing is clearly not an option for the intended obfuscation effect of the original feature.

I think those all have reasonable counterarguments:

(1) This sounds brittle. Are you really going to have a good mental model about what's secret when using ssh and reliably refrain from typing those things? Seems to kinda defeat the idea of securing the channel. Also, as a collection your activities might be more confidential to you than single inputs, or correlated with your other activities outside ssh, etc - it's hard to keep a mental model of this as well. Aka optimism is not a form of security.

(2) There isn't a reason to think this is a difficult attack that only a powerful adversary could mount. Seems like a college lab level thing to me. And very amenable to AI help as well. Also here optimism is not a form of security. It's a 25 year old attack[1] so there's a lot of existing research[2] around.

(3) Saving 100x bandwidth on single keystrokes on an internet dominated by video traffic just because it's 100x doesn't make sense. Also it's good to cultivate a mindset that steers away from trading off security in favour of trivial resource savings.

[1] https://www.usenix.org/conference/10th-usenix-security-sympo... (probably older stuff exists outside open literature)

[2] eg https://crzphil.github.io/posts/ssh-obfuscation-bypass/

There's no way to know in advance if some leaked cleartext will provide enough information to an attacker to be useful. Attackers profit from making creative use of information they didn't have before.

That said, plenty of people disable the most useful security features of SSH, like verifying host key signatures, with no ill-effects (as far as they know). For the majority of users, using Telnet and unencrypted HTTP would make no difference, as nobody's trying to hack them, and who really cares about privacy anyway?

Did you know SSH has long-standing performance limitations due to its design that need patches to eliminate? It was never intended to be a high-performance tool. If you want really high performance, use Telnet. If you want real security, use SSH with all strong security options enabled plus a server using ContainerSSH with the OAuth2 plugin (SSH's keys are static, which can be captured and reused, which is bad). If you don't care either way, use SSH with the defaults.

> I'm pretty much never typing secrets into an SSH tunnel; these days if there's a secret I need to transmit over SSH I'm going to be copying and pasting it, which will not reveal info from keyboard timing

One common secret that goes through a tty ssh connection is a sudo password. You are probably typing sudo command so without obfuscation the attacker can find out the sudo keystrokes, the command keystrokes and then the encrypted bytes of the password. They don't have the timing data to decode them as easily as the previous parts but if they record enough traffic they might be able to decrypt the password. But maybe they won't, because the ssh session key is probably different each time. Furthermore I don't know how many times they should capture your encrypted password to be able to decrypt it. Maybe it's unfeasible.

Anyway, in case of the sudo password, if the attacker gets it what would happen? The attacker is hopefully not able to get a shell into the server. If they do they have different ways to get root privileges.

By the way, I also copy and paste secrets from either the password manager or the clipboard, because nobody remembers long random strings. The only exceptions are the passwords of a few accounts.

Typing with an extra 50ms latency will be fairly unpleasant.

> send all typed characters at 50ms intervals

Wouldn't this just change the packet interval from 20ms to 50ms? Or did you mean a constant stream of packets at 50ms intervals, nonstop?

I think the idea behind the current implementation is that the keystrokes are batched in 20ms intervals, with the optimization that a sufficiently long silence stops the chaff stream, so the keystroke timing is obfucated with an increased error bar of 20ms multiplied by number of chaff packets.

I'm anti-LLM in most cases, but:

> I bet this mystery could gave been solved much quicker by simply looking at the packet capture in Wireshark.

For some people who are used to using Wireshark and who know what to look for, probably yes. For the vast majority of even technical people, probably not.

In my case, I did a packet capture of a single keystroke using tcpdump and imported it into Wireshark and I get just over 200 'Client: encrypted packet' and 'Server: encrypted packet' entries. Nothing useful there at all. If I tcpdump the entire SSH connection setup from scratch I get just as much useful information - nothing - but, oddly, fewer packets than my one keystroke triggered.

So yeah, I dislike LLMs entirely and dislike the reliance on LLMs that we see today, but in this case the author learned a lot of interesting stuff and shared it with us, whereas without LLMs he might have just shrugged and moved on.

Unfortunately with SSH specifically, the dissectors aren't very mature - you only get valid parsing up to the KeX completion messages (NEWKEYS), and after that, even if the encryption is set to `none` via custom patches, the rest of the message flow is not parsed.

Seems because dumping the session keys is not at all a common thing. It's just a matter of effort though - if someone put in the time to improve the SSH story for dissectors, most of the groundwork is there.

Sure it could have been, if you knew about SSH packet inspectors in Wireshark...

The author didn't, and used a general tool to their aid - why is that unfortunate?

Hey! I'm the author.

My thinking was:

  * Yes, I clearly know what tcpdump is / how to capture network traffic
  * It has been several years since I have looked at a pcap
  * I don't have wireshark installed on this computer
  * I've done the thing where you decrypt TLS with wireshark exactly once, years ago, and I found it frustrating for reasons I can't remember[1]. Wasn't sure if I could do this with ssh
  * When I started investigating this, I didn't remotely think that ssh was the root cause. I thought it was a quirk of my game
  * I *did* make a client that printed out all the data it was receiving, but it was useless because it was operating at the wrong layer (e.g. it connected over SSH and logged the bytes SSH handed it)
  * I'm experimenting with Claude Code a lot because it has a lot of hype and I would like to form an opinion
  * Looking up flags is annoying
  * Being able to tell an agent "look at this pcap and tell me what you see" is *cool*

And the big leap here was realizing that this was my SSH client and not a quirk of my game. The time at which I would have read the SSH spec was after I captured traffic from a regular SSH session and observed the same pattern; before that I was thinking about the problem wrong.

I don't think that this is unfortunate. In fact, I think I got what I wanted here (a better sense of Claude Code's strengths and weaknesses). You're right that an alternative approach would have taught me different things, and that's a worthy goal too.

[1] I suspect this is because I was doing it for an old job and I had to figure out how to run some application with keys I controlled? It would have been easier here. I don't remember.

obviously OPs empirical and analytical rigor are top notch. He applied LLMs in the best way possible: fill gaps with clumsy command line flags or protocol implementations. Those aren't things one needs to keep in their head all the time.

Asking an LLM about SSH (hint: the two S-es stand for security) would tell you why only having packet capture in Wireshark isn't going to reveal shit.

Way to gatekeep. God forbid people use tools to help them investigate instead of knowing the exact approach to take.

How much are you staking on that bet?

I seem to hit this logic often recently for some reason.

There are two issues with it:

- a primary is not a totality: if "security is the #1 consideration for SSH", that implies there's a #2, maybe even a #3 and so on consideration. So the question that follows becomes tautological: "but if the author doesn't need security, why use ssh?" -> surely for one or more of the #2, #3, etc. considerations, right?

- overabstraction (*): you ended up strawmanning the author. What they had issue with was keystroke timing obfuscation, which is a privacy feature. Timing attacks are (in part) a privacy concern, and privacy is a security concern, yes, but security is not just privacy concern, and privacy concerns are not just about timing attacks; these groups are not equal. For example, they might very well want the transmitted keypresses themselves to remain confidential, or they might very well want to retain cryptographic assurance of their integrity. These are security features they can continue to utilize by sticking with SSH.

All of this is to say, it's not even necessarily them using SSH for a hypothetical #2 or #3 (...etc...) reason, but likely because they still very much want to make use of large chunks of #1, which disabling keypress obfuscation does not actually rid SSH of, only at most weakens it in ways they clearly seem to be okay with.

(*) although if I zoom out enough, this is once again just "a primary is not a totality", just implicitly

> For example, "nc" (netcat) is pre-installed on all platforms where ssh is.

This is technically incorrect, because Windows now includes SSH too!

Depends on what kind of security. They might care about connection integrity. If a faulty (or malicious) router in-between client and server starts malforming packets, `nc` will display those malformed packets. SSH will only show you what the server intended, or nothing.

the obtuseness is the point! This is true of a lot of my work[1][2][3].

The problems you run into when doing things you shouldn't do are often really fun.

[1] https://news.ycombinator.com/item?id=42342382

[2] https://news.ycombinator.com/item?id=37810144

[3] https://news.ycombinator.com/item?id=42674116

Yea UDP is technically more performant, but then you need a crypto layer + reliable message delivery layer + bespoke client. Using a plain old SSH client is cool.

However, there are existing libraries for exactly this use case - see https://github.com/ValveSoftware/GameNetworkingSockets

I guess QUIC libraries would also work.

Yes! While this post was written entirely by me, I wouldn't be surprised if I had "smoking gun" ready to go because I spent so much time debugging with Claude last night.

ChatGPT too. And "lines up perfectly" when it doesnt actually line up with anything

Or the "Eureka! That's not just a smoking gun, it's a classic case of LLMspeak."

Grok, ChatGPT, and Claude all have these tics, and even the pro versions will use their signature phrases multiple times in an answer. I have to wonder if it's deliberate, to make detecting AI easier?

I've love to delve into that.

https://pshapira.net/2024/03/31/delving-into-delve/

Without knowing how LLM's personality tuning works, I'd just hazard a guess that the excitability (tendency to use excided phrases) is turned up. "smoking gun" must be highly rated as a term of excitability. This should apply to other phrases like "outstanding!" or "good find!" "You're right!" etc.

I'm working on a little SRE agent to pre-load tickets with information to help our on-call and I'm already tired of Claude finding 'smoking guns'.

smoking gun, you're absolutely right, good question, em dash, "it isn't just foo, it's also bar", real honest truth, brutal truth, underscores the issue, delves into, more em dashes, <20 different hr/corporate/cringe phrases>.

It's nauseating.

You might see certain phrases and mdashes ;-) rather often, because … these programs are trained on data written by people (or Microsoft's spelling correction) which overused them in the last n years? So what should these poor LLMs generate instead?

At this I'm just so glad that "you're absolutely right!" phase is over.

They love clichés, and hate repeating the same words for something (repetition penalty) so they'll say something like "cause" then it's a "smoking gun" then it's something else

I don't think claude has even once used this in my conversations (Claude Desktop, Claude Code, Voice conversations...) Sycophancy, yes absolutely!

Maybe it has something to do with your profile/memories?

It's a smoking gun of Claude usage.

Yes, it’s kind of a corpus delicti. ;)

I see it from GPT5 too a lot

> Speaking of smoking guns

Oh shoot! A shooting.

So the TL;DR of this post is: don't change this setting unless you know what you're doing.

Chastise it with a reminder that you're using smokeless powder.

> Or you could use anycasting to terminate SSH sessions on the moral equivalent of one of a number of geography based reverse proxies and then forward the packet over an internal network to the app server over a link tuned for low latency.

I've been thinking about some stuff like this! Not being able to put my game behind Cloudflare[1] is a bummer. Substantial architectural overhead though.

> The idea of letting Claude loose on my crypto[graphy] implementation is about the most frightening thing I've heard of in a while [though libnss is so craptastic, I can't see how it would hurt in that case.]

I hear you, but FWIW the patch I was reverting was trivial (and it's also in the go crypto library, which is pretty easy to read). It's a couple-of-line change[2], and Claude did almost exactly what I would have done (I was tired and would have forgotten to shrink the handshake payload).

[1] This isn't strictly true, Cloudflare spectrum exists, but its pricing is an insane $1/GB last I checked.

[2] https://cs.opensource.google/go/x/crypto/+/833695f0a57b30373...

https://news.ycombinator.com/item?id=37307708

2023 discussion about it here.

The author wanted people to be able to just "ssh mygame", no? In that sense, ssh was a design requirement.

> It is trivial to do 10x that packet per second rate.

When making this statement, are you taking into account that SSH encrypts the traffic by default?

Jitter could be filtered out, I presume.

You surely mean the latency in its embedded terminal and not the code editor, right? I use VSCode’s remote SSH specifically so that code editing doesn’t suck. It really does not.

Is it possible that this is on your end?

The extension is "ping@openssh.com." It shows up in the blog reliably for me across several browsers and devices.

Dont let this article be blinders to you. Ssh does much more than obfuscate keypress timings. Not needing the chaff means turn it off and keep all the other benefits. It doesn't mean "revert to telnet"

Your security is safe. This game isn’t going to cause SSH to degrade.

I assume this is done for novelty. There is also Terminal coffee which is a coffee company that takes orders for delivery over SSH.

That would require end users to install additional software though, which they do not want

Telnet nowadays typically isn’t available by default for security reasons, and OP wants people to be able to play the game just by typing “ssh thegamehost”.

On Debian 13 I get a bunch when just typing interactively on shell instance

That doesn't sound right to me. This obfuscation isn't about a side-channel on a crypto implementation, this is about literally when your keystrokes happen. In the right circumstances, keystroke timing can reduce the search space for bruteforcing a password [1] but it's overstating to describe that as broken encryption.

[1] https://people.eecs.berkeley.edu/~daw/papers/ssh-use01.pdf

They literally explain the mechanism in the post and then explain why the security tradeoff made sense for their ssh game………

It is to prevent timing attacks but there are many ssh use cases where it is 100% computer to computer communications where there is no key based timing attack possible.

The fix seems kind of crazy though, adding so much traffic overhead to every ssh session. I assume there's a reason they didn't go that route, but on a first pass seems weird they didn't just buffer password strokes to be sent in one packet, or just add some artificial timing jitter to each keystroke.

But they'd have to be on the same network as me to do that attack, right?

I agree with your general point that most companies/projects do a terrible job optimizing for slow computers/networks, but OpenSSH is from the OpenBSD people, who are well-known for supporting ancient hardware [0]. Picking a random architecture, they fully support a system with only 64MB of memory [1], and the base install includes SSH. So I suspect that OpenSSH is fairly well tested on crappy computers/networks.

[0]: https://www.openbsd.org/plat.html

[1]: https://www.openbsd.org/landisk.html#hardware

There's a good chance you have other options. Regardless of how you feel about the company's head, Starlink would probably be one of them, with likely better performance than you're dealing with on ADSL.

But you cannot just sue a company because their network connected software doesn't work well on slow networks. Let alone a project like OpenSSH. It would be like me suing a game studio because my PC doesn't meet their listed minimum requirements to play the game.

You're not ok with a security/privacy tool using defensive techniques because of ... the lack of fiber in Africa?

The openssh team does not owe you anything.

If you want a “1990s” mode, add it yourself or pay some to do it for you.

> One thing you notice if you have ADSL

This is funny to me, because ADSL used to be the fast thing, as opposed to dialup modems.

You just opened a huge nostalgia portal, never thought that Dreamweaver would still be around, I used that somewhere around 2003 I believe. Good memories

Amen brother

It's vibe coded

You'd think the cover traffic would automatically cut out once the connection reached a certain rate though.

Just think of the trees burnt in the name of security!

Why not? If it's high-performance, it's fine.