Exactly. Please DO NOT mess with protocols, especially legacy critical protocols based on in-band signaling.

HTTP/1.1 was regrettably but irreversibly designed with security-critical parser alignment requirements. If two implementations disagree on whether `A:B\nC:D` contains a value for C, you can build a request smuggling gadget, leading to significant attacks. We live in a post-Postel world, only ever generate and accept CRLF in protocols that specify it, however legacy and nonsensical it might be.

(I am a massive, massive SQLite fan, but this is giving me pause about using other software by the same author, at least when networks are involved.)

> I'm hoping this is satire. Why intentionally introduce potential bugs for the sake of making a point?

It’s worse than satire. Postel’s Law is definitively wrong, at least in the context of network protocols, and delimiters, especially, MUST be precise. See, for example:

https://www.postfix.org/smtp-smuggling.html

Send exactly what the spec requires, and parse exactly as the spec requires. Do not accept garbage. And LF, where CRLF is specified, is garbage.

> I'm hoping this is satire.

Me too. It's one thing to accept single LFs in protocols that expect CRLF, but sending single LFs is a bridge to far in my opinion. I'm really surprised most of the other replies to your comment currently seem to unironically support not complying with well-established protocol specifications under the misguided notion that it will somehow make things "simpler" or "easier" for developers.

I work on Kestrel which is an HTTP server for ASP.NET Core. Kestrel didn't support LF without a CR in HTTP/1.1 request headers until .NET 7 [1]. Thankfully, I'm unaware of any widely used HTTP client that even supports sending HTTP/1.1 requests without CRLF header endings, but we did eventually get reports of custom clients that used only LFs to terminate headers.

I admit that we should have recognized a single LF as a line terminator instead of just CRLF from the beginning like the spec suggests, but people using just LF instead of CRLF in their custom clients certainly did not make things any simpler or easier for me as an HTTP server developer. Initially, we wanted to be as strict as possible when parsing request headers to avoid possible HTTP request smuggling attacks. I don't think allowing LF termination really allows for smuggling, but it is something we had to consider.

I do not support even adding the option to terminate HTTP/1.1 request/response headers with single LFs in HttpClient/Kestrel. That's just asking for problems because it's so uncommon. There are clients and servers out there that will reject headers with single LFs while they all support CRLF. And if HTTP/1.1 is still being used in 2050 (which seems like a safe bet), I guarantee most clients and servers will still use CRLF header endings. Having multiple ways to represent the exact same thing does not make a protocol simpler or easier.

[1]: https://github.com/dotnet/aspnetcore/pull/43202

Not just potential bugs, there'll be definite security failures.

Changing the line endings can invalidate signatures over plaintext content. So an email MTA, for example, could never do so. Nor most proxy implementations. Then there's the high latent potential for request smuggling, command injection, and privilege escalation, via careful crafting of ambiguous header lines or protocol commands that target less robust implementations. With some protocols, it may cause declared content sizes to be incorrect, leading to bizarre hangs, which is to say, another attack surface.

In practice, retiring CRLF can't be safely performed unilaterally or by fiat, we'll need to devise a whole new handshake to affirm that both ends are on the same page re. newline semantics.

Not disagreeing with you, but implementation diverges from spec a lot anyway.

I've had to write decoders for things like HTTP, SMTP, SIP (VoIP), and there's so many edge cases and undocumented behavior from different implementations that you have to still support.

I find that it affects text based protocols, a lot more than binary protocols. Like TLS, or RTP, to stick with the examples above, have much less divergence and are much less forgiving to broken (according to spec) implementations.

> Why intentionally introduce potential bugs for the sake of making a point?

It seems spiteful, but it strikes me as an interesting illustration of how the robustness principle could be hacked to force change. It’s a descriptivist versus prescriptivist view of standards, which is not how we typically view standards.

FYI, Sendmail accepts LF without CR, but Exchange doesn't.

It seems to me the author is not suggesting to update the protocols themselves but rather to stop sending them CR even if the spec requires it. And to patch the corresponding software to it accepts simple newlines.

> What would be the benefit

Easy - being able to use a plain text protocol as a human being without having to worry if my terminal sends the right end of line terminator. Using netcat to debug SMTP issues is actually something I do often enough.

At least for CSV, there's a divergence between usage in practice and the spec. The spec requires CRLF, but all of the commonly used tools I've encountered for reading and writing CSVs can read files with CR, LF, or CRLF line endings, and when writing CSVs they'll default to either LF or platform-specific line endings. (Even Excel for Mac doesn't default to CRLF!) I think this divergence is bad and should be fixed.

But IMO the right resolution is to update the spec so that (1) readers MUST accept any of (CR, LF, CRLF), (2) writers MUST use one of (CR, LF, CRLF), and (3) writers SHOULD use LF. Removing compatibility from existing applications to break legacy code would be asinine.

>> I'm hoping this is satire. Why intentionally introduce potential bugs for the sake of making a point?

It's not satire and it's not just trying to make a point. It's trying to make things simpler. As he says, a lot of software will accept input without the CR already, even if it's supposed to be there. But we should change the standard over time so people in 2050 can stop writing code that's more complicated (by needing to eat CR) or inserts extra characters. And never mind the 2050 part, just do it today.

>What would be the benefit...

It is interesting that you ignore the benefits the OP describes and instead present a vague and fearful characterization of the costs. Your reaction lies at the heart of cargo-culting, the maintenance of previous decisions out of sheer dread. One can do a cost-benefit analysis and decide what to do, or you can let your emotions decide. I suggest that the world is better off with the former approach. To wit, the OP notes for benefits " The extra CR serves no useful purpose. It is just a needless complication, a vexation to programmers, and a waste of bandwidth." and a mitigation of the costs "You need to search really, really hard to find a device or application that actually interprets U+000a as a true linefeed." You ignore both the benefits assertion and cost mitigating assertion entirely, which is strong evidence for your emotionality.

Thinking about it. Using CR alone in protocols actually make infinitely more sense. As that would allow use of LF in records. Which would make many use cases much simpler.

Just think about text protocols like HTTP, how much easier something like cookies would be to parse if you had CR as terminating character. And then each record separated by LF.

Copying my comment from lobste.rs in case you didn't see it:

I really appreciate this attitude. As programmers, we love to complain and grumble to each other about how the state of things suck, or that things are over complicated, but then too often the response is the software engineering equivalent of “I paid my student loans, so you should have to, too”. A new person joins the project, and WTFs at something, and the traumatized veterans say, “haha oh boy welcome, yeah everything sucks! You’ll get used to it soon.”

I hate that attitude.

We are at the very, very beginning of software protocols that could potentially last for millennia. From that perspective, you would look back at this situation and think of Richard’s blog post as super obvious, the clear voice of reason, and the reaction of everyone here as myopic.

Even if our software protocols for whatever reason don’t last that long, we need to be working on reducing global system complexity. Beauty and elegance aside, there is such a thing as complexity budget which is limited by the laws of information theory, the computer science equivalent of the laws of physics. People like Richard understand this intuitively, and actively work towards reconstructing our world to regain complexity currency so that it can be spent on more productive things.

I would have backed you 100%.

The problem with trying to legislate a specific case of Postel's Law is that everyone who might get on board because of it, is already on board because of it, and vice versa.

It has been reverted.

Nope. Git should not mess with line endings, the remote repository not matching the code in your local clone can bite you when you least expect it. On Windows, one should disable the autocrlf misfeature (git config --global core.autocrlf false) and configure their text editor to default to LF.

The letters after the dot in my filename don't map 1 to 1 with the file format.

People don't seem to mind when Chrome does it [0]. The response "standards aren't a death pact" stands out in particular.

[0] https://news.ycombinator.com/item?id=13860682

Why would you want that?

All Unix text processing tools assume that every line in a text file ends in a newline. Otherwise, it's not a text file.

There's no such thing as a "trailing newline," there is only a line-terminating newline.

I've yet to hear a convincing argument why the last line should be an exception to that extremely long-standing and well understood convention.

What's wrong with trailing newlines?

Yeah, I have no idea what the author is smoking. Deliberately breaking standards is simply not an acceptable solution to the problem, even if it were a serious problem (it's not).

It is a standard line ending. ANSI X3.4-1968 says:

10 LF (Line Feed). A format effector that advances the active position to the same character position on the next line. (Also applicable to display devices.) Where appropriate, this character may have the meaning “New Line” (NL), a format effector that advances the active position to the first character position on the next line. Use of the NL convention requires agreement between sender and recipient of data.

ASCII 1968 - https://www.rfc-editor.org/info/rfc20

ASCII 1977 - https://nvlpubs.nist.gov/nistpubs/Legacy/FIPS/fipspub1-2-197...

Counter-counterpoint: using 2 bytes to signal one relevant operation creates ambiguity out of thin air. If all you care about is "where does the line end?", having CRLF as a line ending creates edge cases for "there is only CR" and "there is only LF". Are those line endings or not? How do you deal with them? And what's LFCR?

Personally speaking, I've always written my parsers to be permissive and accept either CR¹, LF, or CRLF as line endings. And it always meant keeping a little extra boolean for "previous byte was CR" to ignore the LF to not turn CRLF into 2 line endings.

¹ CR-only was used on some ancient (m68k era?) Macintosh computers I believe.

P.S.: LFCR is 2 line endings in my parsers :D

But, like the article says, LF is not useful. I could always interpret LF as NL and if you send CRs too it won't break anything. If you know I'm interpreting LF that way you can just stop sending the CRs. That's what happened in Unix.

Yeah. It's weird how Unix picked LF given its love of terminals. CRLF is the semantically correct line ending considering terminal semantics. It's present in the terminal subsystem to this day, people just don't notice because they have OPOST output post processing enabled which automatically converts LF into CRLF.

I have always felt that somehow Linux and proponents of it default to every decision it made being right and everything else, namely Windows, being wrong. I honestly feel Linux is orders of magnitude more complex. It is much easier, in my experience to make software just work on Windows. (This is not to say Windows doesn't have bad decisions. It has many. All the major OSs are terrible.)

> (He calls LF "newline". OK, fine, his reasoning seems legit.) He is not advocating changing the language of the standard.

The Unicode standard does call it NL along with LF.

    000A  <control>
      = LINE FEED (LF)
      = new line (NL)
      = end of line (EOL)

[flagged]

Doesn’t this show that ignoring CR and only processing LFs is a good idea? If I’m understanding right (probably wrong), this vuln relied on some servers using CRLF only as endings, and others supporting both CRLF and LF.

If every server updated to line-end of LF, thereby supporting both types, this vuln wouldn’t happen?

Of course if there’s is a mixed bag then I guess this is still possible, if your server only supports CRLF. At least in that scenario you have some control over the issue though.

> There is no such character as NL,

There is, copying from a helpful comment above:

> The Unicode standard does call it NL along with LF.

    000A  <control>
      = LINE FEED (LF)
      = new line (NL)
      = end of line (EOL)

And things don't work fine, there are many issues with this historical baggage

U+0085 is sometimes called NL (it is the standard in EBCDIC), but more often NEL in the ASCII world.

> There is no such character as NL ...

More specifically the Unicode control character U+000a is, in the Unicode standard, named both LF and NL (and that comes from ASCII but in ASCII I think 0x0a was only called LF).

It literally has both names in Unicode: but LINEFEED is written in uppercase while newline is written in lowercase (not kidding you). You can all see for yourself that U+000a has both names (and eol too):

https://www.unicode.org/charts/PDF/U0000.pdf

> and the article does not at all address that fact that the "ENTER" key on every keyboard sends a CR and not a LF.

what a key on a keyboard sends doesn't matter though. What matters is what gets written to files / what is sent over the wire.

    ... $  cat > /tmp/anonymousiam<ENTER>
    <ENTER>
    <CTRL-C>

    ... $  hexdump /tmp/anonymousiam
    00000000  000a

> Things work fine the way they are.

I agree

>> instead it focuses on making people change their understanding of LF in to NL which is as unnecessary complication that will cause inevitable bikeshedding around this idea.

Not really. In order to ignore CR you need to treat LF as NL.

Indeed. Very strange to hear a break-the-world suggestion from a person leading a company famous for never breaking the world.

I'm kind of confused by this whole post.

I do understand the desire for simplification (let's ignore the argument of whether this is one), but...

Line feed is "move the cursor down one line". It's irrelevant what is currently on the line. These are printer/terminal control instructions, not text editing instructions.

If you are thinking of it being more like pressing "Home" then "Enter", it would seem that "Enter" actually works more like LFCR ?

No, it's not true.

It was used for "graphics" on character-only terminals.

isn't CR without LF how CLI progress bars work?

Nope.

Just in case... Unicode doesn't define anything about C0 control characters. Everything you see from the code chart is from ISO/IEC 6429 and only shown there for information. Some parts of Unicode and related standards do assign a special meaning to U+000A, but often also to U+000D for the obvious reason.

He is not reinventing anything. Unicode also defines 0a as LF, NL and EOL. In modern software, 0a is used as NL anyway.

https://www.unicode.org/charts/PDF/U0000.pdf

Is defined in 4 points at the end

Not sure whether this changes anything about your critique, but note that the IBM 2741 terminal embedded a Selectric typewriter:

> Selectric-based mechanisms were also widely used as terminals for computers, replacing both Teletypes and older typebar-based output devices. One popular example was the IBM 2741 terminal

https://en.wikipedia.org/wiki/IBM_Selectric

It hasn't like it never does. In this case your mistake is that number of standards doesn't change

Right, you don't need to search that hard for a device which interprets 0xA as a line feed, just set your terminal to raw mode, done.

But given the very first sentence:

> CR and NL are both useful control characters.

I'm willing to conclude that he doesn't intend A Blaste Against The Useless Appendage of Carriage Return Upon a New Line, or Line Feed As Some Style It, to apply to emulators of the old devices which make actual use of the distinction.

Conversely, I'd argue most brilliant people tend to have more dumb ideas than others, usually on oddly specific topics which most people would find inconsequential.

Even Notepad.exe supports LF only text files now.

Why do we _have to_ keep bringing this legacy baggage with us for the next decades though?

Allowing CRFL-less operation intentionally, especially in new implementations. Abusing protocol tolerance is (just a bit) to switch current ones. Should allow relatively gradual progress towards Less Legacy:tm: with basically no cost.

Not every change is "breaking your back" especially if you should be updating your systems anyways to implement other, larger and more important changes.

> Decide upon on new UTF-8 code point.

Unicode have already done so - (NEL) https://www.compart.com/en/unicode/U+0085

So break everything.

Which would need to be encoded in at least two bytes at which point, why not just use CRLF?

You mean U+2028 LINE SEPARATOR?

https://xkcd.com/927/