I agree this doens't seem too ambiguous - it's "you may do this.." and they said "or we may do the reverse". If I say you're could prefix something.. the alternative isn't that you can suffix it.

But also.. the programmers working on the software running one of the most important (end-user) DNS servers in the world:

1. Changes logic in how CNAME responses are formed

2. I assume some tests at least broke that meant they needed to be "fixed up" (y'know - "when a CNAME is queried, I expect this response")

3. No one saw these changes in test behavoir and thought "I wonder if this order is important". Or "We should research more into this", Or "Are other DNS servers changing order", Or "This should be flagged for a very gradual release".

4. Ends up in test environment for, what, a month.. nothing using getaddrinfo from glibc is being used to test this environment or anyone noticed that it was broken

Cloudflare seem to be getting into thr swing of breaking things and then being transparent. But this really reads as a fun "did you know", not a "we broke things again - please still use us".

There's no real RCA except to blame an RFC - but honestly, for a large-scale operation like there's this seems very big to slip through the cracks.

I would make a joke about South Park's oil "I'm sorry".. but they don't even seem to be

The article makes it very clear that the ambiguity arises in another phrase: “difference in ordering of the RRs in the answer section is not significant”, which is applied to an example; the problem with examples being that they are illustrative, viz. generalisable, and thus may permit reordering everywhere, and in any case, whether they should or shouldn’t becomes a matter of pragmatic context.

Which goes to show, one person’s “obvious understanding” is another’s “did they even read the entire document”.

All of which also serves to highlight the value of normative language, but that came later.

> I don't find the wording in the RFC to be that ambiguous actually.

You might not find it ambiguous but it is ambiguous and there were attempts to fix it. You can find a warmed up discussion about this topic here: https://mailarchive.ietf.org/arch/msg/dnsop/2USkYvbnSIQ8s2vf...

I agree with you, and I also think that their interpretation of example 6.2.1 in the RFC is somewhat nonsensical. It states that “The difference in ordering of the RRs in the answer section is not significant.” But from the RFC, very clearly this comment is relevant only to that particular example; it is comparing two responses and saying that in this case, the different ordering has no semantic effect.

And perhaps this is somewhat pedantic, but they also write that “RFC 1034 section 3.6 defines Resource Record Sets (RRsets) as collections of records with the same name, type, and class.” But looking at the RFC, it never defines such a term; it does say that within a “set” of RRs “associated with a particular name” the order doesn’t matter. But even if the RFC had said “associated with a particular combination of name, type, and class”, I don’t see how that could have introduced ambiguity. It specifies an exception to a general rule, so obviously if the exception doesn’t apply, then the general rule must be followed.

Anyway, Cloudflare probably know their DNS better than I do, but I did not find the article especially persuasive; I think the ambiguity is actually just a misreading, and that the RFC does require a particular ordering of CNAME records.

(ETA:) Although admittedly, while the RFC does say that CNAMEs must come before As in the answer, I don’t necessarily see any clear rule about how CNAME chains must be ordered; the RFC just says “Domain names in RRs which point at another name should always point at the primary name and not the alias ... Of course, by the robustness principle, domain software should not fail when presented with CNAME chains or loops; CNAME chains should be followed”. So actually I guess I do agree that there is some ambiguity about the responses containing CNAME chains.

Isn't this literally noted in the article? The article even points out that the RFC is from before normative words were standardized for hard requirements.

Even if 'possibly preface' is interpreted to mean CNAME RRSets should appear first there is still a broken reliance by some resolvers on the order of CNAME RRsets if there is more than one CNAME in the chain. This expectation of ordering is not promised by the relevant RFCs.

100%

I just commented the same.

It's pretty clear that the "possibly" refers to the presence of the CNAME RRs, not the ordering.

My initial reading was "you can place them wherever you want". And given that multiple parties are naturally interpreting the wording in different ways, that means the wording is ambiguous by definition.

[dead]

You just caused flashbacks of error messages from BIND of the sort "cannot have CNAME and other data", from this proximate cause, and having to explain the problem many, many times. Confusion and ambiguity of understandings have also existed since forever by people creating domain RR's (editing files) or the automated or more machined equivalents.

Related, the phrase "CNAME chains" causes vague memories of confusion surrounding the concepts of "CNAME" and casual usage of the term "alias". Without re-reading RFC1034 today, I recall that my understanding back in the day was that the "C" was for "canonical", and that the host record the CNAME itself resolved to must itself have an A record, and not be another CNAME, and I acknowledge the already discussed topic that my "must" is doing a lot of lifting there, since the RFC in question predates a normative language standard RFC itself.

So, I don't remember exactly the initial point I was trying to get at with my second paragraph; maybe there has always been some various failure modes due to varying interpretations which have only compounded with age, new blood, non-standard language being used in self-serve DNS interfaces by providers, etc which I suppose only strengthens the "ambiguity" claim. That doesn't excuse such a large critical service provider though, at all.

Is a deliberate violation of a spec really a bug? And I don't think their choice was "move fast and break things" at all.

It is a nightmare, but the spec is the source of the nightmare.

Postel's law is considered more and more harmful as the industry evolved.

That's true, but sort of misses the spirit of Hyrum's law (which is that the world is filled with obscure edge cases).

In this case the broken resolver was the one in the GNU C Library, hardly an obscure situation!

The news here is sort of buried in the story. Basically Cloudflare just didn't test this. Literally every datacenter in the world was going to fail on this change, probably including their own.

that means you did a leaky abstraction indirectly but it is on the people level

Obligatory xkcd for Hyrum's Law: https://xkcd.com/1172

I think it's more of a server side ordering, in which there were not that many DNS servers out there, and the ones that didn't keep it in order quickly changed the behavior because of interop.

CNAMES are a huge pain in the ass (as noted by DJB https://cr.yp.to/djbdns/notes.html)

It's more likely because the internet runs on a very small number of authorative server implementations which all implement this ordering quirk.

It’s not remarkable, because it’s the way all DNS servers work. Order is important in DNS results. It’s why results with multiple A records are returned in shuffled orders: because that impacts how the client interprets the results. Anyone who works with DNS regularly beyond just reading the RFCs ought to recognize this intuitively.

People probably ran into this all the time, but no single party large enough to have it gain attention produced the failure state.

If a small business or cloud app can't resolve a domain because the domain is doing something different, it's much easier to blame DNS, use another DNS server, and move on. Or maybe just go "some Linuxes can't reach my website, oh well, sucks for the 1-3%".

Cloudflare is large enough that they caused issues for millions of devices all at once, so they had to investigate.

What's unclear to me is if they bothered to send patches to broken open-source DNS resolvers to fix this issue in the future.

The last time this came up, people said that it was important to filter out unrelated address records in the answer section (with names to which the CNAME chain starting at the question name does not lead). Without the ordering constraint (or a rather low limit on the number of CNAMEs in a response), this needs a robust data structure for looking up DNS names. Most in-process stub resolvers (including the glibc one) do not implement a DNS cache, so they presently do not have a need to implement such a data structure. This is why eliminating the ordering constraint while preserving record filtering is not a simple code change.

The whole world knows this except Cloudflare who actually did know it but are now trying to pretend that they didn’t.

Some people (myself included) read that as "would ideally come first, but it is not neccessary that it comes first". The language is not clear IMHO and could be worded better.

The RFC is also 39 years old! At this point, DNS is what existing software expects it to be, not what someone proposed in the mid-eighties. The fact that they did not have any testing to match exact byte-by-byte responses with existing behavior and other DNS resolvers for this layer of service is massively irresponsible.

> People configure things similar to https://mxtoolbox.com/dmarc/dmarc-setup-cname instructions (which I find in conflict with RFC1034).

I don't think they're advising anyone create both a CNAME and TXT at the same label - but it certainly looks like that from the weird screenshot at step 5 (which doesn't match the text).

I think it's mistakenly a mish-mash of two different guides, one for 'how to use a CNAME to point to a third party DMARC service entirely' and one for 'how to host the DMARC record yourself' (irrespective of where the RUA goes).

And I'm glad they came to it. Even if everyone else is wrong (I'm not saying they are) sometimes you just have to play along.

> So clients querying a broken domain will retry each of their configured DNS servers, our caching layer (Unbound) will also retry each of their upstreams etc...

I expect this is why BIND 9 has the 'servfail-ttl' option. [0]

Turns out that there's a standards-track RFC from 1998 that explicitly permits caching SERVFAIL responses. [1] Section 8 of that document suggests that this behavior was permitted by RFC 1034 (published back in 1987).

[0] <https://bind9.readthedocs.io/en/v9.18.42/reference.html#name...>

[1] <https://www.rfc-editor.org/rfc/rfc2308#section-7.1>

DNS search is stupid by itself and shall be avoided everywhere

For you sanity, only deal with FQDNs;

re: your SERVFAIL observation, oh man did I run into this exact issue about a year or so ago when this came up for a particular zone. all I was doing was troubleshooting it on the caching server. Took me a day or two to actually look at the auth server and find out that the issue actually rooted from there.

This case would only happen if a CNAME chain first expired from the cache in the wrong order and then subsequently was queried via glibc. Theirs tests may test both that glibc resolving works and that re-querying expired records works, but not the combination of the two.

Agreed. Seems like a pretty risky optimization that fundamentally changed behavior; like it or not the ordering of vectors is often part of the data structure.

Could have just used a prepend to preserve behavior instead pf going down the rabbit hole of re-interpreting the RFC (which is a cop out IMO; it worked before, a change broke it).

The DNS specification should be updated to say CNAMES _must_ be ordered at the top rather than "possibly". Cloudflare was complying with the specification. Cisco was relying on unspecified behavior that happened to be common.

This really depends on what side of the fence you are on.

As a website host/maintainer, I am happy that the DNS 'ANY' query has been deprecated.

I am sure if you are a network engineer or ISP, then it propbably annoys you no end.

That won't cause implementations to be fixed. The implementations in question are in devices that are old (DNS is over 40 years old) and will never be upgraded. Affected users will just choose a different DNS resolver. Pretty soon word will get around that "if you don't want a broken device, don't use CloudFlare for DNS". It's less hassle for CloudFlare to just maintain the existing de-facto standard.

Is the property of an answer being ordered in the order that resolutions were performed to construct it /that/ fragile?

Randomization within the final answer RRSet is fine (and maybe even preferred in a lot of cases)

That would be a Flag Day initiative. ;-)

Honestly, it shouldn't matter. Anybody who's using a stub resolver where this matters, where /anything/ matters really, should be running their own local caching / recursing resolver. These oftentimes have options for e.g. ordering things for various reasons.

The memory involved is a kilobyte. The optimization isn't important anywhere. The fragility is what's important.

Also no, the client doesn't need more memory to parse the out-of-order response, it can take multiple passes through the kilobyte.

For most client interfaces, it's possible to just grab the addresses and ignore the CNAMEs altogether because the names do not matter, or only the name on the address record.

Of course, if the server sends unrelated address records in the answer section, that will result in incorrect data. (A simple counter can detect the end of the answer section, so it's not necessary to chase CNAMEs for section separation.)

I took it as being "we wrote the tests to the standard" and then built the code, and whoever was writing the tests didn't read that line as a testable aspect.

My reading of that statement is their test, assuming they had one, looked something like this:

    rrs = resolver.resolve('www.example.test')
    assert Record("cname1.example.test", type="CNAME") in rrs
    assert Record("192.168.0.1", type="A") in rrs

its pretty concerning that such a large organisation doesnt do any integration tests with their dns infrastructure

I don't know the official process, but as a human that sometimes reads and implements IETF RFCs, I'd appreciate updates to the original doc rather than replacing it with something brand new. Probably with some dated version history.

Otherwise I might go to consult my favorite RFC and not even know its been superseded. And if it has been superseded with a brand new doc, now I have to start from scratch again instead of reading the diff or patch notes to figure out what needs updating.

And if we must supersede, I humbly request a warning be put at the top, linking the new standard.

There already is an I-D on this topic (based on previous work): https://datatracker.ietf.org/doc/draft-jabley-dnsop-ordered-...

Or when working on massive infrastructure like this, you write plenty of tests that would have saved you a month worth of work.

They write reordering, push it and glibc tester fires, fails and you quickly discover "Crap, tests are failing and dependency (glibc) doesn't work way I thought it would."

It was glibc's resolver that failed - not exactly obscure. It wasn't properly tested or rolled out, plain and simple.

Or — hot take — to find out that you made some silly misinterpretation of the RFC that you then felt the need to retrospectively justify.

We should probably all be glad that CloudFlare doesn't have the ability to update its entire global fleet any faster than 1h 28m, even if it’s a rollback operation.

Any change to a global service like that, even a rollback (or data deployment or config change), should be released to a subset of the fleet first, monitored, and then rolled out progressively.

Given the seriousness of outages they make with instant worldwide deploys, I’m glad they took it calmly.

They had to update all the down detectors first.

Yeah, but you know, they needed to save extra bytes in the Rust implementation of their services, so wherever Rust pops up it apparently justifies any such action. ;)

It appears that some people prefer IPv4 and do not need IPv6

For example, from another story (about IP addresses) on today's HN front page:

https://news.ycombinator.com/item?id=46693867

https://news.ycombinator.com/item?id=46695198

Sometimes software written by others provides compile-time options to disable getaddrinfo or IPv6. Sometimes the software onllly uses gethostbyname

For example,

tinydns (only uses gethostbyname)

nsd (--disable-ipv6)

I compile static binaries with musl. I do not use glibc

NB. This is not code that belongs to me

When compiling software written by others, sometimes there are compile-time options that allow not using getaddrinfo or IPv6

For example,

links (--without-getaddrinfo)

haproxy (USE_GETADDRINFO="")

tnftp (--disable-ipv6)

elinks (--disable-ipv6)

wolfssl (ipv6 disabled by default)

stunnel (--disable-ipv6)

socat (--disable-ipv6)

and many more

Together with localhost TLS forward proxy I also use lots of older software that only used gethostbyname, e.g., original netcat, ucspi-tcp, libwww, original links, etc.

Generally I avoid mobile OS (corporate OS for data collection, surveillance and ad services)

Mobile data is disabled. I almost never use cellular networks for internet

Mobile sucks for internet IMHO; I have zero expectation re: speed and I cannot control what ISPs choose to do

For me, non-corporate UNIX-like OS are smaller, faster, easier to control, more interesting

O5QXGIBLGIXC4LQK

Your code runs slower on mobile devices, since (as a rule of thumb) mobile networks are ipv6-only and ipv4 traffic has to pass through a few layers of tunneling.