Why We're Having to End Our Direct Peering Relationship with Deutsche Telekom

37 pointsposted 9 months ago
by omnibrain
(about.fb.com)

45 Comments

steilpass

9 months ago

Deutsche Telekom’s view https://www.telekom.com/en/company/details/meta-is-not-above...

edward28

9 months ago

It looks like DT is complaining meta doesn't want to pay even though they don't connect directly anymore?

immibis

9 months ago

Exactly correct. A lot of business in Germany is like this, passive-aggressive money taking.

DT is saying they had an agreement of interconnection for payment, Facebook stopped paying, a court said it has to keep paying, and Facebook decided to end the interconnection instead of continuing to pay. DT, in typical German fashion, insists on continuing to receive payment. Because it's Germany, they'll probably go back to court and be able to win another case and forcibly extract payment even though the service is no longer provided.

squigz

9 months ago

> DT is saying they had an agreement of interconnection for payment, Facebook stopped paying, a court said it has to keep paying, and Facebook decided to end the interconnection instead of continuing to pay.

This (and the DT article) seems to imply Meta stopped paying without ending the interconnect. That seems like a reasonable reason to want to be paid?

throw0101d

9 months ago

> This (and the DT article) seems to imply Meta stopped paying without ending the interconnect. That seems like a reasonable reason to want to be paid?

It takes two to keep a BGP peering session up: if DT didn't want the traffic without being paid they could have shut things down on their end too:

* https://www.juniper.net/documentation/us/en/software/junos/c...

* https://www.cisco.com/en/US/docs/ios-xml/ios/iproute_bgp/con...

greyface-

9 months ago

From DT's response:

> the company feeds a gigantic 3.5 terabytes into Deutsche Telekom's network [...] Deutsche Telekom has done everything in its power to ensure smooth data traffic. [...] The start of the rerouting of data traffic in the night from Tuesday to Wednesday went smoothly.

It sounds to me like DT was not prepared to handle the volume of this traffic along the alternate paths it would have taken if Meta's paid transit sessions were removed, so in the interest of avoiding disruption of their network, kept them running temporarily, settlement-free, until mitigation was completed. Great deal for Meta while it lasted!

throw0101d

9 months ago

> Great deal for Meta while it lasted!

The word "deal" implies some kind of (commercial) transaction that has a price tag associated with it (deal = good price).

But DT keeping this peering connection up should be (IMHO) DT just doing the job they're paid for.

DT's customers pay for "Internet access", and since Meta is on the Internet, the customers should be able to connect to it. Customer's send requests to Meta, DT passes those requests along, and it is DT's job to pass along Meta's responses.

Why should Meta pay DT anything? DT is claiming to offer a service, and part of that service is transporting the responses to DT's customer's requests. If those responses are high bandwidth it is part of DT's job to deal with it. If 'dealing with it' entails special connections to particular corners of the Internet to better serve their customer's then it is DT's job to do that.

greyface-

9 months ago

> The word "deal" implies some kind of (commercial) transaction that has a price tag associated with it (deal = good price).

The value that Meta received is equivalent to what Meta is now paying their third-party transit provider to transit this same volume of traffic to DT. My napkin math suggests that this is tens to hundreds of thousands of dollars per month, at 3.5Tbps and pennies per megabit.

> DT's customers pay for "Internet access", and since Meta is on the Internet, the customers should be able to connect to it.

What do you mean by "on the Internet"? Meta is "on the Internet" by virtue of paying transit providers to propagate their IP prefixes within the DFZ.

You can't just show up at an IXP or other peering location, announce your routes, and expect them to propagate globally. You need to convince at least one transit-free network to carry them before you're truly "on the Internet". Meta used to pay DT for transit, and now they don't. Meta is now "on the Internet" via third-party transit, rather than directly via DT.

> Why should Meta pay DT anything?

Meta doesn't need to pay DT anything, but if they're not paying, they shouldn't expect free connectivity services from DT. This traffic can now flow between DT and Meta via transit, instead of directly between DT and Meta.

> If 'dealing with it' entails special connections to particular corners of the Internet to better serve their customer's then it is DT's job to do that.

Which is exactly what DT did by providing settlement-free peering, temporarily, while their links with other ISPs, who they do have a business relationship with, could be beefed up. Now those links can handle the traffic, and now the traffic is flowing over them rather than over direct peering with Meta. No traffic was dropped in the process; neither DT's customers nor Meta were or are being harmed.

user

9 months ago

[deleted]

stop50

9 months ago

The Deutsche Telekom didn't force Meta to use the private interconnecT. They kept it open after meta wanted 40% off the previous price and the talks broke up.

mananaysiempre

9 months ago

TFA manages to say everything except what actually happened and why it’s this particular peering negotiation they decided to whine about. (Settlement-free peering is cutthroat at the best of times, as best as I can understand from the outside, so it is unclear why this is even news.) So could you elaborate on the actual history here?

nicholasbraker

9 months ago

What do you mean with "cutthroat"?

mananaysiempre

9 months ago

I mean stuff like “we agreed on settlement-free peering at a time when you were sending me about as much as I you, but now you’re sending me 20% more / your traffic is small enough I figure I can pressure you / etc. so the deal’s off, you pay me now.” That’s what I’ve heard, anyway—I’m sure there are people maintaining actual tier-1 networks lurking around here.

immibis

9 months ago

Tier-1 networks have very clear requirements for settlement-free peering, and only do it with other tier 1 networks. This is about tier 2.

alias_neo

9 months ago

"but given the court ruling concerning the unprecedented and unacceptable fees demanded"

Sounds like Meta threw its toys out of the pram because German courts didn't bow to their demands. I'd love to hear more details on the "real" story, and not Meta's PR bullshit.

stop50

9 months ago

I only have an german article: https://www.golem.de/news/kabel-nicht-gezogen-meta-muss-tele...

alias_neo

9 months ago

Thanks. The article sounds like Meta wanted 40% off, DT offered 16%, they couldn't agree. DT allowed the traffic to continue while it waited for an agreement to be made and Meta abused it, then said something along the lines of "well, they've been letting us send traffic for free so...", and the court told Meta to fuck off, because that wasn't the terms of the previous contract.

rich_sasha

9 months ago

What is the actual dispute? I can't isolate it out from the whine of Meta saying DT are baddies (which maybe they are but I can't find where they say that).

lutoma

9 months ago

I don't know about this specific case but Deutsche Telekom has a history of highly shady behavior when it comes to peering (or lack thereof). Things like intentionally running ports at public exchanges below capacity so they're always congested and then attempting to bully content providers into paying outrageous sums for peering (even though that traffic is already paid for by their customers, so really they're just trying to double dip. Peering is usually a mutually beneficial affair with no exchange of money).

For a very long time, there were substantial issues with getting traffic from Hetzner to Deutsche Telekom for similar reasons. At the time, when you rented a server at Hetzner you could actually pay extra to get 'premium' routing to Deutsche Telekom though Core Backbone (AS33891), which did make the ransom payments (unlike Hetzner itself).

They were also hard at work undermining net neutrality for a while by zero-rating some companies on their mobile network until they were stopped by courts.

I'm not sure what's going on in this specific case, but I'm not inclined to give them the benefit of the doubt.

bilekas

9 months ago

It's not clear from this statement, but it sounds like FB are alleging that DT are limiting access to FB apps and services to clients who are DT customers because FB are not paying DT ?

This would go against Net neutrality principles but I don't see any facts which support those claims by FB. So it's not clear what this is all about when it seems it broke down during contract renegotians.

This seems dubious though given the German market is extremely well regulated. Also I've never heard of a provider requesting payments for delivering externally hosted services. Maybe some special cases might exists for things like Netflix bandwidth tunnels with ISP's but that's usually a request from the service provider to have special traffic lanes.

Edit : DT customers are being limited.

rsingel

9 months ago

It's not dubious. That's exactly what DT has been doing for a long time. DT is partly owned by the German government, which might explain why it's been allowed to be a bully.

It ran this same shakedown routine on an academic network in the middle of the pandemic.

It's run this against Hetzner.

It's the same playbook that Comcast/Verizon/Time Warner used vs Netflix/Cogent/League of Legends in 2014.

https://cyberlaw.stanford.edu/blog/2024/09/a-deutsche-teleko..."

dathinab

9 months ago

they don't really say, they just say direct peering important, telecom bad, should not affect customers with a lot of words

sshine

9 months ago

tl;dr:

Meta is ending its direct peering relationship with Deutsche Telekom due to the company's actions that undermine net neutrality and put the open internet at risk.

Deutsche Telekom is using its market power to potentially restrict its subscribers' access to internet services that do not pay the company additional fees, creating a de facto paywall.

Importantly: This is Facebook complaining, hoping to pressure Deutche Telekom through public humiliation. They have peering agreements elsewhere where strict access to Facebook without open Internet access is allowed. So if shutting down open Internet access benefits Facebook, they're happy.

dathinab

9 months ago

> Meta is ending its direct peering relationship with Deutsche Telekom due to the company's actions that undermine net neutrality and put the open internet at risk. [..]

This isn't quite correct, and thats why I was complaining about the article missing essential information because it's really misleading and makes you think that that is what is happening.

It turns out what happens is Meta wanted is a price cut of 40% on a contract with DT, DT was only open for 16% and decided to move to a different provider. Also (and here other sources where also a bit unclear) there was some curt ruling which boils down to yes the prices from DT are legal and Meta has to pay if they continue to use it (which they did while trying to get a 40% price cut). What is unclear here is who sued for what. FB trying to force lower prices by ab(?) using net neutrality laws? DT suing because FB not paying for continous usage of the service while discussing a new contract? Something else? Did they sue or was it a declaratory action? etc.

Most important this might not have as much to do with net neutrality as it seems.

Net neutrality is about not unfairly prioritizing some traffic over other traffic. It's also somewhat about you not having to pay every network provider your traffic goes through. But it is _not_ about content providers not paying at all for network traffic, they still pay the provider which connects them to the internet for traffic the same way you as a private person do (just probably much less).

And that is where they wanted a 40% price cut as far as I can tell. And that is also where they then moved to a different provider by now and where they don't have to pay DT for traffic flowing from their new provider through the DT network to customers (as far as I can tell).

So if you look at it from that perspective it's FB trying to abuse net neutrality by labeling traffic from their servers as transit traffic (but also some articles say the judges said it doesn't matter weather it's transit traffic or not in that case, which makes it even more unclear what exactly the curt case was about, probably some contractual clauses in their previous contract in which case it probably really doesn't matter)

nisa

9 months ago

Your answer also isn't quite correct. And it's related to net neutrality. However there two things interwingled here that should be separate:

* Contract law: Meta lost a court case about a contract for IP-Transit with DT. This is legit and correct - they continued using the private interconnect and by german law this requires you to pay. This is clear cut.

* Telekom peering policy / brigandage: DT is the only major provider that is not present at any IX in Europe. They don't do any public peering. My local small ISP has better IX connectivity than DT while having 1/100th of the userbase. They require from hosters/content providers to sign contracts with them that are very overpriced (about 10x more than regulary private peering deals). They basically use their market power in germany (40% of internet users using their network) to force any internet service into buying an expensive contract with them.

This is a thing that should be regulated via the german regulator or EU wide. If other ISPs act like DT any server hosting provider needs to pay every bigger ISP for access to their customers and this would be the end to the internet as we know it. That's why it's a big topic. But legally DT can do it at the moment and they act like this for over 10 years now. There were older disputes with a big german hoster Hetzner, Init7, Level3... now Meta is playing on another level and hopefully that bully behavoir from DT stops and others follow suit and these overpriced contracts stop.

dathinab

9 months ago

interesting

then I would agree that the combination of unusual high prices for direct commit + absence on IX or in general not enough bandwidth for "fair" cross network pairing is indeed effectively undermining net neutrality and needs to be stopped

through IMHO it's a bit of an oversight of Meta to not be very clear about it, missing information in their article and other articles starting their reason section with 40% price cut doesn't make them look good

user

9 months ago

[deleted]

mananaysiempre

9 months ago

Still not clear what’s actually happening.

1. DT no longer peers with Facebook for free, and Facebook isn’t willing to pay? A bit more latency for Facebook users using DT, probably a small drop in Facebook visits from DT due to that, but overall, big whoop.

2. That + DT throttles Facebook traffic aggressively in retaliation? That would count as “undermining net neutrality principles” and bad in general. But it’s the kind of thing that already happens in a lot of places generally speaking (don’t know about Germany specifically), and we should push telecom and/or antitrust regulators to prohibit it, but I’d say there’s no urgency here.

3. That + DT sells customers some sort of “Internet except without Facebook, pay more to get Facebook” or “Internet except with slow Facebook, pay more to get fast Facebook”? That would officially be a Very Bad Thing, worth boycotts and other kinds of shouting from the rooftops.

The post is very very unclear about what’s actually happened. The actual court decision is about something like 1, but the talk of a “de facto paywall” would imply 3. Overall, I feel trampled by a Gish-galloping cavalry regiment here.

rsingel

9 months ago

DT tries to make everyone that peers pay them. Facebook used to pay the ransom, then decided to stop. Now its going to move to a transit connection.

The problem is DT keeps its transit routes congested or artificially limits how much traffic a transit provider can deliver.

So now when FB moves to transit, will DT widen those connections or will it keep them narrow to try to make Facebook's applications get terrible enough performance that Facebook will pay DT.

It's a showdown. DT wants to get paid twice for the same service, once by its subscribers and once by every app and website on the planet.

https://cyberlaw.stanford.edu/blog/2024/09/a-deutsche-teleko..."

SR2Z

9 months ago

FB should allow DT to degrade the quality of service and display a banner to DT users - the people it pisses off are going to be the Instagram-using public who in theory were the ones paying to connect to Facebook anyways.

hs86

9 months ago

Is this an issue with all networks that have a "restrictive" peering policy, as shown in [0], or is Deutsche Telekom particularly problematic in this case?

[0] https://www.peeringdb.com/net/196

kkfx

9 months ago

Another proof we need a less concentrated internet to few hubs. These hubs are harmful as harmful is the need of telco for violating net neutrality to serve such hubs well enough.

They have both reasons and they are both wrong.

immibis

9 months ago

I'm a Deutsche Telekom mobile customer. What's my chances of getting extraordinary contract termination in court due to their misbehavior?

shakna

9 months ago

Wouldn't a hit piece like this run afoul of most courts? There's a reason that talks after a ruling are so clipped.

NotYourLawyer

9 months ago

The reason is confidentiality agreements after a settlement. This isn’t a settlement.

shakna

9 months ago

The other reason is defamation. The line that constitutes it is lowered, when you're talking about something that has gone before the courts.

highcountess

9 months ago

This whole release reads like manipulative propaganda. All the emotive language, front loading/priming with a widely accepted idea among the consumer of the information, framing the title in victimization terms, “Takeaway” in a Propaganda Relations release, meaning “here is what we hope you will believe unflinchingly”, etc. It’s all there.

anilakar

9 months ago

> Takeaways: As a result of a recent German court ruling

Ten words. That's as far as anti-EU techbros will read before jumping to conclusions.

nottorp

9 months ago

They say nothing specific about the why, just random vague threats to "europeans".

I believe they forgot to mention "innovation" though?