Kwpolska
6 hours ago
What is it about Python that makes developers love fragmentation so much? Sending HTTP requests is a basic capability in the modern world, the standard library should include a friendly, fully-featured, battle-tested, async-ready client. But not in Python, stdlib only has the ugly urllib.request, and everyone is using third party stuff like requests or httpx, which aren't always well maintained. (See also: packaging)
woodruffw
10 minutes ago
AFAICT, lacking a (good) standard HTTP library is kind of the norm in popular languages. Python, Ruby, Rust, etc. all either have a lackluster standard one or are missing one. I think it sits between two many decision pressures for most languages: there are a _lot_ of different RFCs both required and implied, lots of different idioms you could pick for making requests, lots of different places to draw the line on what to support, etc.
The notable exception is Go, which has a fantastic one. But Go is pretty notable for having an incredible standard library in general.
dirkc
5 hours ago
You would think that sending HTTP requests is a basic capability, but I've had fun in many languages doing so. Long ago (2020, or not so long ago, depending on how you look at it) I was surprised that doing an HTTP request on node using no dependencies was a little awkward:
const response = await new Promise( (resolve, reject) => {
const req = https.request(url, {
}, res => {
let body = "";
res.on("data", data => {
body += data;
});
res.on('end', () => {
resolve(body);
});
});
req.end();
});wging
5 hours ago
These days node supports the fetch API, which is much simpler. (It wasn't there in 2020, it seems to have been added around 2022-2023.)
dirkc
4 hours ago
Yes, thankfully! It's amusing to read what they say about fetch on nodejs.org [1]:
> Undici is an HTTP client library that powers the fetch API in Node.js. It was written from scratch and does not rely on the built-in HTTP client in Node.js. It includes a number of features that make it a good choice for high-performance applications.
Pay08
24 minutes ago
Why is it amusing?
b450
25 minutes ago
Note that node-fetch will silently ignore any overrides to "forbidden" request headers like Host, since it's designed for parity with fetch behavior in the browser. This caused a minor debugging headache for me once.
rzmmm
3 hours ago
Web standards have rich support for incremental/chunked payloads, the original node APIs are designed around it. From this lens the Node APIs make sense.
simlevesque
2 hours ago
And you don't handle errors at all...
thedanbob
2 hours ago
> Sending HTTP requests is a basic capability in the modern world, the standard library should include a friendly, fully-featured, battle-tested, async-ready client.
I've noticed that many languages struggle with HTTP in the standard library, even if the rest of the stdlib is great. I think it's just difficult to strike the right balance between "easy to use" and "covers every use case", with most erring (justifiably) toward the latter.
ivanjermakov
5 hours ago
HTTP client is at the intersection of "necessary software building block" and "RFC 2616 intricacies that are hard to implement". Has nothing to do with Python really.
WhyNotHugo
9 minutes ago
httpx has async support (much like aiohttp), whereas urllib is blocking-only. If you need to make N concurrent requests, urllib requires N threads or processes.
maccard
6 hours ago
> Then I found out it was broken. I contributed a fix. The fix was ignored and there was never any release since November 2024.
This seems like a pretty good reason to fork to me.
> Sending HTTP requests is a basic capability in the modern world, the standard library should include a friendly, fully-featured, battle-tested, async-ready client. But not in Python,
Or Javascript (well node), or golang (http/net is _worse_ than urllib IMO), Rust , Java (UrlRequest is the same as python's), even dotnet's HttpClient is... fine.
Honestly the thing that consistently surprises me is that requests hasn't been standardised and brought into the standard library
francislavoie
5 hours ago
What, Go's net/http is fantastic. I don't understand that take. Many servers are built on it because it's so fully featured out of the box.
maccard
2 hours ago
The server side is great. Sending a http request is… not
lenkite
6 hours ago
Your java knowledge is outdated. Java's JDK has a nice, modern HTTP Client https://docs.oracle.com/en/java/javase/11/docs/api/java.net....
ffsm8
4 hours ago
Ahh, java. You never change, even if you're modern
HttpClient client = HttpClient.newBuilder()
.version(Version.HTTP_1_1)
.followRedirects(Redirect.NORMAL)
.connectTimeout(Duration.ofSeconds(20))
.proxy(ProxySelector.of(
new InetSocketAddress("proxy.example.com", 80)
))
.authenticator(Authenticator.getDefault())
.build();
HttpResponse<String> response = client.send(request, BodyHandlers.ofString());
System.out.println(response.statusCode());
System.out.println(response.body());
lenkite
2 hours ago
Your http client setup is over-complicated. You certainly don't need `.proxy` if you are not using a proxy or if you are using the system default proxy, nor do you need `.authenticator` if you are not doing HTTP authentication. Nor do you need `version` since there is already a fallback to HTTP/1.1.
HttpClient client = HttpClient.newBuilder()
.followRedirects(Redirect.NORMAL)
.connectTimeout(Duration.ofSeconds(20))
.build();awkwardpotato
3 hours ago
What's the matter with this? It's a clean builder pattern, the response is returned directly from send. I've certainly seen uglier Java
freedomben
an hour ago
Just my opinion of course, but:
> What's the matter with this?
To me what makes this very "Java" is the arguments being passed, and all the OOP stuff that isn't providing any benefit and isn't really modeling real-world-ish objects (which IMHO is where OOP shines). .version(Version.HTTP_1_1) and .followRedirects(Redirect.NORMAL) I can sort of accept, but it requires knowing what class and value to pass, which is lookups/documentation reference. These are spread out over a bunch of classes. But we start getting so "Java" with the next ones. .connectTimeout(Duration.ofSeconds(20)) (why can't I just pass 20 or 20_000 or something? Do we really need another class and method here?) .proxy(ProxySelector.of(new InetSocketAddress("proxy.example.com", 80))), geez that's complex. .authenticator(Authenticator.getDefault()), why not just pass bearer token or something? Now I have to look up this Authenticator class, initialize it, figure out where it's getting the credentials, how it's inserting them, how I put the credentials in the right place, etc. The important details are hidden/obscured behind needless abstraction layers IMHO.
I think Java is a good language, but most modern Java patterns can get ludicrous with the abstractions. When I was writing lots of Java, I was constantly setting up an ncat listener to hit so I could see what it's actually writing, and then have to hunt down where a certain thing is being done and figuring out the right way to get it to behave correctly. Contrast with a typical Typescript HTTP request and you can mostly tell just from reading the snippet what the actual HTTP request is going to look like.
looperhacks
18 minutes ago
> but it requires knowing what class and value to pass
Unless you use a text editor without any coding capabilities, your IDE should show you which values you can pass. The alternative is to have more methods, I guess?
> why can't I just pass 20 or 20_000 or something
20 what? Milliseconds? Seconds? Minutes? While I wouldn't write the full Duration.ofSeconds(20) (you can save the "Duration."), I don't understand how one could prefer a version that makes you guess the unit.
> proxy(ProxySelector.of(new InetSocketAddress("proxy.example.com", 80))), geez that's complex
Yes it is, can't add anything here. There's a tradeoff between "do the simple thing" and "make all things possible", and Java chooses the second here.
> .authenticator(Authenticator.getDefault()), why not just pass bearer token or something?
Because this Authenticator is meant for prompting a user interactively. I concur that this is very confusing, but if you want a Bearer token, just set the header.
Pay08
20 minutes ago
> why can't I just pass 20 or 20_000 or something? Do we really need another class and method here?
If you've ever dealt with time, you'll be grateful it's a duration and not some random int.
colejohnson66
2 hours ago
The boilerplate of not having sane defaults. .NET is much simpler:
using HttpClient client = new();
HttpResponseMessage response = await client.GetAsync("https://...");
if (response.StatusCode is HttpStatusCode.OK)
{
string s = await response.Content.ReadAsStringAsync();
// ...
}pjmlp
21 minutes ago
Yeah, so much simpler,
"Common IHttpClientFactory usage issues"
https://learn.microsoft.com/en-us/dotnet/core/extensions/htt...
"Guidelines for using HttpClient"
https://learn.microsoft.com/en-us/dotnet/fundamentals/networ...
And this doesn't account for all gotchas as per .NET version, than only us old timers remember to cross check.
lmz
2 hours ago
That's just an example. It does have defaults: https://docs.oracle.com/en/java/javase/11/docs/api/java.net.... (search for "If this method is not invoked")
PxldLtd
2 hours ago
Yeah this is all over Rust codebases too for good reason. The argument is that default params obfuscate behaviour and passing in a struct (in Rust) with defaults kneecaps your ability to validate parameters at compile time.
Pay08
4 minutes ago
It does have defaults, the above example manually sets everything to show people reading the docs what that looks like.
umvi
5 hours ago
What's wrong with Go's? I've never had any issues with it. Go has some of the best http batteries included of any language
Orygin
4 hours ago
I guess he never used Fiber's APIs lol
The stdlib may not be the best, but the fact all HTTP libs that matter are compatible with net/http is great for DX and the ecosystem at large.
maccard
an hour ago
Thr comment I replied to was talking about sending a http requests. Go’s server side net/http is excellent, the client side is clunky verbose and suffers from many of the problems that Python’s urllib does.
localuser13
6 hours ago
>Honestly the thing that consistently surprises me is that requests hasn't been standardised and brought into the standard library
Instead, official documentation seems comfortable with recommending a third party package: https://docs.python.org/3/library/urllib.request.html#module...
>The Requests package is recommended for a higher-level HTTP client interface.
Which was fine when requests were the de-facto-standard only player in town, but at some point modern problems (async, http2) required modern solutions (httpx) and thus ecosystem fragmentation began.
Spivak
5 hours ago
Well, the reason for all the fragmentation is because the Python stdlib doesn't have the core building blocks for an async http or http2 client in the way requests could build on urllib.
The h11, h2, httpcore stack is probably the closest thing to what the Python stdlib should look like to end the fragmentation but it would be a huge undertaking for the core devs.
Kwpolska
5 hours ago
Node now supports the Fetch API.
pjc50
5 hours ago
> dotnet's HttpClient is... fine.
Yes, and it's in the standard library (System namespace). Being Microsoft they've if anything over-featured it.
xnorswap
3 hours ago
It's fine but it's sharp-edged, in that it's recommended to use IHttpClientFactory to avoid the dual problem of socket exhaustion ( if creating/destroying lots of HttpClients ) versus DNS caching outliving DNS ( if using a very long-lived singleton HttpClient ).
And while this article [1] says "It's been around for a while", it was only added in .NET Framework 4.5, which shows it took a while for the API to stabilise. There were other ways to make web requests before that of course, and also part of the standard library, and it's never been "difficult" to do so, but there is a history prior to HttpClient of changing ways to do requests.
For modern dotnet however it's all pretty much a solved problem, and there's only ever been HttpClient and a fairly consistent story of how to use it.
[1] https://learn.microsoft.com/en-us/dotnet/core/extensions/htt...
pixl97
20 minutes ago
>"It's been around for a while"
is 14 years not a while?
gjvc
4 hours ago
requests is some janky layer onto of other janky layers. last thing you want in the stdlib.
it's called the STD lib for a reason...
tclancy
3 hours ago
Don't think it's Python-specific, it's humanity-specific and Python happens to be popular so it happens more often/ more publicly in Python packages.
BigTTYGothGF
24 minutes ago
I think the python maintainers are still feeling burnt by the consequences of the "batteries included" approach from the old times.
yoyohello13
20 minutes ago
Most Python developers these days weren't even programming when the 2 -> 3 split happened. Unless you're referencing something else.
denimnerd42
13 minutes ago
the batteries included approach is the stdlib that can do everything. turns out it’s hard to maintain and make good.
matheusmoreira
4 hours ago
Everybody's got a different idea of what it means for a library to be "friendly" and "fully-featured" though. It's probably better to keep the standard library as minimal as possible in order to avoid enshrining bad software. Programming languages could have curated "standard distributions" instead that include all the commonly used "best practice" libraries at the time.
duskdozer
3 hours ago
matheusmoreira
2 hours ago
That situation should be avoided. People should have to create their own libraries until everyone empirically converges into a de facto standard that can then be made official.
functionmouse
2 hours ago
Bram's Law: https://files.catbox.moe/qi5ha9.png
Python makes everything so easy.
fsckboy
3 minutes ago
converted to text:
I realized this the other day, and dub it Bram's Law -- Bram
Bram's Law
The easier a piece of software is to write, the worse it's implemented in practice. Why? Easy software projects can be done by almost any random person, so they are. It's possible to try to nudge your way into being the standard for an easy thing based on technical merit, but that's rather like trying to become a hollywood star based on talent and hard work. You're much better off trading it all in for a good dose of luck.
This is why HTTP is a mess while transaction engines are rock solid. Almost any programmer can do a mediocre but workable job of extending HTTP, (and boy, have they,) but most people can't write a transaction engine which even functions. The result is that very few transaction engines are written, almost all of them by very good programmers, and the few which aren't up to par tend to be really bad and hardly get used. HTTP, on the other hand, has all kinds of random people hacking on it, as a result of which Python has a 'fully http 1.1 compliant http library which raises assertion failures during normal operation.
Remember this next time you're cursing some ubiquitous but awful third party library and thinking of writing a replacement. With enough coal, even a large diamond is unlikely to be the first thing picked up. Save your efforts for more difficult problems where you can make a difference. The simple problems will continue to be dealt with incompetently. It sucks, but we'll waste a lot less time if we learn to accept this fact.
LtWorf
4 hours ago
The HTTP protocol is easy to implement the basic features but hard to implement a full version that is also efficient.
I've often ended up reimplementing what I need because the API from the famous libraries aren't efficient. In general I'd love to send a million of requests all in the same packet and get the replies. No need to wait for the first reply to send the 2nd request and so on. They can all be on the same TCP packet but I have never met a library that lets me do that.
So for example while http3 should be more efficient and faster, since no library I've tried let me do this, I ended up using HTTP1.1 as usual and being faster as a result.
mesahm
2 hours ago
I spend 3 years developing Niquests, and believe me, HTTP is far from easy. Being a client means you have to speak to everyone, and no one have to speak to you (RFC are nice, but in practice never applied as-is). Once you go deep under the implementation, you'll find a thousand edge cases(...). And yes, the myth that as developer http/1 is "best" only means that the underlying scheduler is weak. today, via a dead simple script, you'll see http/2+ beat established giant in the http/1 client landscape. see https://gist.github.com/Ousret/9e99b07e66eec48ccea5811775ec1... if you are curious.