Compiling Python extensions is a nightmare because we allow Autoconf, CMake, Visual Studio, Bazel etc. to make it complicated and nonportable; when someone sets out to wrap some library for Python the quality of the result is limited by the quality of the tools and by low expectations.

A serious engineering effort along the lines of the Zig compiler would allow Python to build almost everything from source out of the box; exotic compilers and binary dependencies, not "Frankenstein code bases" per se, are the actual obstacles.

Wheels do predate conda (though manylinux did base itself on the experience of Anaconda and Enthought's base set of libraries), and there were distributions like Enthought (or even more field specific distributions like Ureka and individuals like Christoph Gohlke) that provided binaries for the common packages.

What the conda ecosystem did was provide a not-horrible package manager that included the full stack (in a pinch to help fix up a student's git repository on a locked-down Windows system I used conda to get git, and you can get R and do R<->python connections easily), and by providing a standard repository interface (as opposed to a locked down and limited version that the other providers appeared to do), conda pushed out anyone doing something bespoke and centralised efforts so spins like conda-forge, bioconda, astroconda could focus on their niche and do it well.

If I didn’t have access to binary wheels, I wouldn’t have been able to learn Python in high school as a heavy Windows user.

Even now, running into package compile issues is a sure fire way to lose half an hour of my time.

> was here before Anaconda popularized the idea of binary packages for Python

This has incredible "I was there, Gandalf, three thousand years ago" vibes =P

the article does not suggest removing wheeels

I think pypi should require larger packages, like tensorflow, to self-host their releases.

There is all support for that already - the pypi index file contains arbitrary URL for data file and a sha256 hash. Let pypi store the hashes, so there is no shenanigans with versions being secretly overridden, but point the actual data URLs to other servers.

(There must obviously be a balance for availability vs pypi's cost, so maybe pypi hosts only smaller files, and larger files must be self-hosted? Or pypi hosts "major releases" while pre-releases are self-hosted? And there should be manual exceptions for "projects with funding from huge corporations" and "super popular projects from solo developers"...)

I believe tensorflow does remove old pre-releases (I know other projects do), so that number I think might be fairly static?

That tensorflow is that big isn't surprising, given the install of it plus its dependencies is many gigabytes (you can see the compressed sizes of wheels on the release pages e.g. https://pypi.org/project/tensorflow/#files), and the "tensorflow" package (as opposed to the affiliated packages) based on https://py-code.org/stats is 965.7 GiB, which really only includes a relatively small number of pre-releases.

Why tenserflow is that big comes down to needing to support many different kinds of GPUs with different ecosystem versions, and I suspect the build time of them with zig cc (assuming it works, and doesn't instead require pulling in a different compiler/toolchain) would be so excessive (especially on IoT/weaker devices) that it would make the point of the exercise moot.

Is it though? If it saves one engineer one afternoon that storage has paid for itself, and this thing has hundreds of thousands of downloads a day.

Wouldn’t it be more horrifying to force everybody who wants to use a prerelease to waste an afternoon getting it to build just to save half a hard drive?

The problem is not space, it's the fact that bandwidth costs 4x the total operating income that the PSF has. Everything else is context to understand and offer some insight into this problem.

This is a pitch for Zig as a C/C++ build system because, as you can see in other comments in this submission, there's still a lot of people that don't even believe this is a solvable problem at all, while in reality Zig does a pretty good job at solving it.

Only somebody actively involved with Python can really say if and how this could help improve the situation for PyPI, but if the people who actually happen to be working on something potentially beneficial weren't allowed to speak "because it's a pitch" then what even is the point of software interoperability.

As an occasional Python user, I also think that it's silly that people keep putting containers everywhere and pre-building a ton of binaries when I know for a fact that you could trivially `zig biuld` that shit, since I do it in my projects all the time (see https://github.com/allyourcodebase/).

> Binary artifacts don't cause exponential growth in storage requirements. It's still just linear.

I think what the author was getting at was the combinatoric explosion as you add different variants (CPU architecture, operating systems, python versions) - but I agree that "exponential growth" is not the right term to use here.

CI providers should definitely start proxying PyPI with their own cache.

Yeah we use Artifactory internally as a proxy for DockerHub, Pypi, NPM, etc.

Also, this is very much assuming that the code is both C or C++ and that LLVM is the right compiler to use. Fortran is still a major part of the ecosystem, which the Zig compiler isn't going to solve. There already exists numerous options to provide compilers to the problematic platform, the fact is binary wheels (mostly) solve the issue far better than doing local builds.

Also, the large packages are typically due to the need to support the huge number of possible GPU combinations (because you care about what CUDA versions are supported).

This feels like a solution being forced on a problem (not that zig cc isn't cool), but the post has really misunderstood the issues around wheels.

One strategy would be making PyPI packages fetch any external resources from PyPI, or at least add PyPI URLs as mirrors for such resources.

What do Java and Rust (and your C++ package manager of choice) do to mitigate those things?

Once you have control over generating the binaries, you can create a system that is more akin to what Nix offers, where other parties can setup secondary caches for their projects, and if they don't the main registry has the option to decide what to do about it without breaking any package.

Currently PyPI is railroaded into a fairly tight set of choices because it can't exercise a lot of control over binary data.

The blog post mentions this point directly.

Really depends on how you get the storage/interface with it. Cloud/VPS is some of the worst 'bang for the buck' in my experience, where S3 or dedicated can be more favorable at a point. Not all gigglebytes cost the same :)

The article may shed some light on this, I'm not aware. Haven't read yet! I've managed a few RPM package mirrors. Dedicated gear that's a few generations old with no-name providers was the best experience.

There are currently 570K+ projects in pypi, 60k+ in debian repos.

It can take several months of work to approve one single package to the official repos, for a single distribution. And each have different rules and setup.

Now explain to me how you think this is going to work.

Also, do you place to force everyone to use chroot or containers to replace their virtual env systems to have variations on deps? Or maybe everybody should use nix?

Do you want to do that also for JS, Ruby and PHP?

There are many problems with this idea.

* Windows exists.

* The system package manager is meant for the system environment. The typical Python environment is separate from that - a container or a venv. The system Python installation is there to make the OS work, not to provide a working dev environment, and it's often not suitable as such (especially on Debian).

* There simply are far too many packages out there to make it workable.

* Part of the point of how Pip works is that compiled code can be pre-built for specific platforms. A system package manager typically loses that advantage and needs to rebuild from source.

* Installing Python packages "from source" commonly involves running arbitrary Python code at install time, by design. (The build system simply doesn't provide any standardized way to invoke compilers, nor to apply conditional logic - you're stuck with Turing-complete, unsandboxed Python. Python is really hard to sandbox anyway.) For obvious reasons it's not appropriate to do that as root, which rather interferes with using a system package manager. Yes, obviously packages that mirror PyPI distributions and are made to be installed with the system package manager, will be adapted to avoid security issues. But this requires significant, parallel maintenance effort. The devs of the individual projects likely don't have resources to spare to help distro maintainers, either.

AUR is not a source of system packages. It is a community-maintained source of package build scripts which each user must download and build themselves.

This is how it should be, but it isn't how it is. The system packages are generally very outdated, and don't support multiple versions on a single system (which makes dependency locking a fantasy).

I dream of the day where we only have one package manager on a single machine.

Python builds weren't really hermetic before, but now your entire operating system is a part of the software definition.

This is a slide backwards.

I wish there was an easy way for pip to list what it's missing so it's easier to install dependencies via the system package manager. Also it's sad how setup tools bdist_rpm seems to be considered deprecated.

Doesn't brew compile from source every time on the user's machine?

Nope. System packages and language dependencies are better not to be mixed, and the horrors if you do that are very real.

I can't tell if this is sarcasm.

> Dig in, finally figure out that my older version was actually importing PyJWT even though I had specified “jwt.” The new container was breaking because it was actually installing a different “jwt” library.

Names on PyPI have no necessary correlation with names used to `import` modules in the code. This is known and by design and has always been so; and it is thoroughly documented (e.g. https://packaging.python.org/en/latest/discussions/distribut...). There are at important architectural reasons for it:

* A package installed from PyPI can legally provide any number of importable packages and/or modules - including zero.

* Making it work the way you imagine would require additional verification work, with no clear idea of who's responsible for that work nor what to do when the assumption is violated.

* Sometimes projects have valid reasons to rename what you `import` at top level. It could be to avoid common conflicts; it could be because of internal open-source politics (see: PIL/Pillow); it could be because a legacy name was needed for some disambiguation purpose with Conda; for whatever rare reasons it might just make more logical sense. In the extreme, some name could become a keyword in a later Python version. (There's an actual `async` package on PyPI, although it was last updated 10 years ago.) In all of these cases, there's value in keeping the name used on PyPI the same.

* There are valid reasons why people might want to swap implementations of the same library. The PyPI system allows you to just change the name provided to `pip install` without modifying the code.

You actually ran into that last point: both `jwt` and `PyJWT` on PyPI refer to implementations of JSON Web Tokens that expect you to `import jwt` in the code. They may or may not have different interfaces and supply different functionality, and are maintained by different people. You can't seriously expect to forbid everyone else from publishing such an implementation just because one already exists.

> This PyJWT import is missing an algorithm. To fix that, I also need to pip install “cryptography”

There are a few possibilities:

* The maintainer messed up and forgot to list it as a dependency in the manifest. These things happen. Other people's projects are allowed to have their own dependencies.

* The maintainer listed `cryptography` as an optional dependency or "extra" - so that you don't have to grab it if you aren't using that algorithm. You're expected to read the documentation to understand what your project will need.

* You really were intended to use the `jwt` package, but a new version of that project removed functionality you were depending on. Other people's projects are allowed to change their interfaces and go through deprecation cycles. In this case, you would presumably find that your code also needed to be rewritten to invoke the `cryptography` functionality explicitly/directly.

>making sure to get the compatible version

Pip solves dependencies for you. If you're running into version incompatibility issues, generally it's either the fault of the package maintainers, or the environment you want to create is fundamentally not possible due to conflicting version requirements of your transitive dependencies. (Pip is designed to only install one version of a given package in a given environment, because Python is designed to only `import` a given module from one place and then reuse the cached module object.)

>So maybe this is all very obvious and “duh” to veterans out there but this was impossibly silly and wasted a stupid amount of time for something that should be dead simple

Yes, it's obvious. Yes, normally it is dead simple. Veterans commonly run into hairy problems regardless. There are plenty of faults in the system - this just isn't generally considered one of them.

The scope of the problem solved (or at least undertaken) by the Pip/PyPI system is much larger than you seem to imagine. In particular, it often has to do that same wrangling with linkers and Makefiles that you describe, and in an automated fashion at that. (Python code commonly has to interoperate with extensions written in C - and FORTRAN, and other languages - which is provided as part of the same package.)

My read was it enables both: PyPI can delete all packages that haven't been downloaded in a year or more, safe in the knowledge that they can be quickly recreated and cached, but also that more packages can be downloaded as source, and compiled at target.

Because packages shouldn't be upgraded in a project until they have been tested. If you give me no way of downloading version 1.0 of a package in my project because 1.1 supersedes it and 1.1 breaks my project, what am I supposed to do?

>Why would you want to install an old version of ruff?

Because something else in your project is broken by the new version.