Wheels for free-threaded Python now available for psutil

72 pointsposted 3 months ago
by grodola
(gmpy.dev)

20 Comments

simonw

3 months ago

https://hugovk.github.io/free-threaded-wheels/ is looking pretty healthy - 130 of the 360 most downloaded C extension PyPI packages are now free-threaded Python compatible, up from 92 on 15th August https://web.archive.org/web/20250815071755/https://hugovk.gi...

I was curious as to how that site works - it has a build script at https://github.com/hugovk/free-threaded-wheels/blob/cdae0b45... which checks the PyPI available file downloads for a package and looks for a bdist_wheel that matches this:

  abi_tag = download["filename"].removesuffix(".whl").split("-")[-2]
  if abi_tag.endswith("t") and abi_tag.startswith("cp31"):
      has_free_threaded_wheel = True

rogerbinns

3 months ago

Note that free threaded compatible doesn't necessarily mean the package supports free threading (concurrent execution), just that it can be loaded into a free threaded interpreter.

This is the case with my own package which is on the hugovk list (apsw) which will cause the GIL to be re-enabled if you load it into a free threaded Python. The reason I provide a binary wheel is so that you don't have to keep separate GIL full and free threaded interpreters around. They have a different ABI so you can't use extensions compiled against one with the other.

Free threading is at the beginning of its journey. There is a *lot* of work to on all C code that works with Python objects, and the current documentation and tools are immature. It is especially the case that anyone doing Python concurrent object mutation can cause corruption and crashes if they try, and that more auditing and locking need to be done in the C code. Even modules in the standard library have only been partially updated.

You can see a lot details and discussion in the comments at https://news.ycombinator.com/item?id=45633311

LtWorf

3 months ago

Why do you provide it at all then if it's not working as intended yet?

rogerbinns

3 months ago

As I stated:

> so that you don't have to keep separate GIL full and free threaded interpreters around

It means the user doesn't have to keep two Pythons around, install packages in both of them, etc.

It is also possible with the free threaded Python to keep the GIL disabled even if a package such as mine says it needs the GIL. And my package will indeed work just fine, until you supply it with mutable data and concurrently modify it in another thread.

LtWorf

3 months ago

But the users install the free threaded python to do free threaded stuff. The second they use your package they have a GIL again, which entirely defeats the point.

Wouldn't it be much better to just not support it if it's not supported?

rogerbinns

3 months ago

What is better:

1) Saying your package supports free threading, but it isn't safe - ie concurrent mutation can result in corruption and crashes

2) Allowing the package to be loaded into a free threaded Python, which immediately enables the GIL. Concurrent mutation does not result in corruption and crashes because of the GIL. The user doesn't have to maintain two Python installations. They can set the environment variable PYTHON_GIL=0 or start Python with -Xgil=0 which will keep the GIL disabled, and they will be fine if they avoid concurrent mutation.

I chose 2. The stdlib json package (along with many others) picked 1. Heck I'll guarantee that most that picked 1 aren't 100% safe either, because doing the changes is hard work, *every* case has to be covered, and tools like thread sanitizers don't work.

The reason I chose 2 is because I care about data integrity. I will eventually reach 1, but only once I can be certain the code is correct.

LtWorf

3 months ago

3) Saying your package doesn't support free threading instead of adding a GIL and force the users to stick to regular python.

rogerbinns

3 months ago

You aren't forced to use a GIL as I keep stating. You can set an environment variable or a command line flag to Python and the GIL will remain disabled. My package will work just fine if you do that, unless you provide it with data you concurrently modify in which case you can get corruption and crashes.

cyanydeez

3 months ago

Do you atleast warn users? This sounds like madness.

rogerbinns

3 months ago

Yes. The interpreter warns by default, and requires steps to disable the warning. My release notes say that the GIL will be enabled when the package is loaded.

Is it madness that other packages claim they support running without the GIL, yet it is possible to cause corruption and crashes just by writing concurrent Python code? That is the case with the standard library. Compiler thread sanitizers don't work with free threaded Python. Diligent code inspection by humans is the only way to update C code so far.

Free threading is at the beginning of the project. It works. You get slow downs in single threaded performance due to extra locking, and speedups in concurrent performance due to threading. But it is possible to cause corruption and crashes via Python code. Don't expose it to untrusted data and code.

But do investigate it and see what works well and what doesn't. See what code patterns are now possible. Help improve the tools and documentation. It had to start somewhere, and the current state is somewhere.

LtWorf

3 months ago

I think what you are doing is hiding problems. I think crashes and bugs are preferable to find the issues at this point. People who want the safe option will run the regular python.

rogerbinns

3 months ago

I provide 3 options:

1) Use regular GIL Python and you get the highest levels of integrity and correctness of operation of my package

2) Use a free threaded Python, the GIL will be enabled at load time, and you get the highest levels of integrity and correctness

3) Use a free threaded Python, and set $PYTHON_GIL=0 or start with -Xgil=0 to keep the GIL disabled, and providing you do not do concurrent mutation of data provided to my package, you get the highest levels of integrity and correctness

BTW I did not randomly choose to provide the free threaded builds. I specifically asked the setuptools maintainers (under the Python Packaging Authority) how to prevent free threaded builds for PyPI. They encouraged me to do the free threaded builds so that a user doesn't to have maintain parallel regular and free threaded Python installations. And it allows option 3 above.

semiinfinitely

3 months ago

It has never been clear to me what the term "wheels" means in software releases. seems like something along the lines of "thing that works" – does it mean anything more specific than that?

zahlman

3 months ago

"Wheel" is the Python-specific terminology for a pre-built package (all non-Python code has been compiled ahead of time, and the necessary object files put in the right places in the package's folder hierarchy), such that installers can simply copy the files (and do a little bit of book-keeping, and possibly pre-compile Python source code to bytecode although the interpreter will do that on demand anyway). The format is originally documented in https://peps.python.org/pep-0427/ .

This contrasts with "sdist", short for "source distribution", which potentially contains code in other languages which must be compiled on the end user's machine (which possibly involves downloading a corresponding build system and setting up a temporary build environment at install time).

Wheel and sdist distributions are successors to a legacy "egg" format. The PEP doesn't explain, but it's commonly held that the name "wheel" is a reference to wheels of cheese, based on PyPI's prior existence as "the cheese shop" (itself a reference to https://www.youtube.com/watch?v=Hz1JWzyvv8A). There might be some ancient discussion on the (now archived and inactive) mailing list to support this.

borntyping

3 months ago

It doesn't mean anything useful in this context. It's a leftover from when PyPI was called "the cheeseshop", which in turn was a reference to a Monty Python sketch.

morkalork

3 months ago

Cheese wheels?! TIL. Also, for once it isn't a car analogy.

woodruffw

3 months ago

It's a Python packaging specific term. A "wheel" is a built distribution, which in the most basic sense just means that it can be unarchived directly into a Python import prefix instead of needing a build step (historically `setup.py`) to prepare it for installation.

In practice, many built distributions contain binary artifacts (e.g. builds of CPython extensions). This differentiates them from source distributions, where you'd build the extension from source on your local machine.

zahlman

3 months ago

> In practice, many built distributions contain binary artifacts (e.g. builds of CPython extensions). This differentiates them from source distributions

This seems like a good opportunity to mention that that the wheel isn't required to contain any non-Python content, and indeed it's strongly preferred to provide one even (I would say "especially") when there is no meaningful "build step" (https://pradyunsg.me/blog/2022/12/31/wheels-are-faster-pure-...). There are of course abandoned legacy projects around from before the option was available, that may never get fixed. But developers of new code in pure Python should ensure they understand the system.

(I would strongly prefer if we could have a single type of package artifact with an optional post-install hook for building any needed non-Python content. Alas....)

woodruffw

3 months ago

Yeah, I was simplifying. Wheels are in many ways a strictly better alternative to source distributions even in the “pure” case!

zahlman

3 months ago

Wow, long time no see.

psutil is a great project and I do have some future plans involving it.