> But there is a problem - you can upload binaries that don't match the source.

Yes, this is well understood by the relevant people working on packaging. Unfortunately, reproducible builds are hard enough when you have a single, compiled language to work with; the Python ecosystem really can't support it. The main approach for ensuring source to binary correspondence now is "Trusted Publishing": https://docs.pypi.org/trusted-publishers/ This is opt-in, but it protects several of the most popular packages in case a maintainer is somehow compromised.

It's important to keep in mind that source packages bring their own security risks, because automated builds are the default. And since build requirements and environments can be arbitrarily complex and support arbitrary languages, this system requires relying on arbitrary code supplied by the package running at install time. This has been the most common attack vector recently, along with taking advantage of hooks that run at interpreter startup (such that the malicious package can operate without any contents being explicitly imported).