Unless you also mount some partitions noexec, making things not executable is useless. And if you have access to python/perl/ruby, you can construct any binary in memory anyway. And that's assuming someone's targeting some vulnerability chain which uses the compiler which is a stretch anyway.

It's a chain. Take PCI DSS v4.0 for example.

Requirement 2.2.1 says: "Configuration standards are developed, implemented, and maintained to <...> Be consistent with industry-accepted system hardening standards or vendor hardening recommendations."

Then in the third column, it mentions explicitly: "Sources for guidance on configuration standards include but are not limited to: Center for Internet Security (CIS), International Organization for Standardization (ISO), National Institute of Standards and Technology (NIST), Cloud Security Alliance, and product vendors."

CIS, at least in the past, was a significant source of overzealous pseudo-hardening. Yet, that's what auditors' automated tools check compliance with, as that's the only configuration standard with a written procedure, often a command that can be copy-pasted, to check compliance with each rule. And I am not allowed to object to the recommendations or not follow the "best practices" because otherwise the next breach will be fully on me (in financial terms).

That is also why Lynis does not follow a specific set, but applies generic principles from multiple sources. Yes, some of the items may be default (now) in Linux distributions, but often they are still aren't. For example, most systemd services definitely can use more strict defaults. The distribution is typically not making the changes, to avoid breaking things for the end-user. This is where Lynis comes in, being independent of any big commercial organization (yes, looking at you Red Hat). While working on Lynis for 17 years now, I can say some things definitely improved in Linux distributions, but still so many things that could be much better secured out-of-the-box.

CIS itself may have good ideas, but the implementation is mostly bullshit. Compare the actual differences between a CIS Ubuntu docker images and a plain one. There's 3 valid changes you can do by hand and the rest is snake oil that makes the image larger as a bonus.

Lynis author here. While some defaults definitely became better, often due to the kernel itself being better protected, there is still a lot of room for improvement. The distribution often can't make things too strict, to prevent common issues. Keep also in mind that it is not just the OS itself, but especially the parts that get added over time (users, software, configuration file changes) that introduce the biggest flaws. The aim of Lynis is to do a regular health check, giving the sysadmin the chance to tighten things where needed or correct those things that got out of spec.

We are looking for something to run as part of our ami/docker testing and as you say, stays fresh on standards (whatever soc2/iso, but ideally also FIPS) , any prefs?

If auditors are going to use this, it would benefit even the most competent sysadmin to know what it's gonna say. The average compliance analyst isn't going to understand why some enumerable risk isn't actually a threat because; your threat model makes said issue actually impossible. Even if you can prove it, they're still gonna include it in their needless risk findings. I'd postulate (for fun) that most competent sysadmins would be more likely to have that problem, because they've already identified it, and are using it as a makeshift 'honeypot'.