OP here. Actually I tried to use it but apparently prematurely -> https://github.com/acmesh-official/acme.sh/issues/7085#issue...

Once it's supported I think my next iteration will be DNS persist + internal ip addresses on the public zone.

Thank you all for comments and feedback! It's cool to see real interest in this blog post

Thank you, I was unaware of that. It looks like it's already support in the acme.sh client, but there is a Let's Encrypt discussion saying it's still pending at LE:

https://community.letsencrypt.org/t/dns-persist-01-deploymen...

I wonder if the interim version has been rolled out to some CAs.

At this point, can't we make a standard that skips the middlemen and just embeds the certificate directly inside the DNS entry? It seems the challenge system is converging towards that anyway.

I never understood the issue with DNS-01. if you have a process that you trust to maintain a zone's TLS identity what is the big deal about letting it control a record in that zone?

Or, just use IPv6 and host Internet services in a routable address. Then, use ACLs at web server / proxy / L7 lb level to allow acme-challenge unauthenticated but everything else authenticated.

Lets encrypt supports IPv6 for validation.

Just use IPv6: no "internal" (10/8) or "external" address, just an address. ;)

> because it is using the public ip instead of the private one.

For this edge case, the issue is not with DNS; it's that the router is not configured to allow internal clients to access the network through the public IP, vis-à-vis hairpin routing.

And you still have issues sometimes around dns not able to figure out which interface to use. We had issues around that before.

[dead]

I do this exactly, using ACME DNS:

https://github.com/acme-dns/acme-dns

CAUTION, though, the last time I downloaded a binary release, ClamAV triggered on it, so I kept my old version which worked. I was using the 1.0 series (without any problems!), and now it seems the project has picked up development again with a 2.0 series.

> You need a DNS provider which supports API calls (I use DNSimple) but the core is all very straightforward.

Library/CLI that speaks the API of several dozen DNS providers so you don't have to re-invent the wheel:

* https://github.com/dns-lexicon/dns-lexicon

I get antsy about a private key being in lots of places. You've also got to worry about renewal so you might as well just have "consumer" of the wildcard just provision its own non-wildcard certificate.

I feel like I did. >smile<

> Use DNS validation to allow these internal services to pull ACME certs...

The major ACME clients support DNS validation. If your DNS host doesn't have an API that your ACME client supports either get a new ACME client or a new DNS host.

If you are going to have all the home stuff on a subdomain (int.example.com) would it work to delegate int.example.com to a DNS server running at home what has internet access, and could handle the ACME DNS challenges for machines on int.example.com?

If it does then you don't have to mess with your public DNS whenever you want to add or renew certificates for home machines.

I'm using the free DNS my registrar provides, which doesn't provide API access unless you upgrade to their paid DNS service and so if I could use a local DNS server for the ACME challenges for the home network I could pick one that is friendly to automation.

> I use the form of hostname.int.example.com […]

Note that int is a valid TLD:

* https://en.wikipedia.org/wiki/.int

* https://datatracker.ietf.org/doc/html/rfc1591

Can they dig for exploitable CVEs if they're not on the Wireguard network? It is a clue to your infrastructure, but I personally think the simplicity is worth it.

Do the names resolve to publicly routeable IPs? If not, I wouldn't worry about it.

My IaC is on public GitHub. They could do a network scan to find software then fingerprint to find version anyway.

Removing attack surface is better than trying to hide it.

How many people out there have attackers doing individualized research to identify services on their home LAN so they can chain a network attack with CVEs in their self-hosted service?

I get to use the same host names and IP addresses at home or on the VPN. I run a simple wireguard server on a Raspberry pi, and the whole setup has been rock solid.

I did set up tailscale, way back. After using it a few times to test, it failed me when I really needed it (I was out of the country and it failed - can't remember exactly what went wrong but it wwas 100% 'in my tailscale account'). I immediately dropped it and went back to OpenVPN (shit but reliable) before building my current setup.

Because hostname.tail62bc83.ts.net is a mouthful.

Personally, I use a custom local CA with name constraints so that it can only sign domains for The .internal TLD. This is the most important bit: because if the cert is ever leaked, it cannot be used to MITM connections to other domains.

I have to secure the CA's key, but I also have to secure all the keys for the certificate it signs, both being a similar level of challenge.

For personal use, or for very small organisations, using a passphrase-protected Yubikey as a "cheap HSM" should suffice.

I wish that in addition to the CA setting Name Constraints (and client software validating that), that end users could add additional constraints when adding a new trusted CA, so that even if the CA cert doesn't have Name Constraints, you can restrict it to a specific domain.

Yeah, adding CAs to the store sucks. On Linux it needs to be in /etc/ssl/certs (this varies slighly per distribution), which is only writable by root. A single user can't trivially trust a CA, and a great deal of applications/libraries don't support overriding the store's path.

The behaviour of curl depends on how your version was built.

Python? The widely used 'requests' relies on 'certifi' and skips the OS store - but 'pip' on the other hand does use the OS store.

A tool that might use java, like a database or IDE, means it might have its own store.

Node? Make sure you set NODE_USE_SYSTEM_CA=1

Firefox and Chrome AFAIK both have their own stores.

Building a Docker container? That's intentionally isolated from the host, of course your container won't inherit the OS trusted CAs. Running a VM locally? Same.

Installed something using snap? The container-like isolation means it won't pick up the OS trusted CAs.

And of course you need the certs set up right on your cloud servers, your CI servers, the dozen different smartphones the mobile team uses for testing.

Python for example, although nowadays you can simply install pip_system_certs to change that behaviour.

A lot of non-language tools bring their own certificate bundle as well, like uv, git, curl and Firefox. (I think they might all be the same Mozilla bundle even).

However, it seems like the situation here has improved slightly: git can read Windows certificates now once a flag has been set, Firefox has a flag for Windows and macOS, and it supports p11-kit. curl can be built with Windows/macOS support and respects OpenSSL environment variables. uv can be configured to use system-certs.

And obviously any VM or container has to be set up separately. That often includes stuff like pipelines in forges. As mentioned in the sibling comment, at least here it's definitely by design.

Node and (probably by extension) Electron infuriate me with their custom requirements.

Several Rust libraries also tend to default to a predefined set of certificates (which makes sense for libraries supposed to run on bare metal as there are no system certificates there, but that's not really a problem on most Linux installs). I make it a point to always use the native OS roots in the code I write, but unfortunately that's not universal.

Having to bind-mount certificates inside of docker containers is also always an annoyance I forget about until I see the first TLS errors in the logs, but that's by design and probably a good thing.

I do have a DNS server in my LAN, with some records served to internal clients only. But the _acme-challenge record needs to be public for the DNS-01 validation to succeed.

The point was that you can obtain a certificate for a domain name without creating any records other than the _acme-challenge TXT record. I.e., that the domain might be completely empty all the time except for this record.

> How do clients get the address for a nice domain name without the addresses in public DNS?

They don’t. You put the addresses in public DNS.

Wait…what? You have a fully internal service and you dont have an internal DNS server? I guess those exist in theory, but not in practice. :)

If you have a heavy enough tech stack to run fully internal services, than you can also run an internal DNS service (even pihole is enough) and load internal only entries there.

Or add everything to your hosts file if you have a central config service.

I think you'd at least have to have the fallback DNS set to your local DNS resolver in order to get the server.internal resolved to a A record.

I guess I mean treat it as a clear first class feature. Right now most browsers treat it as an arcane error. I’m thinking more “This is the first time you’re connecting to this site. Do you trust it?”

And later if something changes, then they can do the whole DOING SOMETHING NASTY! thing, which is effectively the experience today

Firefox now started that you can't even go on the page on some occasions.

Using a browser in an air gapped environment is so much more pain than it should be.

Well, if your cert-manager distributes its own CA, you'd still need the clients to trust the CA, even in k8s.

That support is limited. Browsers refuse to support passkeys when you TOFU untrusted certs, and for good reason.

That's not it. If you have an internal network, where every host is provisioned by you, you already control identity.

In that case there's no need to validate anything as names, dns records, certificates and anything else should already be in place.

One of the draft authors is already working on a cert-manager implementation. I'm optimistic it'll make it into cert-manager once the spec is ready and CAs actually start to implement it.

I have a few Traefik instances that request wildcards independently of each other. Each with the same config, per server.

Leaking is an issue but we're talking about internal services too.

well, I configure my services te request their own wildcard certs from a caching proxy acme to letsencrypt. Easy peasy.

Yeah you could run a single instance with multiple frontends - but the way I'm currently running things the public haproxy instance doesn't have network connectivity to the private services, meaning there is no way I can misconfigure things and expose an internal service to the internet.

There's really two ways, the other is to manage your own CA. But it seems like every browser/piece of software/etc out there is hell bent on making that as difficult as possible. It'd also be nice if it was easier to scope a certificate authority to a specific domain, but support for that is pretty patchy which is functionally the same as no support at all. And that's not to mention software that ignores the system certificate store. Or how tedious and nonstandardized it can be to get a trusted certificate store in a Docker container in cases where you have services that need to trust each other. Or how annoying it is to install your own trusted CA on devices (though, step-cli does help a lot at least on normal computers... phones however...). On and on and on, the barriers to what should be the obvious solution are extremely high.

He's the submitter

Thats what this Github Action uses is DNS validation through acme.sh and updating a Route53 domain.

Yes, it feels like I echo the same sentiment at work every few weeks. No, we can’t tick the PR with verify=false. Yes, you have to configure the trust store. Yes, it’ll take maybe an hour to work this out. Go do so please.

The only reason is laziness.