Note to self on setup for auto-renewing Let’s Encrypt SSL certificates from an (almost) private network.
The Usual Rant
This would’ve been really simple if Google Domains provided an API interface to add/update/remove DNS TXT records – and would be fully workable from a fully private network, I could (or maybe should?) use a DNS provider with an API and be done, but this setup is worth documenting for myself. For DNS providers with APIs to change DNS records, Certbot’s DNS plugins are, IMO, easy ways to renew certs. You could also go manual with your own auth, cleanup, and deploy hooks1. Anyway, here goes.
The Idea
The idea behind this setup is that of the Standalone DNS Authenticator Plugin for Certbot – I just could not get it to work2 out of the box even with port forwarding enabled, so I improvise by replicating the setup. Basically: send DNS resolution requests off to a DNS that you can update really quick (or even better, automate). This is achieved by inserting an NS
record for the subdomain in question and pointing _acme-challenge.subdomain
to a dummy.subdomain.domain.ext CNAME
that sends off resolvers to your new DNS, where you work your magic.
Prerequisites:
- A DNS provider with no API support to add/update/remove DNS
TXT
records. - A local/remote instance of AdGuardHome v0.105.03 or later – this nifty application has a REST API with a swagger spec!
- An internet router that allows port forwarding (forward 53 to your local server hosting AdGuardHome)4 1.
- Local server with certbot on a private network.
The Setup
For each subdomain of domain.ext, set up these two DNS entries:
- subdomain NS ns.domain.ext
- _acme-challenge.subdomain CNAME dummy.subdomain.domain.ext
Pro tip: always use –staging servers else you will run up against the really low (~5 failures/hour) rate limits for production servers.
- Next, run the certbot command to renew certificate manually:
# certbot --manual --preferred-challenges dns certonly -d subdomain.domain.ext --verbose --debug-challenge --staging
- Go to AdGuardHome > Filters > Custom Filtering Rules and add appropriate entries
||subdomain.domain.ext^$dnsrewrite=NOERROR;A;W.X.Y.Z
(private IP is acceptable)5||_acme-challenge.subdomain.domain.ext^$dnsrewrite=NOERROR;TXT;challenge-txt
, where challenge-txt is from the output ofcertbot
command above.
- Continue the command above, your certificates from staging Let’s Encrypt’s servers would be ready.
- Revoke these test certs with
# certbot revoke --cert-path /etc/letsencrypt/live/subdomain.domain.ext/cert.pem [--staging]
- Repeat the
certbot
command without the--staging
flag.
Best part – all of this can be automated.1 6
References & Notes
- https://digwebinterface.com/?hostnames=_acme-challenge.saibox.saicharan.in&type=A&showcommand=on&colorize=on&trace=on&ns=resolver&useresolver=8.8.4.4&nameservers=sains.mooo.com
- https://community.letsencrypt.org/t/solved-dns-problem-servfail-looking-up-caa-for-mit42-de/20965/5
- https://github.com/siilike/certbot-dns-standalone
- https://dnsviz.net/d/saitest.saicharan.in/X_QJ1g/dnssec/
- https://github.com/AdguardTeam/AdGuardHome/blob/master/openapi/openapi.yaml
- https://www.withoutthesarcasm.com/letsencrypt-for-private-networks/
This renewal can be automated by using auth- and deploy- hooks. Example commands
certbot -d subdomain.domain.ext --agree-tos --manual --preferred-challenges dns --manual-auth-hook ./adguard-dns-auth.sh --manual-cleanup-hook ./adguard-dns-clean.sh --manual-public-ip-logging-ok --force-renewal certonly
↩︎ ↩︎ ↩︎For my record, this command did not work:
certbot --non-interactive --agree-tos --email certmaster@domain.ext certonly --preferred-challenges dns --authenticator certbot-dns-standalone:dns-standalone --certbot-dns-standalone:dns-standalone-address=0.0.0.0 --certbot-dns-standalone:dns-standalone-ipv6-address=:: --certbot-dns-standalone:dns-standalone-port=53 -d subdomain.domain.ext --force-renewal --staging --verbose --debug-challenges
↩︎AdGuardHome v0.105.0+ is required to support
dnsrewrite
↩︎By default, only allow allow: 127.0.0.1, and your public IP:
curl -k https://domains.google.com/checkip
. Temporarily update firewall rules to allow out-of-network requests. This too can be automated via AdGuardHome’s REST API:GET /control/access/list
andPOST /control/access/set
in combination withufw
or similar. ↩︎I ran into
SERVFAIL looking up CAA for ...
error from certbot. Turns out the problem was a missingA
record for subdomain.domain.ext in AdGuardHome, which is where the need for this entry originates from. https://dnsviz.net was very helpful deduce this problem. It was on top of the list of DNS errors reported! ↩︎GET /status
, add/update corresponding dnsrewrite entry intouser_rules[]
andPOST /control/filtering/set_rules
can be used in conjunction with the auth- and deploy- hook scripts to automate the entries that get inserted forchallenge-txt
. ↩︎