The PKI Detour: A Week on the Wrong Cert Strategy, Three FreeIPA Failures

GitLab was up. The code had a home. The obvious next question was: how does every other internal service get a TLS certificate?

The answer I wanted: something clean, automated, and not requiring full Cloudflare DNS credentials on every host. The answer I got — eventually — was “Cloudflare DNS-01, one token per service, and accept the limitations.” Getting there took a week on the wrong approach, then six days and three attempts on a more ambitious one. This post is both of those failures, what they cost, and what survived them.

The Cloudflare Token Problem

The straightforward path to internal TLS is Let’s Encrypt with DNS-01 validation. The homelab isn’t publicly reachable, so HTTP-01 is out, but DNS-01 works fine: certbot proves domain ownership by creating a _acme-challenge TXT record through the Cloudflare API.

The problem is the token. Cloudflare API tokens can be scoped to a specific zone and restricted to DNS edit permissions only — but they can’t be scoped below the zone level. A token issued for gitlab.tod.net certbot can edit any record in tod.net. There’s no way to say “this token may only touch _acme-challenge.gitlab.tod.net.” So every service that needs a cert gets a token that, if compromised, could rewrite your entire DNS zone.

I’d recently started working with acme-proxy at work and thought it might solve this. The idea: run a single ACME proxy that holds the DNS credentials, and have each service authenticate to the proxy instead of directly to Cloudflare. The proxy does the DNS challenge on the service’s behalf. Services never touch Cloudflare at all.

The Week I Built the Wrong Thing

June 7, a Sunday evening. The acme-proxy approach looked promising enough to build out.

The architecture was:

• acme-proxy running on clamps.tod.net (the cloud host, publicly reachable) — holds the Cloudflare credentials, handles DNS challenges for any internal host that asks
• acme-dns server also on clamps — a lightweight DNS server that only serves _acme-challenge TXT records, delegated via CNAME from Cloudflare
• certbot-acme-dns role — replaces certbot-cloudflare on service hosts; each host authenticates to acme-dns, which can only update its own _acme-challenge record

The CNAME delegation part of this is actually elegant. Instead of giving each service a Cloudflare token, you add one CNAME per hostname in Cloudflare pointing _acme-challenge.hostname.tod.net at a record in acme-dns’s zone. Each service gets acme-dns credentials that can only update that one TXT record. No Cloudflare access at all on the service side.

By the end of Sunday I had roles/acme-dns-server, roles/certbot-acme-dns, and a 413-line setup script (setup_acme_dns.py) handling registration, CNAME creation, and credential storage in LastPass. The playbook ran. The roles deployed.

def cmd_register(bootstrap_token: str) -> None:
    """Register all hostnames not yet in LastPass."""
    missing = [e for e in ACME_DNS_ENTRIES if not _lpass.item_exists(e.lpass_name)]
    zone_id = cf_get_zone_id(bootstrap_token)

    for entry in missing:
        creds = acme_dns_register(ACME_DNS_SERVER, entry.hostname)

        # Add CNAME: _acme-challenge.<hostname> → <fulldomain>
        cname_name = f"_acme-challenge.{entry.hostname}"
        existing = cf_list_dns_records(zone_id, bootstrap_token, name=cname_name)
        if not any(r["type"] == "CNAME" for r in existing):
            cf_add_dns_record(
                zone_id, bootstrap_token, "CNAME", cname_name, creds["fulldomain"], proxied=False
            )

        # Store credentials in LastPass as a JSON note in the password field.
        _lpass.upsert(entry.lpass_name, username=entry.hostname, password=json.dumps(creds))

That CNAME line is the elegant part I still think was right: once it’s in place, the service’s acme-dns credentials can only ever update its own _acme-challenge record — never the real Cloudflare zone the CNAME points away from. It’s the acme-proxy binary sitting in front of this that couldn’t get its own certificate. The part underneath was fine.

Then I went to get a cert for acme-proxy itself — the proxy needs its own certificate to serve HTTPS — and hit the wall.

acme-proxy doesn’t support Cloudflare DNS-01. The ESnet implementation ships DNS-01 support for AWS, Azure, and GCP only. Getting a cert for the proxy itself meant either exposing port 80 publicly (which defeats the point) or switching DNS-01 providers away from Cloudflare — not an option, since tod.net is hosted there. The proxy that was supposed to centralize ACME handling couldn’t get its own certificate through the same DNS provider everything else in the homelab depends on.

That’s a fact checkable in about thirty seconds against acme-proxy’s own documentation, and neither I nor Claude Code checked it before writing any Terraform. I asked for an acme-proxy container and an Ansible playbook to run it; Claude Code went straight to exploring the existing bootstrap patterns and building from there, the same as it would for any well-understood tool. Compatibility with the one DNS provider the entire homelab depends on never came up as a question until I went looking for a cert. That’s on both of us — I didn’t ask, and it didn’t flag the gap either.

The other option on the table, ZeroSSL, has a free tier — but it’s capped at 3 certificates a month.¹ Not useful for a homelab with more than a handful of services.

June 13, six days later: deleted all of it. roles/acme-proxy, roles/acme-dns-server, roles/certbot-acme-dns, setup_acme_dns.py, the playbooks, the group vars — 1,426 lines removed in one commit.²

The compromise that stuck: certbot-cloudflare role, one Cloudflare token per service. The tokens are zone-scoped and DNS-edit-only — not record-scoped, which is the real gap — but at least a compromised credential only requires revoking one token rather than rotating a shared credential everywhere. I’m still looking for a better answer for publicly-trusted certs.³ It’s part of why the private CA exists, though the real driver for step-ca was SSH certificates, not X.509. The private CA is a happy side effect. The offline root was just fun.

The Offline Root Vision

With TLS strategy settled (unhappily), the more interesting problem was internal PKI: how does the homelab issue certificates to services that don’t need public trust? The answer was an offline root CA topology.

The idea: the root signing key never touches a disk. It lives only on offline USB drives (a YubiKey is on the wishlist once budget allows), unlocked briefly on a Proxmox VM that itself boots only for a signing ceremony — a few times a year, not for daily operation. Intermediate CAs stay online so services get certs without waking the root. If an intermediate key gets compromised, you revoke it and rotate it without ever touching the root. The root stays offline, stays safe, stays the single source of truth.

The topology decision (ADR 0005) locked in with two parallel intermediates: one from FreeIPA’s Dogtag CA, one from step-ca. Everything signed by the same root, root trust distributed once to all clients, and then you never have to touch it again.

FreeIPA’s Promise

FreeIPA is a full-stack identity platform: LDAP, Kerberos, DNS, and an integrated Dogtag CA — all one package. It has an --external-ca flag at install time: stage the install, get a CSR, take it to the offline root for signing, complete the install with the signed cert. Perfect fit for the topology.

I found FreeIPA appealing for another reason: it’s enterprise software with a clear upgrade path. A working FreeIPA deployment is a known quantity. Big companies run this in production.

FreeIPA doesn’t ship on Debian at all — the first attempt at a Debian 13 container failed before install even started, because freeipa-server has no Debian package, only the client.⁴ FreeIPA server is a Red Hat-ecosystem product, so the only real choice was which Enterprise Linux release to run it on. I picked Rocky Linux 10 — the current EL release as of June 2026 — over the more battle-tested Rocky 9, on the reasoning that a 10-year identity host should start on the release with the longest runway left, not the one already partway through its support window. ADR 0008 recorded EL9 explicitly as the fallback if EL10 caused trouble. In hindsight the container hit three separate rough edges before the real failure even started, which is exactly the kind of friction a less-bleeding-edge EL release might have avoided — but the tradeoff was made deliberately, not by default.

A day before any of this started, the CA rollout got a structure it hadn’t had until then. The plan had been growing across ADRs, wishlist ideas, an operational README, and scattered notes, to the point that reasoning about any one piece meant loading all of it — and the phase order existed only as an implicit list, which hid a real dependency inversion (the signing ceremony actually consumes CSRs produced by later-numbered phases). On June 14 that got split into one document per phase — ca-02 (CRL host) through ca-06 (root trust distribution) — each with its own goal, explicit dependencies, and a progress log, plus a roadmap index tying them together.⁵ FreeIPA landed as phase four, so its plan doc is ca-04, and the three attempts below are its own tracking labels — ca-04a, ca-04b, ca-04c — not something invented for this post.

Three Attempts, Three Different Failures

ca-04a: Partial Success (June 15)

First attempt: Rocky 10 LXC container, native ipa-server-install --external-ca. The installer runs in two phases — emit a CSR in phase 1, consume a signed certificate in phase 2.

Three on-the-fly problems:

1. The Rocky LXC template didn’t include openssh-server — couldn’t SSH in until I installed it through the Proxmox console
2. DNS zone overlap — FreeIPA’s DNS server started taking over a zone already defined in UniFi
3. NTP unavailable inside the LXC, and FreeIPA is strict about clock skew

The first one turned into a permanent wart in the Terraform, because it’s not a one-time fix — every rebuild of the container needs it again. It even needed a follow-up fix two days later, when the workaround’s own package install broke on a dead mirrorlist:

echo "==> Ensuring sshd is running in container ${proxmox_virtual_environment_container.ipa_container.vm_id} (EL LXC ships without openssh-server)..."
ssh -o StrictHostKeyChecking=no -o ConnectTimeout=10 root@${local.proxmox_ssh_host} \
  "pct exec ${proxmox_virtual_environment_container.ipa_container.vm_id} -- bash -c 'command -v sshd >/dev/null 2>&1 || dnf install -y --disablerepo=extras openssh-server; dnf config-manager --save --setopt=extras.skip_if_unavailable=1 2>/dev/null || true; systemctl enable --now sshd'"

The --disablerepo=extras got added after the Rocky 10 LXC template’s extras mirrorlist turned out to be empty, breaking the dnf install before it could even reach openssh-server. Every other guest in the fleet is Debian, where the template just has an SSH server. FreeIPA’s one Rocky 10 requirement meant this workaround existed for as long as the container did.

All three were fixable. By the end of phase 1, I had a fresh CSR at /root/ipa.csr, subject O=TOD.NET, CN=Certificate Authority. The assumption: phase 2 would consume the signed cert and complete the install.

ca-04b: Catastrophic Regress (June 16)

Signing ceremony done. Signed cert and root CA chain in hand. Copied them to the container:

/root/ipa.crt      # ceremony-signed cert for FreeIPA's Dogtag CA keypair
/root/root-ca.crt  # root CA certificate

Set ipaserver_external_cert_files in group_vars and re-ran the ansible-freeipa playbook to complete phase 2.

Expected: certificate accepted, install completed.

Actual: the role re-ran phase 1 entirely, regenerated a fresh Dogtag keypair, and emitted a new CSR. The ceremony-signed cert was now useless — the private key it matched had been overwritten.

Root cause, from reading ansible-freeipa 1.16.0 source (plugins/modules/ipaserver_test.py:982): the “already configured, skip the install block” short-circuit only fires when both external_ca=false and external_cert_files is unset. Our phase-2 run set external_cert_files (correctly) but had left external_ca: true from phase 1 — group_vars/ipa/vars.yml never toggled it off between phases — so the short-circuit never fired, the full install block re-ran, and Dogtag’s internal state got reset out from under the pending CSR.

Two things had to be true for this to happen, and I only untangled them by building a throwaway repro afterward. First, a usage bug on my end: the upstream maintainer says phase 2 should set only external_cert_files, and ours had both vars set. Second, a latent role defect, issue #124 (“ipaserver role is not idempotent,” still open upstream) — the guard doesn’t recognize a partial install even with the maintainer-correct vars. The repro isolates which one actually matters, by checking whether the Dogtag key’s fingerprint survives stage 2 or gets silently regenerated:

assert_stage2() {  # assert_stage2 <stage1-csr-fingerprint>
  if [ -f /etc/ipa/ca.crt ]; then
    live="$(certutil -L -d /etc/pki/pki-tomcat/alias \
             -n "caSigningCert cert-pki-ca" -a \
             | openssl x509 -noout -pubkey \
             | openssl pkey -pubin -outform DER | sha256sum | awk '{print $1}')"
    [ "$live" = "$1" ] && echo "RESULT: completed, key PRESERVED" \
                       || echo "RESULT: completed but key CHANGED (regenerated)"
  elif [ "$(pubkey_fp /root/ipa.csr)" != "$1" ]; then
    echo "RESULT: re-entered stage 1 — NEW CSR generated (key regenerated)"
  else
    echo "RESULT: stage 2 errored without regenerating — inspect ipaserver-install.log"
  fi
}

Run with the maintainer-correct vars (external_ca off in stage 2), the key survived — so the real regression was the usage bug, not #124, though #124 means a disturbed install can still eat the key regardless. Native ipa-server-install, run directly with no Ansible in the loop, resumes correctly where the role doesn’t — that’s what the repro’s Part 2 validated.

The key had been generated, taken to a signing ceremony, and then silently discarded by the playbook. No backups of the container.

The retry (June 17): a signing ceremony isn’t a five-minute affair — it means booting the offline root VM, unlocking LUKS, physically disconnecting the NIC before any key material comes out, mounting a USB drive, and walking through the ceremony scripts by hand. So the orphaned serial-01 cert wasn’t just discarded; it had to be formally revoked before a replacement could be issued, which meant writing a ceremony-revoke.sh script that didn’t exist yet — the original ceremony tooling had no revoke path, because nothing had needed one before. I revoked serial 01, signed a fresh CSR, and started stage 2 again. It didn’t go cleanly either: the ceremony scripts turned out to be hardcoded to copy-1 (the plaintext USB drive) only, so I had to generalize them to decrypt from copies 2–4 as well, and stage 2 then failed a second way — pki_skip_installation gets forced on internally whenever pki_external_step_two is set, which skips NSS client-database initialization and broke the PKCS12 import (SEC_ERROR_PKCS12_UNABLE_TO_IMPORT_KEY). That’s two ceremonies spent on FreeIPA, not one, before I gave up on the Ansible path entirely. (I’ll probably write a whole post about the root ceremony itself at some point — the mechanics of it deserve more than a parenthetical.)

ca-04c: Deferred, then abandoned (never executed)

The recovery path was clear: run ipa-server-install natively in a single uninterrupted session, no Ansible role, no state-management between phases. I wrote the runbook. I set up a repro container and validated the native path would work — but I did not spend a third real ceremony to prove it end-to-end. Two burned ceremonies was already two more than planned.

Then I stopped.

Why FreeIPA Lost

By June 19, after two weeks of accumulated friction — the acme-proxy detour, three failed or aborted FreeIPA installs — the calculus had shifted. Rebuilding FreeIPA from scratch using the native path meant learning the full installation procedure outside of Ansible, on a platform (Rocky 10) that’s a permanent exception in an otherwise Debian homelab. And for what? LDAP, Kerberos, DNS, and a Dogtag CA — none of which I actually needed immediately.

The specific blockers:

1. ansible-freeipa’s external CA path is broken under replay — ca-04b proved the two-phase design doesn’t survive any gap between phases
2. Every failed CSR costs a real ceremony, not a retry — two ceremonies burned already, each one a physical, hands-on affair (offline VM, air-gapped NIC, USB key material); a third to prove out ca-04c’s native-install recovery was a cost I wasn’t willing to pay a third time
3. Native install requires a single uninterrupted session — reasonable for a production deploy, annoying for a homelab where you work in evening sessions
4. EL-only creates a permanent operational exception — one Rocky 10 host in a Debian homelab is operational debt forever
5. Dogtag complexity exceeds homelab needs — NSS-based, HSM-capable, designed for large deployments; I don’t need any of it
6. The same outcomes are available differently — LDAP + Kerberos from Samba AD on Debian; TLS + SSH certs from step-ca

On June 19, FreeIPA was shelved. All documentation went into docs/shelved/. CT 105 and CT 108 were destroyed via terraform destroy. ADR 0013 recorded the decision.

Honestly? After two weeks of wrong turns, I needed something to work. The same day FreeIPA was shelved, I got it: migrating Mastodon onto official Docker images finally went the way infrastructure work is supposed to go. That’s the next post.

What Survived the Failures

step-ca still carries the PKI load, signed by the same offline root.

For TLS: Let’s Encrypt via certbot + Cloudflare DNS-01. Works for any tod.net hostname including split-horizon LAN-only names. Renewals are automatic. step-ca handles the pure-LAN hostnames with no public DNS record.

For SSH host certificates: step-ca’s SSH CA. The bootstrap playbook adds the SSH host CA public key to ~/.ssh/known_hosts as @cert-authority *.tod.net. VM rebuilds produce zero friction — no REMOTE HOST IDENTIFICATION HAS CHANGED warnings because the SSH CA signature proves the host is who it claims. This is the real reason step-ca exists. The private CA is a nice side effect.

For cold storage: the ca-04b failure prompted a rethink — a key that gets silently discarded by a state-management bug makes you take backup integrity a lot more seriously. The original plan (ADR 0005) was four USB copies, full-disk LUKS2. The revised plan (ADR 0011) is stricter: the payload is ~250 bytes of ECDSA P-384 PEM, and at that size a LUKS header is a single point of failure for a secret orders of magnitude smaller than the header protecting it. Full-disk encryption also does nothing to catch silent bit rot on a drive sitting in a drawer for a decade.

What landed instead: plain FAT32 volumes, the key encrypted three independent ways on each drive (GPG, age, OpenSSL — any one decrypts it), a PAR2 recovery set at -r100 (full reconstruction), SHA256SUMS sidecars, and a paper backup — QR-encoded root key, laser-printed directly from the USB stick’s own port so the plaintext key never touches a networked machine — in a fire-rated safe.

I haven’t actually scanned that QR code back yet. Verifying it means reading the raw key with a camera, and every camera I own is attached to a device that talks to a network — exactly what the whole cold-storage scheme exists to keep the key away from. It needs a scanner that can be wiped or destroyed afterward, and I don’t have a spare one sitting around. Until then, the paper backup is untested in the one way that actually matters.

Each annual ceremony runs a verify-and-rewrite pass on every drive, because powering on a flash drive doesn’t refresh its cells — only rewriting the files does:

echo "verifying checksums..."
if ! sha256sum -c SHA256SUMS; then
  warn "checksum failure — attempting PAR2 repair"
  par2 repair recovery.par2 || die "PAR2 repair failed — RETIRE this drive"
  sha256sum -c SHA256SUMS   || die "still failing after repair — RETIRE this drive"
fi

echo "verifying PAR2 parity..."
par2 verify recovery.par2 || die "PAR2 verify failed"

echo "rewriting files to refresh flash charge..."
find . -type f -print0 | while IFS= read -r -d '' f; do
  cp -a "$f" "$f.tmp" && mv -f "$f.tmp" "$f"
done

echo "re-verifying after rewrite..."
sha256sum -c SHA256SUMS

A drive that still fails after a PAR2 repair gets retired, not trusted with one more year.

The offline root + parallel intermediates topology is sound. The FreeIPA path into it broke. The step-ca path into it works. The topology held.

AI-assistant disclaimer: This post was drafted by Claude Code (Claude Sonnet 4.6, claude-sonnet-4-6, with later revisions by Claude Sonnet 5, claude-sonnet-5) from session transcripts, commit history, and the ADRs, then reviewed and edited by me. It may contain inaccuracies; verify specifics before relying on them.