Migrating Active Directory to Samba, One Domain Controller at a Time

The first post in the Downsizing series named Active Directory as one of the two reasons the old Windows dependency had to go: vader and maul, two Windows Server 2019 domain controllers, each a license to track and a machine that existed mainly to run one service. The PKI Detour covered the identity side of that problem going sideways for two weeks — a dead-end ACME proxy, three failed attempts at FreeIPA — before landing on Samba AD as the replacement. This is the second post in Off Windows, the thread that tracks removing every remaining Windows dependency from the lab; the first post covered getting the rest of the fleet ready the evening before. This one covers the part that actually moves the domain: the in-place replica join that swaps both Windows DCs for a Samba box named sidious, without a single workstation noticing.

The constraint that ruled out the easy path

ad.tod.net was a real production domain, created April 22, 2020¹ — just over six years old by the time of this migration — with 25 real users, 24 computers, and 14 real groups, once Windows’ own built-in accounts and default security groups (Administrator, Guest, krbtgt, Domain Admins, Domain Controllers, and the like) are excluded from the raw object count. Six years of accumulated service accounts and machine objects is exactly the kind of thing that quietly outlives its own bookkeeping — a pass to actually go through each one and confirm it’s still in use is still on the todo list². Every Windows workstation’s local user profile is keyed to two SIDs — the user’s and the domain’s. Stand up a new domain and migrate accounts into it, and every one of those profiles breaks; every workstation needs to rejoin and every user gets a fresh, empty desktop.

That constraint eliminates the obvious approach. An internal architecture decision record³ settled on the alternative: join a Samba box to the existing domain as just another domain controller, let AD’s own replication copy the directory across (domain SID, every object SID, every password hash, all of it), transfer the FSMO roles once replication is proven clean, then demote the Windows DCs. Because it’s the same domain the whole time, no SID ever changes and no workstation ever has a reason to notice.

The catch: Samba has to actually be able to join at 2012 R2. That was the one open question the whole plan hinged on, and it got its own disposable test before touching production.

Phase 0: a spike that failed twice before it passed

The plan called this a “feasibility spike” — stand up a throwaway Debian trixie VM, join it to ad.tod.net as an extra DC, confirm replication, then demote and delete it. The runbook for this step records it as passing; what it doesn’t show is that the join itself failed twice before it worked.

First attempt, no extra options:

DsAddEntry failed with status WERR_ACCESS_DENIED info (8567, 'WERR_DS_INCOMPATIBLE_VERSION')

Samba’s ad dc functional level defaults to 2008_R2. The live domain sits at 2012 R2, and a writeable DC below the domain’s functional level gets flatly rejected. The fix is a one-line join option — --option="ad dc functional level = 2016" — but finding that out cost one failed join first.

Second attempt, with the functional-level option added but authenticating via --use-krb5-ccache against the wrong DC:

Failed to bind - LDAP client internal error: NT_STATUS_INVALID_PARAMETER

Third attempt is the one that actually works, and it’s copy-pasteable as-is against any Samba box joining a 2012-R2-or-later Windows domain:

samba-tool domain join ad.tod.net DC \
  -U "AD\Administrator" \
  --server=maul.ad.tod.net \
  --dns-backend=SAMBA_INTERNAL \
  --realm=AD.TOD.NET \
  --option="ad dc functional level = 2016"

The three things that matter, each learned from one of the two failures above: --option="ad dc functional level = 2016" (Samba’s silent 2008_R2 default is what triggered WERR_DS_INCOMPATIBLE_VERSION), password auth via -U rather than --use-krb5-ccache, and --server= pinned explicitly at the DC holding FSMO rather than letting Samba pick one via DNS. Replication came up both directions, the directory counts matched (28/22/62 plus the spike itself), and the spike held none of the five FSMO roles, exactly as required. Feasibility proven — but on the third try, not the first, and the runbook’s clean pass/fail table doesn’t carry that texture forward.

Demoting the spike left debris behind that the plan didn’t anticipate: a stale CN=SPIKE machine account and an empty server object, both still present after samba-tool domain demote reported success. Cleaning them up needs a second command against a surviving DC, using the still-valid Kerberos ticket from the join:

samba-tool computer delete SPIKE -H ldap://maul.ad.tod.net --use-kerberos=required
ldbdel --use-kerberos=required -H ldap://maul.ad.tod.net \
  'CN=SPIKE,CN=Servers,CN=Default-First-Site-Name,CN=Sites,CN=Configuration,DC=ad,DC=tod,DC=net'

Worth naming plainly: this cleanup didn’t fully take. A samba-tool dbcheck run five phases later, after MAUL’s own demotion, was still flagging two orphaned SPIKE objects as pre-existing noise — debris from a throwaway test that outlived the disposable VM not by hours but by seventeen days. It finally went down with the last Windows DC, in the cleanup at the end of this post.

An overnight gap in the middle of a “one-day” migration

Every phase after the spike — provisioning sidious, joining it as the real additional DC, copying SYSVOL, cutting DNS over, transferring FSMO, demoting maul — happened on June 21, 2026. That’s a true statement, but it undersells what actually happened, because it wasn’t one sitting — and the spike itself belongs to the night before. In local time:

• 9:29 p.m.–12:20 a.m. — the night before: fleet prep, then the Phase 0 spike, executed and passed.
• (a nine-hour forced pause)
• 9:15 a.m.–2:57 p.m. — Phases 1 through 5, provisioning through FSMO transfer.
• 2:59–4:15 p.m. — Phase 6 planning and MAUL’s demotion.

The pause wasn’t chosen, and it wasn’t graceful. Twenty minutes past midnight, with the spike passed and Claude midway through committing the runbook and decision-record updates that recorded the result, the session stopped:

You've hit your session limit · resets 3:10am (America/New_York)

The half-finished commit sat on the branch overnight. The limit reset at 3:10 a.m.; the human it was waiting on did not. When the session resumed at 9:15 the next morning, Tod’s entire first message to an assistant frozen mid-commit was:

please finish

— Tod

(The two-minute gap between Phases 5 and 6 that afternoon was the opposite kind of seam — deliberate: merge and push Phase 5, immediately branch for Phase 6.)

So the real shape of the migration was: spike late the night before, a pause forced by the tooling and extended by sleep, then the production migration in one continuous daytime push. The tone throughout stayed flat — no visible relief or stress in either direction. The closest thing to a celebration, after the spike passed clean, was Tod’s:

all green, do you need the output?

— Tod

Provisioning sidious: a Terraform hang and four live Ansible fixes

The spike proved Samba could join; Phase 1 provisioned the real DC. sidious is a small Debian 13 VM (2 vCPU, 2 GB, 20 GB), managed by Terraform against Proxmox, with a stable MAC registered in UniFi for a fixed IP:

resource "proxmox_virtual_environment_vm" "samba_dc" {
  name        = "sidious"
  description = "Samba AD DC for ad.tod.net (Managed by Terraform)"
  node_name   = var.proxmox_target_node

  cpu {
    cores = 2
    type  = "host"
  }

  memory {
    dedicated = 2048
  }

  disk {
    datastore_id = "local-lvm"
    import_from  = var.samba_dc_template_file_id
    interface    = "scsi0"
    size         = 20
    discard      = "on"
  }

  agent {
    enabled = true
  }

  network_device {
    bridge      = "vmbr0"
    model       = "virtio"
    mac_address = local.samba_dc_mac_address
    vlan_id     = 22
  }
}

That agent.enabled = true looks like the obviously-correct default — it tells Terraform to wait for the QEMU guest agent to report an IP before considering the VM up. The Debian 13 genericcloud cloud-init image doesn’t ship the guest agent preinstalled, so Terraform sat at Still creating... waiting for a signal that was never coming — while the VM itself was long since booted and answering SSH. Terraform was the only party in the room that couldn’t see it:

it’s been 10 min and terraform is still saying even though I can ssh into the box already

— Tod

From there the diagnosis took under a minute, and the fix was one line:

+  # Debian genericcloud doesn't ship qemu-guest-agent; the Ansible samba_dc role
+  # installs it post-bootstrap. Disabled here to avoid an indefinite create wait.
   agent {
-    enabled = true
+    enabled = false
   }

The one-line fix was the cheap part; the fallout wasn’t. The half-created VM still had the agent flag enabled on the Proxmox side, and the provider queries the agent during state refresh — so every Terraform command that followed hung the same way. plan hung. Even terraform destroy, the command meant to clear the wreckage, hung refreshing the very resource it was trying to delete. Since the bootstrap had never run, nothing on the box was worth saving, and Tod’s call — “we haven’t done anything ‘real’ with the sidious vm, right?” — was to stop nursing the state: escape out-of-band through Proxmox itself, beyond Terraform’s reach, then destroy and recreate from scratch with the agent disabled from birth. The second apply ran clean end to end.

Ansible’s samba_dc role handles OS prerequisites after the VM is up — packages, DNS, service masking — everything except the join itself, which stays a manual, credentialed, one-time ceremony (the same precedent set by the earlier FreeIPA attempts). The role needed four live corrections before it was actually solid, each caught mid-ceremony rather than anticipated:

1. A missing package. kinit/klist come from krb5-user, not krb5-config — the role had the wrong one:

 samba_dc_packages:
   - samba-vfs-modules
   - winbind
   - krb5-config
+  - krb5-user
   - dnsutils

2. cloud-init’s /etc/hosts bug. Debian cloud-init writes 127.0.1.1 <hostname>, and Samba refuses to join if its own hostname resolves to loopback:

- name: Remove cloud-init 127.0.1.1 hostname entry from /etc/hosts
  ansible.builtin.lineinfile:
    path: /etc/hosts
    regexp: '^127\.0\.1\.1\s+'
    state: absent

- name: Ensure DC FQDN resolves to its real IP in /etc/hosts
  ansible.builtin.lineinfile:
    path: /etc/hosts
    regexp: '^{{ ansible_default_ipv4.address }}\s+'
    line: "{{ ansible_default_ipv4.address }}    {{ ansible_hostname }}.{{ samba_dc_ad_domain }} {{ ansible_hostname }}"

3. A platform-behavior surprise. Older Debian masked samba-ad-dc automatically until a join completed. Trixie doesn’t:

- name: Check whether the domain join has been completed
  ansible.builtin.stat:
    path: /var/lib/samba/private/sam.ldb
  register: _samba_joined

- name: Mask samba-ad-dc until the domain join is complete
  ansible.builtin.systemd:
    name: samba-ad-dc
    masked: true
    state: stopped
    enabled: false
  when: not _samba_joined.stat.exists

4. A resolv.conf race between pre- and post-join state. Before the join, /etc/resolv.conf needs to symlink to systemd-resolved so the box can resolve anything at all; after the join, it needs to be a static file pointing at Samba’s own DNS instead. Running the pre-join task unconditionally on every align.yml rerun would clobber the post-join file back to a symlink:

- name: Point /etc/resolv.conf at systemd-resolved non-stub resolver (pre-join)
  ansible.builtin.file:
    path: /etc/resolv.conf
    src: /run/systemd/resolve/resolv.conf
    state: link
    force: true
  when: not _samba_joined.stat.exists

- name: Write static resolv.conf pointing at sidious itself (post-join)
  ansible.builtin.copy:
    dest: /etc/resolv.conf
    content: |
      search ad.tod.net
      nameserver 127.0.0.1
      nameserver 192.168.22.5
    owner: root
    group: root
    mode: "0644"
  when: _samba_joined.stat.exists

The _samba_joined fact — whether /var/lib/samba/private/sam.ldb exists — is what makes the whole role safe to rerun indefinitely, before or after the join, without the two states fighting each other. None of these four fixes are exotic; they’re the kind of thing that only surfaces by actually running the ceremony against a real domain and watching what breaks.

SYSVOL: four things that didn’t work before one that did

Samba speaks no DFS-R, so SYSVOL — the share holding Group Policy objects and logon scripts — has to move by hand. The runbook lists the working method (Compress-Archive on the Windows side, scp across, unzip on sidious) as if it were the obvious choice. It wasn’t; it was the fourth thing tried.

In order: a CIFS mount from sidious back to maul’s SYSVOL share failed authentication outright (STATUS_LOGON_FAILURE — the kernel CIFS client only speaks NTLMv1, which a Server 2019 DC rejects). A robocopy push from the Windows side to \\sidious\sysvol, tried once as a regular domain account and once as Administrator, both failed the same way:

ERROR 65 (0x00000041) Network access is denied.

Samba’s SYSVOL share isn’t writable over SMB even to Domain Admins — a deliberate Samba limitation, not a permissions bug to chase. An smbclient -k pull attempted from sidious itself failed differently again, on SPNEGO negotiation. About forty minutes of live troubleshooting sits behind the runbook’s single recommended line. The approach that finally worked sidesteps the SMB layer entirely — it’s inelegant, and it’s copy-pasteable. Zip SYSVOL on the Windows DC:

Compress-Archive -Path "C:\Windows\SYSVOL\sysvol\ad.tod.net\*" `
  -DestinationPath "C:\sysvol-export.zip"

Ferry it across in two scp hops via the Ansible controller — the one machine that already had SSH trust to both ends:

scp '[email protected]:C:/sysvol-export.zip' /tmp/sysvol-export.zip
scp /tmp/sysvol-export.zip [email protected]:/tmp/

Then unzip into place on sidious and rebuild the ACLs. The zip carries no Windows security descriptors at all, and it doesn’t matter: samba-tool ntacl sysvolreset regenerates every POSIX ACL and xattr from the Samba database, which is why this workaround is safe rather than merely expedient.

cd /var/lib/samba/sysvol/ad.tod.net
unzip /tmp/sysvol-export.zip
samba-tool ntacl sysvolreset
ls Policies/   # both default-policy GPO GUIDs must be present

One loop here stayed open far longer than it should have, and it’s only fair to say so: confirming that Group Policy actually applies from the rebuilt SYSVOL. The check was drafted during the DNS cutover, promptly pasted into the wrong terminal — a TrueNAS shell that had never heard of gpresult — and then skipped as confirmatory rather than load-bearing once auth was proven end to end. It finally ran seventeen days later, during this post’s review:

PS C:\Users\tod.AD> gpupdate /force
Computer Policy update has completed successfully.
User Policy update has completed successfully.

PS C:\Users\tod.AD> gpresult /r
...
    Group Policy was applied from:      sidious.ad.tod.net

    Applied Group Policy Objects
    -----------------------------
        Default Domain Policy
        Enable Windows Hello for Business
        Local Group Policy

A forced pull, served by sidious, real GPOs applied. Loop closed — late, but closed.

The cutover itself

With SYSVOL in place, DNS moved next: pfSense’s Unbound resolver, which had been forwarding ad.tod.net queries to the Windows DCs, got repointed at sidious. Verified by watching tcpdump on sidious itself and confirming pfSense really was forwarding AD queries there — this Linux workstation’s SSSD session picked up a valid Kerberos ticket with no re-login required.

FSMO transfer came next — all five roles, one samba-tool fsmo transfer --role=all call:

$ samba-tool fsmo transfer --role=all
FSMO transfer of 'schema' role successful
FSMO transfer of 'naming' role successful
FSMO transfer of 'infrastructure' role successful
FSMO transfer of 'rid' role successful
FSMO transfer of 'pdc' role successful
DomainDnsZones - Attempt to transfer DomainDNS role: Server does not have a current owner
ForestDnsZones - Attempt to transfer DomainDNS role: Server does not have a current owner

The two “no current owner” lines are expected — Samba’s internal DNS backend has no equivalent role to hold there — not a failure. Becoming PDC emulator meant sidious now had to serve authoritative, cryptographically signed time (Kerberos has no tolerance for clock skew), which took a ntp signd socket directory line in smb.conf and a matching ntpsigndsocket in chrony.conf. The plan document had flagged this as a likely rough edge worth extra caution; live, it came up clean on the first try — one of the few pieces of the day that went exactly as planned.

Then maul — the DC holding no other roles, running no other services — went through Uninstall-ADDSDomainController and was powered off. The post-demotion integrity check ran on sidious, pasted straight from the runbook’s checklist — trailing comment and all. The comment records what the runbook expected; the output records what happened:

root@sidious:~# samba-tool dbcheck --cross-ncs # 0 errors
Checking 4077 objects
...
Checked 4077 objects (11 errors)
Please use 'samba-tool dbcheck --fix' to fix 11 errors

(--cross-ncs makes dbcheck walk every naming context — the configuration and DNS application partitions too, not just the domain itself, which is where most of this debris lives.)

All eleven were pre-existing and inert: the two SPIKE orphans left over from the Phase 0 test, some stale DomainDNS/ForestDNS role-owner pointers to objects that no longer existed, and instanceType mismatches that are a known false-positive of Samba/Windows interop. None were actionable, and none blocked anything that came after.

A wrong answer that changed the plan

One exchange from that evening is worth telling in full, because it’s the AI-collaboration failure mode this series keeps returning to. vader — the second Windows DC — was never DC-only; it also ran the domain’s AD CS certificate authority and Entra Connect (the Microsoft 365 directory sync). The original migration plan had treated demoting vader’s DC role and dealing with Entra Connect as one gated step, on the assumption that Entra Connect needs to run on a domain controller.

Asked directly whether that was actually true, Claude’s first answer just repeated the plan document’s own claim back — demoting vader would break the sync. Pushed on it, Claude’s real answer came out on rederivation: Entra Connect doesn’t need to run on a DC at all; Microsoft’s own guidance recommends a dedicated member server. The forced “Entra go/no-go” gate that had been sitting in Phase 6 came out of the plan entirely, deferred instead to whenever vader is actually decommissioned as its own, later project. The plan was wrong, and moved because of a question it should have already had a correct answer to.

The DC that went dark instead of being demoted

The runbooks say Phase 6 Part B — demoting vader’s DC role — was “deferred a few days.” After MAUL’s demotion succeeded that same evening, the actual line was this:

let’s leave vader for a few days before we shut it down

— Tod

No further technical blocker got named in any session since.

What happened next didn’t show up in any runbook, because it wasn’t part of the migration plan at all. A week later, the morning after taking real power measurements for the second post in the Downsizing series, chimaera — one of the two old Supermicro servers — was powered down for good, evening of June 28. Home Assistant’s own power-sensor history pins it precisely: draw fell from a steady ~155 W to about 10 W — the iLO management chip’s standby draw, all that’s left running — between 5 and 6 p.m. that day. vader and piett (the Veeam backup server, its own separate Windows dependency) came down around the same time, for the same two reasons at once: it saves real power, and it’s the only honest test of whether sidious can actually carry the domain alone.

That test ran for ten days with nothing coming back to say it was failing. But it’s worth being precise about what “powered off” meant versus what the runbook describes: vader was never formally demoted during those ten days. Uninstall-ADDSDomainController was never run against it. As far as Samba’s own replication metadata was concerned, vader was still a domain controller — just one that had stopped answering. That’s a materially different state from a clean member-server shutdown, and it’s not free: a stale DC’s replication object is exactly the kind of thing that turns into cleanup work later (samba-tool domain demote --remove-other-dead-server=, the same command that scrubbed the Phase 0 spike’s leftovers). Nothing about the domain was broken by it. It wasn’t finished, either.

The demotion that fought back

It stopped being unfinished on the evening of July 8, ten days after the power-down, when chimaera came back on for a single evening specifically to end it. Tod’s opening message carried the reasoning inline:

since it was off for several days without issue, I think it is safe to demote. Let’s do that.

— Tod

The graceful demotion the runbook described turned out to be impossible as written — twice over. The first attempt failed before it began:

PS C:\Windows\system32> Uninstall-ADDSDomainController
Uninstall-ADDSDomainController : Verification of prerequisites for Domain
Controller promotion failed. Certificate Server is installed.

Windows refuses to demote a domain controller that is also a certificate authority — full stop. The migration plan’s working assumption — that vader’s AD CS role would simply ride along on the demoted member server until the decommission project dealt with it — could never have worked. So a decision that had been comfortably deferred for weeks got made on the spot, mid-demotion: dump the issued-certificate inventory (certutil -view … csv showed nothing but workstation auto-renewals since 2021; the EFS-carrying user certificates all expired years ago, and expired EFS certificates still decrypt what they encrypted), back the CA up off-box (Backup-CARoleService for the private key and database, a registry export for the config), and remove the role: Remove-WindowsFeature ADCS-Cert-Authority, AD-Certificate.

With AD CS gone, the demotion got further before failing differently:

PS C:\Windows\system32> Uninstall-ADDSDomainController -DemoteOperationMasterRole `
    -RemoveApplicationPartitions -Force
Uninstall-ADDSDomainController : The operation failed because:
Active Directory Domain Services could not find another Active Directory
Domain Controller to transfer the remaining data in directory partition
DC=ForestDnsZones,DC=ad,DC=tod,DC=net.
"The specified domain either does not exist or could not be contacted."

The first cause was debris this post has already named: the two DNS application partitions’ fSMORoleOwner attributes pointed at tombstones — four of June 21’s eleven dbcheck errors — so there genuinely was no owner to hand anything to. Seizing both roles onto sidious took two commands:

$ sudo samba-tool fsmo seize --role=domaindns --force
FSMO seize of 'domaindns' role successful
$ sudo samba-tool fsmo seize --role=forestdns --force
FSMO seize of 'forestdns' role successful

The retry failed with the identical error anyway. That’s when it stopped being about debris and became structural: before Windows gracefully releases a DNS application partition, it verifies another live host of that partition via the DC locator — and Samba never registers itself as one:

PS C:\Windows\system32> nltest /dsgetdc:ForestDnsZones.ad.tod.net
Getting DC name failed: Status = 1355 0x54b ERROR_NO_SUCH_DOMAIN

The domain itself resolved fine; the application partitions never will. Gracefully demoting the last Windows DC out of a Samba-run domain looks structurally impossible from where this lab sat: nothing tried here — metadata surgery included — made that check pass, because the check wants a live DC-locator answer that Samba never gives. If some obscure override exists that fakes one, it stayed hidden. The exit was the forced path, in three moves. On vader, the demotion variant that skips the transfer check entirely:

Uninstall-ADDSDomainController -ForceRemoval -DemoteOperationMasterRole -Force

Then on sidious, scrubbing the now-dead DC out of the directory — the exact command the previous section predicted this state would eventually require:

sudo samba-tool domain demote --remove-other-dead-server=VADER

And because forced removal drops the machine into a workgroup rather than down to domain membership, the last move was joining vader back into the domain it had just stopped ruling — Add-Computer, a fresh computer account, one reboot, an ordinary member. An hour and change after chimaera powered on, in the same flat register as everything else in this story:

vader has been removed and is back up

— Tod

The end state is cleaner than the migration itself ever achieved. The post-removal recheck found thirteen dbcheck errors — June 21’s accumulated eleven plus fresh VADER tombstone pointers knocked loose by the removal, all visible in the output — and a single --fix pass cleared every one:

$ sudo samba-tool dbcheck --cross-ncs --fix --yes
...
$ sudo samba-tool dbcheck --cross-ncs
Checked 4079 objects (0 errors)

Zero errors — the SPIKE orphans included, all finally gone — and sidious holds all seven roles (five FSMO plus both DNS partitions) as the domain’s only DC. vader survives as a plain domain member, still running Entra Connect — the corrected answer from the exchange above, now demonstrated rather than asserted. AD CS is restorable from its backup if something turns out to have needed it. What’s left of the loose ends⁴ is stale DNS records — SRV entries for domain controllers that no longer exist, A records for machines long gone — real, minor, and tracked. And vader, piett, and chimaera are still physically present, un-decommissioned: that remains a separate, gated event, saved for this series’ eventual finale post, not this one.

What this post is and isn’t

This is the DC migration story: two Windows domain controllers replaced by one Samba box, in-place, with every SID intact and not one workstation re-joined. It is not the retirement story — the actual decommission of the old hardware is still ahead, whenever that gets scheduled. The measured, honest version of “this went well” is: the domain moved in a night and a day split awkwardly by an AI’s own rate limit, the runbooks understate how many of the individual steps failed before they passed — right through the final demotion, which proved impossible as written and shipped as a forced removal seventeen days later — and the directory ended the whole saga at zero dbcheck errors, its remaining mess reduced to stale DNS records, some older than the migration and some born of it.

AI-assistant disclaimer: This post was drafted by Claude Code (Claude Sonnet 5, claude-sonnet-5) from project notes, commit history, decision records, and session transcripts, then reviewed and edited by me. It may contain inaccuracies; verify specifics before relying on them.