The Mastodon Migration

The previous post ended with two weeks of wrong turns and an honest need for something to just work. The same day FreeIPA got shelved — June 19 — I got it: Mastodon came off a dedicated Ubuntu VM running a source-checked-out install and onto official Docker images running in a Proxmox LXC, and every part of it went the way infrastructure work is supposed to.

What was there before

Mastodon had been running on a dedicated Ubuntu 24.04 VM (mastodon.tod.net) since long before this rebuild started — a source-checked-out Ruby/Node install managed by rbenv and nvm, the deployment model the Mastodon project’s own install guide walks you through. mastodon.tod.net reaches the public internet the same way every other externally reachable service here does: a cloudflared tunnel, so nothing about the host itself — old VM or new LXC — needs an inbound firewall rule. It worked, but every version bump meant babysitting Ruby and Node versions by hand, and the data volume was 71 of 95 GB used, with roughly 64 GB of that being nothing but a remote-media cache — other servers’ avatars and post images, fetched and stored locally the first time anyone here saw them, then never cleaned up automatically enough to keep pace.

Two decisions were on the table, and one made the other cheaper.

Media first: RustFS instead of local disk

Rather than migrate the VM’s entire 71 GB filesystem into a new container, media storage moved first to RustFS — an open-source, S3-compatible object storage server, running here as its own app on the TrueNAS box, not a filesystem or a TrueNAS-native feature — before touching the container question at all. Getting RustFS itself stood up and trusted was its own piece of work, not covered here.

This part ran over VPN — my son was at karate practice, and running infrastructure work from a YMCA lobby rules out anything that can’t survive a dropped connection. The pre-sync itself was safe by design — aws s3 sync is idempotent, so re-running it after an interruption just picks up where it left off — but the shell running it wasn’t, so both syncs ran inside a detached screen session on the host instead of directly over the SSH pipe:

ssh mastodon.tod.net "screen -dmS mastodon-sync bash -c '
for dir in accounts media_attachments site_uploads; do
    echo \"=== Syncing \$dir ===\"
    aws s3 sync /home/mastodon/live/public/system/\$dir/ \
        s3://mastodon/\$dir/ \
        --endpoint-url https://rustfs.tod.net:30293 --profile rustfs
done'"

Pruning the disposable cache first, so the container migration wouldn’t have to carry it, turned out to be less mechanical than the plan assumed. The first attempt at Mastodon’s own tootctl admin CLI¹ failed outright — a Bundler error over a gem that needed reinstalling — and even after that was fixed, the straightforward sudo -u mastodon invocation failed again because rbenv’s shims tried to cd into a directory the mastodon user couldn’t read. Running from /tmp with an explicit environment sidestepped it:

ssh mastodon.tod.net "cd /home/mastodon/live && sudo -u mastodon bash -c 'export PATH=\"/home/mastodon/.rbenv/bin:/home/mastodon/.rbenv/shims:\$PATH\" && eval \"\$(rbenv init -)\" && bundle install'"

ssh mastodon.tod.net "cd /tmp && sudo -u mastodon env HOME=/home/mastodon PATH=/home/mastodon/.rbenv/bin:/home/mastodon/.rbenv/shims:/usr/local/bin:/usr/bin:/bin RAILS_ENV=production /home/mastodon/live/bin/tootctl media remove --days=7"

Even once it ran, --days=7 barely touched anything — only 606 MB, because most of the 64 GB cache was recent enough to survive a 7-day cutoff. A du across the system/ subdirectories confirmed the numbers from the plan were otherwise right (cache/ at 64 GB, everything that actually needed to move under 12 MB combined), so the cache itself wasn’t the problem — pruning it with a short retention window just wasn’t the way to clear it before syncing the permanent media.

With the AWS CLI installed — apt install awscli had no installation candidate on this Ubuntu 24.04 image, so it came from AWS’s own installer instead — the permanent media synced in seconds:

for dir in accounts media_attachments site_uploads; do
    aws s3 sync /home/mastodon/live/public/system/$dir/ \
        s3://mastodon/$dir/ \
        --endpoint-url https://rustfs.tod.net:30293 --profile rustfs
done

67 objects, about 10.6 MB, confirmed present in the bucket afterward.

With that pre-synced, the actual cutover was: back up the nginx config, edit it, test it, then reload. The edit itself needed one detour — a Python patch script run over SSH kept silently failing to match the block, because the local shell was expanding $uri before it ever reached the remote host. Editing a local copy and scp-ing it back sidestepped the escaping problem entirely:

# OLD — one combined block served everything, local FS only
location ~ ^/(emoji|packs|system/accounts/avatars|system/media_attachments/files) {
    add_header Cache-Control "public, max-age=31536000, immutable";
    add_header Strict-Transport-Security "max-age=31536000" always;
    try_files $uri @proxy;
}

# NEW — split: local static files stay local, /system/ proxies to RustFS
location ~ ^/(emoji|packs) {
    add_header Cache-Control "public, max-age=31536000, immutable";
    add_header Strict-Transport-Security "max-age=31536000" always;
    try_files $uri @proxy;
}

location /system/ {
    proxy_set_header Host rustfs.tod.net;
    proxy_set_header Authorization "";
    proxy_pass https://rustfs.tod.net:30293/mastodon/;
    proxy_buffering on;
    proxy_cache CACHE;
    proxy_cache_valid 200 7d;
    proxy_cache_valid 410 24h;
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    add_header X-Cached $upstream_cache_status;
    add_header Cache-Control "public, max-age=31536000, immutable";
}

A stray .bak file left in sites-enabled/ from the backup step made nginx -t fail until it was moved out of the directory nginx actually loads configs from. With that cleared, the config tested clean and reloaded — and a curl against a known file confirmed /system/ was already proxying to RustFS correctly, before a single S3 environment variable had even been added to Mastodon itself.

With nginx proved out, the rest followed the plan: stop the Mastodon services, run the sync loop once more to catch anything written since the pre-sync, add the S3 variables² to .env.production — the single file holding every Mastodon configuration value, secrets included, at /home/mastodon/live/.env.production on the old VM — and start the services back up.

sudo systemctl stop mastodon-web mastodon-sidekiq mastodon-streaming@4000
# final delta sync (repeat the loop above)
# then append to .env.production:
#   S3_ENABLED=true
#   S3_BUCKET=mastodon
#   S3_ENDPOINT=https://rustfs.tod.net:30293
#   S3_ALIAS_HOST=mastodon.tod.net/system
sudo systemctl start mastodon-web mastodon-sidekiq mastodon-streaming@4000

S3_ALIAS_HOST is what keeps media URLs looking like mastodon.tod.net/system/... instead of exposing RustFS’s own hostname. Verification was a repeated curl -I against the same file — the first request came back 200 with no X-Cached header at all, and only a second request against the identical URL showed X-Cached: HIT, confirming nginx’s own cache (not just the proxy) was doing what it was supposed to.

The sequencing mattered more than the storage choice itself. With media in S3 before the container migration started, the only thing left to migrate into the new LXC was the database — a few hundred megabytes, not tens of gigabytes. The 64 GB cache never had to move at all; it’s disposable by design, and it just refills in RustFS as other servers’ posts get fetched again.

The container question: Docker, not another source install

The LXC migration was the natural point to also fix the deployment model, and there were really only two options³:

• Source install on LXC — reproduce the same rbenv/nvm/bundler setup inside a container. Familiar, but it carries the same per-version dependency babysitting forward indefinitely.
• Docker Compose with official images — ghcr.io/mastodon/mastodon, the images the Mastodon project itself builds and tests. An upgrade becomes a tag bump, a database migration, and compose up -d.

Docker won. It’s also the only Docker guest anywhere in this fleet — everything else here is either an LXC running a native package install or a Terraform-provisioned VM, and Docker-in-LXC is a nested arrangement with its own overhead, not the leanest possible option. The previous post’s whole argument for Proxmox was native LXC over full VMs — this trades a slice of that back for an upgrade path that doesn’t require hand-rolling the Ruby/Node stack a second time. Architectural purity lost to not wanting to think about rbenv ever again.

services:
  web:
    image: ghcr.io/mastodon/mastodon:{{ mastodon_version }}
    restart: unless-stopped
    env_file: .env.production
    command: bundle exec puma -C config/puma.rb
    ports:
      - "127.0.0.1:3000:3000"
    depends_on:
      db: { condition: service_healthy }
      redis: { condition: service_healthy }
    extra_hosts:
      - "host.docker.internal:host-gateway"

  streaming:
    image: ghcr.io/mastodon/mastodon-streaming:{{ mastodon_version }}
    command: node ./streaming/index.js
    # ...

  sidekiq:
    image: ghcr.io/mastodon/mastodon:{{ mastodon_version }}
    command: bundle exec sidekiq
    # ...

Postgres and Redis run as containers too, with named volumes — the only state that needs backing up now is those volumes and the S3 bucket, not a sprawling application directory. nginx stays on the Mastodon LXC host itself rather than in Docker³: TLS termination and the S3 proxy belong at the host layer, in front of whatever’s running behind them. That’s a separate nginx instance from the reverse proxy fronting other services in the fleet — this one only ever handles Mastodon’s own traffic.

Small correction while writing this: ADR 0014 originally justified that placement as “consistent with how GitLab is deployed.” It isn’t — GitLab doesn’t run in Docker here at all, so there was no comparable decision to be consistent with. Claude wrote that line into the ADR without checking it, and it went unquestioned for two weeks until drafting this post raised the question. Fixed in the ADR now. A small reminder that these tools are genuinely useful but still need a second set of eyes — the same as anyone’s work.

Version was pinned to v4.5.9 — identical to the outgoing VM — specifically so the migration itself wouldn’t also be a schema migration. Get the move done first, upgrade later once the new deployment is confirmed stable.

Where the plan and reality disagreed

Two details in the Terraform were carried over from the closest precedent (gitlab.tf, signing_vm.tf) and turned out to be wrong for this specific VLAN.

The container needed keyctl = true in its features block to run Docker inside an unprivileged LXC — or so the plan assumed, reasoning from how Docker-in-LXC guides usually read. In practice nesting = true alone was sufficient; keyctl additionally requires root@pam Proxmox permissions this deployment didn’t have. Since nesting alone proved enough, there was no reason to go grant those broader permissions just to satisfy an assumption that turned out to be wrong — not a deliberate least-privilege call made in advance, just one less permission to have granted once it turned out to be unnecessary. It came out in the same commit that removed the other carried-over assumption: a unifi_user fixed-IP reservation, copied from the pattern signing_vm.tf established for VLAN 22. Mastodon lives on VLAN 2222 (“Hosting”), and that subnet is routed and DHCP’d by pfSense, not UniFi — there’s no UniFi network object to reserve a fixed IP against. The container’s static IP is set directly in Terraform’s ip_config instead, and pfSense needed no separate reservation at all.

resource "proxmox_virtual_environment_container" "mastodon_container" {
  unprivileged = true

  features {
    nesting = true
  }

  initialization {
    hostname = "mastodon"

    dns {
      servers = [var.vlan2222_gateway]
      domain  = "tod.net"
    }

    ip_config {
      ipv4 {
        address = var.mastodon_ip_address
        gateway = var.vlan2222_gateway
      }
    }
  }

  network_interface {
    name        = "eth0"
    mac_address = local.mastodon_mac_address
    vlan_id     = 2222
  }
}

Small corrections, neither one a real setback — but a reminder that the fleet’s existing Terraform patterns are precedent, not gospel. Each new guest gets checked against its own network and permissions rather than copied wholesale.

Cutover

With media already in S3 and the container built, the plan called for a straightforward pg_dump → copy → pg_restore → rails db:migrate sequence. What actually ran needed two adjustments the plan hadn’t accounted for:

# On the old VM
sudo -u mastodon pg_dump -Fc mastodon_production -f /tmp/mastodon.dump

# Copy to the new LXC
scp mastodon.tod.net:/tmp/mastodon.dump /tmp/mastodon.dump
scp /tmp/mastodon.dump 172.22.22.9:/tmp/mastodon.dump

# On the new LXC — the db container wasn't started yet, so it had to
# come up first (and be waited on for healthy) before anything else:
sudo docker compose -f /opt/mastodon/docker-compose.yml up -d db

# The dump is custom-format (-Fc). docker compose exec has no host-path
# access, so the file has to be copied into the container first. And
# .env.production is mode 0600 owned by root, so every docker compose
# call on this host needs sudo — the plan's commands assumed otherwise:
sudo docker compose -f /opt/mastodon/docker-compose.yml cp /tmp/mastodon.dump db:/tmp/mastodon.dump
sudo docker compose -f /opt/mastodon/docker-compose.yml exec db \
    pg_restore -U mastodon -d mastodon_production --no-owner /tmp/mastodon.dump

# Then apply any schema migrations the new image version needs — this
# also brings up redis and web for the first time:
sudo docker compose -f /opt/mastodon/docker-compose.yml run --rm web rails db:migrate

The 825 MB dump moved and restored cleanly once those two things were sorted out. DNS pointed mastodon.tod.net at the new container’s address once the stack came up healthy, and the cloudflared tunnel moved from the old VM to the new LXC in the same maintenance window.

Because the media proxy config was already identical between the old VM’s nginx and the new LXC’s, and the database was the only thing that actually moved, there was no user-visible gap. The old VM stayed up, untouched, for a few days afterward as a rollback path, then got shut down and later destroyed.

The upgrade path got used almost immediately

The whole point of switching to official Docker images was to make upgrades boring, and that got tested the same day the migration finished. Two security patches had landed upstream since v4.5.9 — an SSRF bypass in .10, and an attribution-domain-spoofing plus message-sanitization fix in .11 — so the version bump to v4.5.11 happened immediately, and the playbook picked up the automation to make that routine going forward:

- name: Pull Mastodon Docker images
  ansible.builtin.command:
    cmd: docker compose pull
    chdir: "{{ mastodon_dir }}"

- name: Run Mastodon DB migrations
  ansible.builtin.command:
    cmd: docker compose run --rm web bundle exec rake db:migrate
    chdir: "{{ mastodon_dir }}"

- name: Restart Mastodon stack
  ansible.builtin.command:
    cmd: docker compose up -d
    chdir: "{{ mastodon_dir }}"

mastodon_version in group_vars/mastodon/vars.yml is now the entire upgrade procedure: bump the tag, run the playbook, done. No rbenv, no nvm, no reading a Ruby-version compatibility matrix before touching anything.

A second upgrade found the gap the migration missed

A further bump — v4.5.11 to v4.6.3, the latest release as of this writing — ran a couple of weeks later, and it’s worth covering because it did two things at once: it confirmed the upgrade path still holds under a real minor-version jump, and it caught a problem the S3 migration itself had quietly created.

The remote-media cache — the ~64 GB this post opened with, dismissed as disposable and left behind on purpose — doesn’t stay small just because it’s disposable. MEDIA_CACHE_RETENTION_PERIOD⁴ was unset both in the Ansible templates and live on the container, and nothing had ever pruned the cache automatically — so it just grew.

By the time of this upgrade, Mastodon’s own admin dashboard claimed 53.7 GB of media storage in use. RustFS’s bucket dashboard told a different story: about 5 GB actually sitting in the bucket. That gap needed chasing down before touching anything — a discrepancy that large is just as likely to mean “the dashboard is lying” as “the database is.” tootctl media usage¹ turned out to be a live SQL sum over Mastodon’s own database columns, not a cached statistic, confirmed against Mastodon’s source directly rather than assumed⁵ — so the 53.7 GB was real according to the database, mostly cached avatars and headers (15.2 GB + 33.5 GB) for federated accounts nobody here follows. A dry run confirmed roughly 46.8 GB of that was safe to reclaim, and the real prune reported removing exactly that — Mastodon’s own usage total dropped to about 6.9 GB afterward. Whether the RustFS bucket dashboard was under-reporting the whole time or the two numbers were never quite measuring the same thing wasn’t fully run to ground; what’s certain is the database’s own bookkeeping shrank by 46.8 GB and stayed there. The original plan’s assumption that the cache was safe to lose was correct. Its unstated assumption that it would stay bounded on its own was not.

That gap is now closed with a weekly cron, not a one-time cleanup:

- name: Cron job to prune remote media cache weekly
  ansible.builtin.cron:
    name: "mastodon: prune remote media cache"
    weekday: "0"
    hour: "3"
    minute: "30"
    user: root
    job: >-
      cd {{ mastodon_dir }} &&
      docker compose exec -T web bin/tootctl media remove --days 30 &&
      docker compose exec -T web bin/tootctl media remove --prune-profiles --days 30
      >> /var/log/mastodon-media-prune.log 2>&1

The upgrade also added a safety net that the v4.5.11 bump hadn’t bothered with: a Proxmox snapshot plus a logical pg_dump before pulling a new image, gated by an interactive prompt (or --tags/--skip-tags pre-upgrade-backup for non-interactive runs) so routine upgrades still default to being backed up rather than trusting db:migrate to always go cleanly. Verification afterward was the same as the first bump: containers on v4.6.3, API responding 200, no custom themes to carry forward.

Two data points now, not one — the Docker-images bet keeps paying out, and the second upgrade paid for itself by finding a real gap in the first migration.

The win, and what it cost to get here

One more thing worth naming, since it shares the same day: proxmox-mini01 — the cluster node everything defaulted to — was sitting at 95% RAM while this migration went in. GitLab alone was 7.25 GB of that, more than half the node’s load, because Terraform’s default target node had quietly become a dumping ground for every guest. Mastodon was on that node too, at well under a gigabyte, and had nothing to do with it. The fix was a single live migration of GitLab to the cluster’s idle second node, done the same day.

Worth being honest about the other side of that, too: Claude has brought this event up as bigger than it was more than once, including once drafting it into an earlier post’s teaser as “the placement mistake that nearly ran one of the nodes out of memory” — dramatic enough that it never survived review and never reached the live site. A capacity note got treated like a near-miss, and it took catching that pattern more than once to keep it out of what actually got published.

Nothing about this migration was clever. Media moved first so the container migration wouldn’t also be a data migration. Docker images replaced a hand-built language toolchain so upgrades stop being a research project. Two Terraform assumptions, copied from other guests, turned out to be wrong for this VLAN and got corrected in the same sitting they were noticed. Even the one thing the migration got wrong — an unbounded cache that took weeks to surface — got caught by the same boring upgrade process it was betting on, not by disk usage sneaking up unnoticed. That’s the whole story — which, after the acme-proxy dead end and three failed attempts at FreeIPA, was exactly the point.

The FreeIPA detour in the previous post was really about identity — the piece that never got a proper resolution here was what runs ad.tod.net instead. That story starts a new thread, Off Windows: replacing a long-running Windows Active Directory domain with Samba, one domain controller at a time, without breaking a single workstation login. It turned out there was a whole evening of prep work worth its own post before that migration even started — the domain replacement itself follows in the post after that.

AI-assistant disclaimer: This post was drafted by Claude Code (Claude Sonnet 5, claude-sonnet-5) from project notes, commit history, and decision records, then reviewed and edited by me. It may contain inaccuracies; verify specifics before relying on them.