Post Incident Report: Tophhie Social – 23rd June 2026

TL;DR

• I enabled Bot Fight Mode on tophhie.cloud and tophhie.dev on Saturday 20th June 2026 at 22:09.
• Two days later Tophhie Social went dark behind a steady HTTP 525 (Cloudflare <-> origin TLS handshake failure).
• The certificate was valid and on disk. The backend was healthy. The TLS-check endpoint returned 200 to every manual test. Everything lied.
• Root cause: Caddy's on-demand TLS asks a Tophhie Cloud endpoint (api.tophhie.dev/pds/tls-check) mid-handshake to decide whether to serve a cert. Caddy's Go HTTP client looked like a bot to Bot Fight Mode, got blocked, the GetCertificate callback threw, and the handshake died with tlsv1 alert internal error.
• Fix: disable Bot Fight Mode for the tophhie.dev and tophhie.cloud domains.

---

The setup

Tophhie Social runs on a self-hosted AT Protocol PDS — the same bluesky-social/pds image that backs Bluesky. In front of it sits Caddy, doing TLS termination and reverse-proxying to the PDS on localhost:3000, with a gatekeeper service in the middle for sensitive XRPC endpoints.

The interesting bit is the TLS. Because every account on a PDS gets its own handle subdomain (alice.tophhie.social, bob.tophhie.social, and so on), I can't pre-provision certificates for names that don't exist yet. So Caddy uses on-demand TLS: it issues a certificate the first time it sees a handshake for a given hostname. To avoid becoming an open certificate vending machine, on-demand TLS calls an 14 endpoint before issuing — a small API that answers "yes, that domain is allowed" or "no, get lost":

{
    on_demand_tls {
        ask https://api.tophhie.dev/pds/tls-check
    }
}

That ask endpoint lives on 16 — a different domain from the PDS itself. Hold that thought. It matters more than anything else in this story.

The whole thing sits behind Cloudflare in Full mode.

What I saw

I woke up to a dead PDS. From the outside, every request returned a flat HTTP 525 — Cloudflare's "I connected to your origin, but the TLS handshake failed" error.

The maddening part was that from the inside, nothing looked wrong:

• The PDS container was up and answering GET /xrpc/_health with 200 on localhost:3000.
• Caddy was up, config valid, listening on :443.
• Disk was at 44%. No memory alarms.
• The certificate for pds.tophhie.cloud was sitting in Caddy's storage, valid from 10 May to 8 Aug 2026 — six weeks of life left.
• The ask endpoint returned 200 to every curl I threw at it, including from inside the Caddy container.

A valid cert, a healthy backend, a reachable validation endpoint, and a completely unreachable service. Cool. Cool cool cool.

The investigation, and every lie it told me

This is the part worth reading, because the lesson isn't the root cause — it's how many plausible-but-wrong conclusions the evidence cheerfully supported.

Lie #1: "It's the 25 endpoint." My first instinct was the on-demand dependency. So I tested it — 200 for pds.tophhie.cloud, 200 for a *.tophhie.social handle, 403 for a bogus domain. Textbook-correct behaviour. Endpoint exonerated. (It was, in fact, the guilty party. More on why the test lied later.)

Lie #2: "The backend's down." The first Caddy logs showed dial tcp 127.0.0.1:3000: connect: connection refused on a health probe. Looked like a smoking gun — until I realised it was a transient blip during a container recreate, and the PDS was back to serving 200s seconds later. A red herring wearing a hi-vis jacket.

Lie #3: "Watchtower bumped an image overnight." The core containers had all been recreated ~23 minutes earlier — fresh Created timestamps, not restarts. That's Watchtower's signature, and an auto-update of caddy changing on-demand behaviour was a beautiful theory. Watchtower's own logs showed no such update. Theory binned.

Lie #4: "The cert expired / storage got wiped." Easy to check, easy to disprove. The cert was valid and present. But here's the subtle trap: I ran find on the host filesystem. That proves the file exists on disk; it proves nothing about what Caddy-in-the-container can see and use. I nearly convinced myself the bind-mount had broken on the recreate.

Two things finally cut through the noise.

First, I hammered the public endpoint five times in a row and got 525 every single time — dead steady, sub-second, no flapping. That killed the "Caddy is crash-looping and you're catching it mid-restart" hypothesis. Caddy was up and consistently refusing the handshake.

Second — and this is the command that ended the guessing — I hit the origin directly, bypassing Cloudflare entirely, with full verbosity:

bash

curl -sv --resolve pds.tophhie.cloud:443:127.0.0.1 \
  https://pds.tophhie.cloud/xrpc/_health 2>&1 | grep -iE 'subject|issuer|alert|HTTP/'

* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, internal error (592):
* error:0A000438:SSL routines::tlsv1 alert internal error

tlsv1 alert internal error. This handshake never touched Cloudflare, so the 525 was always a downstream symptom — Caddy itself accepted the connection, read the ClientHello, and then its on-demand GetCertificate callback threw an error and returned an internal_error alert instead of a certificate.

That single alert reframed everything. The cert being valid on disk was irrelevant, because Caddy was failing before it ever reached for the cert. Something in the on-demand decision path was erroring out.

Root cause

The on-demand decision path is exactly that ask call to api.tophhie.dev/pds/tls-check. And on 20 June at 22:09, I had enabled Bot Fight Mode on tophhie.dev (and tophhie.cloud) while tightening things up across my zones.

Bot Fight Mode challenges traffic that looks automated. And Caddy's 43 call is automated traffic — a bare Go HTTP client with a minimal user-agent, making a machine-to-machine request mid-TLS-handshake. To Bot Fight Mode, that is, definitionally, a bot. So the call got challenged/blocked, the ask request failed, GetCertificate had no "yes" to act on, and it threw. No cert. internal_error alert. Cloudflare reports 525. PDS dark.

So why did my own curl tests pass? Because a human running 49 or 50 doesn't trip the same heuristics the way Caddy's Go client does — different user-agent, different request shape, and Bot Fight Mode's challenge landing inconsistently. The endpoint was "up" for me and "down" for the one client that actually mattered. The test wasn't measuring what I thought it was measuring.

Why it waited two days to detonate

The toggle flipped Saturday night, but the PDS didn't fall over until the early hours of Tuesday. That delay is the tidiest part of the whole thing.

On-demand TLS caches issued certificates in memory. As long as pds.tophhie.cloud was already in Caddy's in-memory cache, handshakes succeeded without re-consulting the ask endpoint. The blocked endpoint was a landmine, not an explosion — harmless until something made Caddy step on it.

That something was a container recreate in the early hours of Tuesday, which flushed the in-memory cache. The very next handshake forced a fresh on-demand decision, Caddy called the now-blocked ask endpoint, and boom. A latent change from Saturday detonating on Tuesday's restart is exactly why "what changed overnight?" produced a confident, useless "nothing."

The fix

Immediate resolution was simply disabling Bot Fight Mode. The PDS came straight back.

Hardening / action items

• Carve 54 out of bot protection. Bot Fight Mode itself can't be scoped with exceptions, so the route's protection moves to a WAF custom rule that 55s bot checks for http.request.uri.path eq "/pds/tls-check", ordered ahead of everything else.
• Add a shared-secret header. Caddy sends a secret header on the ask call; the endpoint only returns 200 when it matches. Now the route can be locked down hard and survive bot protection, because identity comes from the secret, not from "does this look human."
• Monitor it the way Caddy hits it, not the way I hit it. A synthetic check with a Go-ish / empty user-agent against tls-check, so the next time something challenges that endpoint I get paged in minutes — instead of reverse-engineering TLS alerts at the origin at stupid o'clock.
• Treat cross-zone security toggles as changes. This broke because a change on one domain silently took down a service on another. Anything that touches a zone an automated dependency lives in goes in the change log, full stop.

Takeaways

• A passing test can be the wrong test. My ask endpoint returned 200 all day — to me. The only client whose opinion mattered was Caddy, and I never tested as Caddy. When a dependency has a non-human caller, test as the non-human caller.
• Validate from the right vantage point. A cert on the host disk says nothing about the container's view; an endpoint that answers curl says nothing about how it answers a Go client behind a bot filter. Reproduce the failing path exactly, or you're just collecting reassuring noise.
• Latency between cause and symptom destroys "what changed?" In-memory caches, lazy revalidation, and on-demand anything will happily let a breaking change sit dormant for days. "Nothing changed overnight" was technically true and completely misleading.
• 525 is a symptom, not a cause. It means "origin TLS handshake failed," and the origin can fail that handshake for reasons that have nothing to do with Cloudflare. Going direct-to-origin with --resolve was the single highest-value diagnostic of the whole incident.

The certificate was valid. The backend was healthy. The endpoint was reachable. And the villain was a toggle I'd flipped two days earlier, on a domain I wasn't even looking at. Classic.