Blog/Six Months of Deployment Chaos: What Getting D4 Chess Club™ to Production Actually Looked Like
DevelopmentBehind the Scenes

Six Months of Deployment Chaos: What Getting D4 Chess Club™ to Production Actually Looked Like

Shane Foster·March 25, 2026·9 min read

There is a version of the D4 Chess Club™ deployment story that is clean and professional — where we carefully staged our production environment, ran comprehensive pre-launch checks, and shipped a flawless v1 to a waiting audience. That version does not exist.

The real version involved deploying from the wrong directory for an entire week without realizing why the migrations weren't running. It involved discovering that two critical database tables simply didn't exist in production — the puzzle attempts table and the generated puzzles table — on the day we were supposed to go live. It involved an email invite system that silently failed because the sending domain wasn't verified, and we spent two weeks assuming the bug was in our code.

Here is the honest account.

The Wrong Directory Problem

D4 Chess Club™ is a monorepo: frontend lives in /frontend, backend in /backend. The Fly.io deployment for the backend needs to run from the /backend directory — the Dockerfile is there, the fly.toml is there, the migration script references paths relative to that directory.

For an embarrassingly long time, I was running fly deploy from the repository root. The error was "dockerfile not found," which I kept solving by pointing to the Dockerfile path explicitly — which then failed because the build context was wrong. The fix, when I finally realized the problem, was trivially obvious: cd backend && fly deploy. Every deployment after that worked first time. The preceding week of confusion was entirely self-inflicted.

The Missing Tables

We use Alembic for database migrations. The migration system was working — new migrations ran on every deploy. But we discovered, the day before launch, that two tables we needed for core functionality didn't exist in production: puzzle_attempts and generated_puzzles.

The tables had been added in migrations written during a period when the local dev database and production were slightly out of sync. The migrations existed but had never been applied to production because we'd been applying them manually during that period and then forgot to verify they'd run. The Alembic version in production was behind.

The fix was running fly ssh console into the production container and applying the migrations manually, then verifying with a database query. Both tables appeared. The launch proceeded. But the lesson — verify the actual production database state against your expected schema before any launch — is one we now do systematically.

The Email Domain Problem

The invite system failure was the most frustrating because the error was entirely silent for two weeks. We were using Resend to send invitation emails. The RESEND_API_KEY was set. The code was correct. The API calls were returning success codes. But emails were not arriving.

The problem was domain verification. Resend requires that you verify ownership of your sending domain — in our case, d4chess.club — by adding DNS TXT records in Porkbun. We had set up the Resend account, added the API key to our environment, and assumed that was sufficient. It was not. Without domain verification, Resend accepts the API calls and returns success, but does not actually deliver the emails.

We discovered this when we finally added proper error logging to the email sending function — logging the full Resend API response body, not just the status code. The response body, which we'd been ignoring, contained a clear domain verification error. Twenty minutes and a DNS record later, emails were delivering. Two weeks of debugging vanished.

CORS, Environment Variables, and Other Classic Mistakes

The CORS configuration took three iterations to get right — specifically, getting the production frontend domain into the allowed origins list in a way that survived deployments. Environment variables in Vercel that were set for preview deployments but not for production caused subtle issues where the app behaved differently on the production domain than it did in staging. A missing NEXT_PUBLIC_API_URL caused the registration page to hit the wrong endpoint for weeks — the bug that was causing user registrations to silently fail.

None of these are novel problems. They're the standard set of issues every full-stack application runs into going from local development to production. What made them costly was the combination of asynchrony (some failures only manifested under real user conditions), silence (errors that didn't surface in logs without explicitly instrumenting for them), and the general truth that production environments are not local environments, no matter how carefully you try to make them match.

What We'd Do Differently

Three things. First: deploy to production earlier, even with incomplete features, so the production environment is exercised and known-good before it matters. Second: add explicit health checks and integration tests that run against the actual production API — not a mock, not a local instance, the real thing. Third: never trust "success" from an external API without verifying the downstream effect actually occurred. A 200 response means the request was received, not that the intended outcome happened.

D4 Chess Club™ is stable now. But it got there through finding and fixing every one of those problems the hard way. We've documented them here so that someone else doesn't have to.

Related Reading

Ready to improve?

Try D4 Chess Club™ free — no credit card required

AI coaching, 54,000+ puzzles, Stockfish analysis. Built for the player who wants to get better, not just play more.

Start Training Free