RAR staging and deploy pipeline

When I took this over, parts of the running system had been edited directly on the production server, outside version control. That is the actual starting condition of a lot of early-stage software, and it has one consequence that matters more than the untidiness: you cannot deploy, because nobody knows what deploying would overwrite.

So the work went in this order. Reconcile what was on the server back into the repository. Make the deployed branch the source of truth rather than the disk. Then build somewhere to test that is not production.

A runbook that rebuilds the box

Staging is a clean Contabo Ubuntu VPS built up in sixteen numbered steps: a dedicated deploy user, a firewall, build tools, Node under nvm, Postgres with its own staging database, PM2 configured to survive a reboot, Nginx as a reverse proxy, Let’s Encrypt for TLS, a read-only deploy key so the server can pull a private repository, and a separate CI key for GitHub Actions.

The point of writing it as a runbook rather than doing it once is that an environment only one person can rebuild is a single point of failure wearing a hat. It is written to be followed by someone else.

Two steps in it are the kind of thing you only get from having done it wrong first. HTTPS redirection is left off until certificates actually exist, because turning it on beforehand gives you a redirect loop that locks you out of the validation request you need in order to get certificates. And the initial schema is brought up by enabling TypeORM’s auto-sync for exactly one boot and then turning it straight back off, so staging spends its life with the same manual migration discipline as production instead of quietly diverging from it.

Sandboxed by construction

Staging talks to seven external services and not one of them can move real money or send a real message. Stripe runs in test mode with its own webhook signing secret. PayPal uses sandbox credentials and a sandbox buyer. Twilio runs on a trial account against verified numbers. The mobile money gateway is mocked outright. Identity verification, mail, and maps all point at test endpoints.

This is the part I would insist on anywhere. A staging environment that shares one production credential is not a staging environment, it is production with worse monitoring, and you find out which one you had the first time a test booking charges somebody.

Deploying over SSH, and then checking it worked

Pushing to develop deploys staging. Pushing to main deploys production. Both run over SSH from GitHub Actions with the script set to abort on the first failure.

The deploy does a hard reset onto the remote branch rather than a pull. That is the deliberate half of the fix for the original problem: whatever is on the server’s disk loses to what is in the branch, every time, so drift cannot survive a deploy. It also sources nvm explicitly, because a non-interactive SSH session does not run the login shell and would otherwise not have node, npm or pm2 on its path at all.

The part I care about is the end of the staging job. After the process manager restarts, the script waits, reads the process list back as JSON, and fails the deploy unless the app is genuinely online rather than crash-looping. Then it makes an actual HTTP request against the running app, with retries, and fails if nothing answers.

Those are two different claims. A process manager reporting a process as up tells you something was spawned. It does not tell you the app booted, connected to its database, or can serve a request. A deploy that goes green while the site is down is worse than a deploy that fails, because it teaches you to trust the green.

There is a small war story inside that check. With abort-on-failure enabled, the SSH action injects an exit-code test after every newline in the script, so a multi-line inline Node program gets shell statements spliced into the middle of its source. The health check therefore has to stay on a single line, and that constraint is written directly above it, because it looks like something worth reformatting and is not.

Migrations that can safely run twice

Production has TypeORM’s auto-sync switched off on purpose, so entity changes do not silently alter a live schema, and schema changes are hand-written SQL instead.

Those migrations are written to be idempotent, using existence checks on tables and columns and conflict handling on seeded rows, so applying one twice is a no-op rather than an error. The runner reads the same environment the application does, so it cannot connect to a different database than the app it is migrating, and after applying it verifies rather than assumes: it queries the schema catalogue to confirm the new column is really there and counts the rows it seeded, then exits non-zero if anything failed.

That verification step exists because “the migration ran” and “the migration did what it was supposed to” are, once again, two different claims.

What production still does not have

The asymmetry in this pipeline is the wrong way round and I would rather say so than let someone find it.

Staging runs migrations, checks the process is online, and smoke tests over HTTP. Production does none of those. Its job ends at restarting the process manager, and it restarts every process on the box rather than the one it just deployed. The environment with real customers and real money has strictly less verification than the one built for testing.

That is next. The health check and the migration step are already written and proven on staging, so promoting them is mostly a matter of doing it carefully against an environment where a mistake is expensive. Alongside it, the repository carries a known-issues register with root causes written up, because a defect that is documented gets fixed and a defect in someone’s memory gets rediscovered by a customer.