RAR staging and deploy pipeline

When I took this over, parts of the running system had been edited directly on the production server, outside version control. The consequence that matters is that you cannot deploy, because nobody knows what deploying would overwrite.

So the work went in this order. Reconcile what was on the server back into the repository. Make the deployed branch the source of truth rather than the disk. Then build somewhere to test that is not production.

A runbook that rebuilds the box

Staging is a clean Contabo Ubuntu VPS built up in sixteen numbered steps: a dedicated deploy user, a firewall, build tools, Node under nvm, Postgres with its own staging database, PM2 configured to survive a reboot, Nginx as a reverse proxy, Let’s Encrypt for TLS, a read-only deploy key so the server can pull a private repository, and a separate CI key for GitHub Actions.

It is written as a runbook rather than done once because an environment only one person can rebuild is a single point of failure. It is written to be followed by someone else.

Two steps in it come from getting them wrong first. HTTPS redirection is left off until certificates exist, because turning it on beforehand gives a redirect loop that blocks the validation request needed to get the certificates. And the initial schema is brought up by enabling TypeORM’s auto-sync for exactly one boot and then turning it back off, so staging runs with the same manual migration discipline as production instead of diverging from it.

Sandboxed by construction

Staging talks to seven external services and none of them can move real money or send a real message. Stripe runs in test mode with its own webhook signing secret. PayPal uses sandbox credentials and a sandbox buyer. Twilio runs on a trial account against verified numbers. The mobile money gateway is mocked outright. Identity verification, mail, and maps all point at test endpoints.

A staging environment that shares one production credential is production with worse monitoring, and you find out which one you had the first time a test booking charges somebody.

Deploying over SSH, and then checking it worked

Pushing to develop deploys staging. Pushing to main deploys production. Both run over SSH from GitHub Actions with the script set to abort on the first failure.

The deploy does a hard reset onto the remote branch rather than a pull. That is the deliberate half of the fix for the original problem: whatever is on the server’s disk loses to what is in the branch, every time, so drift cannot survive a deploy. It also sources nvm explicitly, because a non-interactive SSH session does not run the login shell and would otherwise not have node, npm or pm2 on its path.

At the end of the staging job, after the process manager restarts, the script waits, reads the process list back as JSON, and fails the deploy unless the app is online rather than crash-looping. Then it makes an actual HTTP request against the running app, with retries, and fails if nothing answers.

Those are two different claims. A process manager reporting a process as up says something was spawned. It does not say the app booted, connected to its database, or can serve a request. A deploy that goes green while the site is down teaches you to trust the green.

There is one awkward detail inside that check. With abort-on-failure enabled, the SSH action injects an exit-code test after every newline in the script, so a multi-line inline Node program gets shell statements spliced into the middle of its source. The health check has to stay on a single line, and that constraint is written directly above it, because it looks like something worth reformatting.

Migrations that can safely run twice

Production has TypeORM’s auto-sync switched off on purpose, so entity changes do not silently alter a live schema, and schema changes are hand-written SQL instead.

Those migrations are idempotent, using existence checks on tables and columns and conflict handling on seeded rows, so applying one twice is a no-op rather than an error. The runner reads the same environment the application does, so it cannot connect to a different database than the app it is migrating. After applying, it queries the schema catalogue to confirm the new column is there and counts the rows it seeded, then exits non-zero if anything failed.

That verification exists because “the migration ran” and “the migration did what it was supposed to” are two different claims.

What production still does not have

The asymmetry in this pipeline is the wrong way round.

Staging runs migrations, checks the process is online, and smoke tests over HTTP. Production does none of those. Its job ends at restarting the process manager, and it restarts every process on the box rather than the one it just deployed. The environment with real customers and real money has less verification than the one built for testing.

That is next. The health check and the migration step are already written and proven on staging, so promoting them is mostly a matter of doing it carefully against an environment where a mistake is expensive. Alongside it, the repository carries a known-issues register with root causes written up, so defects are findable rather than remembered.