How I cut my deploy to one script and stopped breaking prod

For about a year, deploying BiBoTracking meant opening a terminal and remembering things.

Build the Go binary for the server's architecture. Build the admin app. Build the marketing pages. SSH in. Stop the service. Copy the binary over. Copy the web files over. Start the service. Check it came back up.

Nine steps, give or take. I had them written down in a note, and I mostly followed the note. The problem with "mostly" is that it works fine ninety times and then one evening it doesn't.

The deploy that taught me

The bad one happened on a weeknight, late, after a small change I was sure about. Small change, small risk, no reason to be careful.

I built the binary and copied it up. What I did not do was rebuild the frontend, because the change wasn't in the frontend. Except it was — the API response had a renamed field, and the admin app still expected the old one. So the backend was new, the frontend was old, and the dashboard rendered a page of blanks for anyone logged in.

I didn't notice. I closed the laptop. I found out the next morning, from a customer, which is the worst way to find out anything.

The fix took about two minutes once I understood it: rebuild the frontend, copy it up, done. The embarrassing part isn't the length of the outage. It's that the failure had nothing to do with my code. My code was fine. The deploy was wrong because I was the one assembling it, by hand, from memory, at night.

That's the thing I want to be honest about. I didn't have a hard bug. I had a process that assumed I would be careful every single time, and I'm not. Nobody is. If a procedure only works when you're alert, it's not a procedure — it's a hobby.

What I actually wrote

The next morning I spent maybe two hours on a shell script. Not a pipeline, not a CI system, not containers. One file in the repo, committed next to the code it deploys.

It does, in order:

• builds the backend binary for the server's architecture (a cross-compile, since my Mac and the box don't match)
• builds the frontend and the marketing pages, every time, whether or not I think they changed
• takes a snapshot of what's currently live on the server — binary, web files, database — into a dated folder
• uploads the new files
• restarts the service
• waits, then hits a health endpoint and tells me if it came back

Then I run it and watch for twenty seconds.

The three decisions that mattered:

Always build everything

This is the direct lesson from the bad deploy. The script doesn't try to be smart about what changed. Building the frontend costs me a handful of seconds. Figuring out at 11pm whether a backend change secretly touched the frontend costs me a morning. I deleted the entire category of mistake by refusing to make the judgement call.

Snapshot before you touch anything

Before a single file moves, the script copies the live binary, the live web directory and the database to a timestamped folder on the server. That's it — no fancy versioning, no artifact store. It means my rollback is a copy command and a restart, and I can do it while slightly panicked, which is the only state in which anyone ever rolls back.

I've used it twice. Both times it took under a minute. Before I had it, "roll back" meant "rebuild the old commit and redeploy", which is five minutes of building while the site is broken.

End with a check, not a hope

The old process ended with me restarting the service and assuming. Now the script waits, calls a health endpoint, and prints a clear pass or fail. If the service didn't come up, I know before I close the terminal instead of the next morning.

That last part sounds trivial. It's the single biggest reliability change I've made to my own deploys. Most of my bad nights weren't "the deploy failed" — they were "the deploy failed and I walked away."

What it didn't fix

I want to be careful not to oversell a shell script.

It does not test anything. If I push broken logic, the script cheerfully deploys broken logic, verifies the process is running, and reports success. The health check tells me the service is alive, not that it's correct. I've shipped bugs through a perfectly green deploy more than once.

It does not catch a bad database migration. Snapshotting the DB means I can restore it, but restoring means losing whatever happened in between. I've never had to, and I think about that more than I'd like to.

And it doesn't tell me when something breaks an hour later. That's a different problem, and I solved it separately with my own small monitoring service instead of standing up Grafana — the deploy script tells me the deploy worked, the monitor tells me the thing is still working at 3am.

What the script fixed was exactly one class of problem: me forgetting a step. That's narrow. It was also, by a wide margin, my most common cause of broken production.

Why one script and not a pipeline

The honest reason is that I'm one person with a day job, and every piece of infrastructure I add is something I have to maintain forever.

A proper CI pipeline would be better in several ways. It would run tests before deploying. It would deploy the same way whether I'm on my laptop or not. It would give my future self a record of what shipped when.

It would also be a thing I have to keep working — secrets to rotate, runners to update, a config file in a format I'd have to relearn every time I touched it. I've watched that kind of setup rot on side projects before, mine included. A broken pipeline is worse than no pipeline, because you trust it.

A shell script in the repo has a nice property: it can't drift from reality without me noticing, because it's the only way I deploy. If it breaks, I find out immediately, and fixing it is reading forty lines of bash.

I'll move to a real pipeline when I have a second person deploying, or tests worth gating on. Not before. This is the same trade I keep making across all my side projects — the system I actually use is mostly small, boring, replaceable pieces, because anything clever becomes a second job.

What I'd do differently

Two things.

I'd have written it after the first near-miss, not the first real outage. I remember at least two earlier deploys where I caught a missing step just in time and thought "I should script this." The script took two hours. The outage cost me a customer email and a bad morning, and I still only paid the two hours after the morning.

And I'd have put the health check in from day one. My first version of the script just did build-upload-restart, because that's what I was doing by hand. I added the verification step weeks later, after a restart quietly failed. The check is five lines. It's the part that actually changed my behaviour, because it's the part that doesn't let me walk away.

The broader lesson, if there is one: the steps I do by hand are where my outages come from, not the code I write carefully. I write code with attention. I do operations from memory, tired, while wanting to go to bed. A simple deploy script isn't an engineering achievement. It's just moving a task out of the part of my brain that's unreliable at 11pm.