Number Six Was the Door
/A sixth build script failed silently after the hardware move, caught two days late by accident. Why silent failures never get louder, and the fix.

Yesterday I wrote a build-log about five pieces of this operation that died after the hardware move. Green lights, no errors, five dead things. I listed them out because the point of this feed is that you watch the build happen, cracks included.
That post did not publish.
The publish script runs twice a day, on its own, and has for months. Nothing crashed.
After the move, it started failing on the first step every single time, for the same reason as the other five: it reached for a tool that used to be in one place and is now in another. It never got as far as touching the site.
It just stopped, wrote four words into a log nobody reads, and waited to do it again six hours later.
So the count is not five. It is six. And number six is the door the other five were supposed to walk out of.
The part that actually stings
I found the first five by going and looking. I found this one by accident, chasing something else, two days after it started.
Two days is not a disaster. The site sat there serving the same posts it served on Tuesday, and nobody noticed, which tells you something about how much the world was waiting on my build-log.
But run the arithmetic forward. Nothing in that failure gets louder with time. On day thirty it looks exactly like it looked on day one, which is to say it looks like nothing.
The brief that went twenty-eight days without sending did not get twenty-eight times more obvious. It got twenty-eight times more expensive and stayed exactly as quiet.
That is the whole shape of this class of problem.
It does not degrade or escalate. It sits at a constant, perfectly survivable, perfectly invisible zero.
What it cost, honestly
Nothing yet. The safety catch held.
The script is built to stop dead the moment any step fails, so it quit before it could push anything, and the live site was never touched by a half-finished build.
That is the one piece of this I got right on purpose, years before it mattered, and it is the only reason today is an annoyance instead of a cleanup.
The fix is one line: teach the script where to find its own tools instead of trusting whatever handed it the job.
It is written, it is tested against the exact broken condition, and it is sitting on a branch waiting on me. I do not let the machine publish to the live site on its own authority at six in the morning.
The rule I am writing down
Strip the timestamps and a log is just a pile.
The file that would have told me the exact minute this started has no dates in it. I had to work backwards from when it was last written to guess which runs failed.
Every silent failure in this operation so far has had the same tell: something was writing output that nobody could date, so nobody could tell working from stopped without reading it by hand.
Six for six now. Different scripts, different weeks, same fingerprint.
I keep finding these because I keep looking. The question I do not have a good answer to is how many I have not gone looking for.
Comments are open to members. Sign up free →