My AI Agents Now Snitch on Each Other. It Was the Only Fix That Worked. - Stay N Alive

A phone screen showing AI agent status checks

Two weeks ago I found out one of my agents had been dead for days and none of the others said a word. I wrote about it. A bunch of you emailed me, mostly some version of the same question: okay, smart guy, so what did you actually do about it?

Here’s what I did about it. And no, it wasn’t pretty either.

The first fix I tried was the obvious one. A cron job. Every hour, ping each agent, log whether it answered. Clean, simple, the kind of thing I’d have built in an afternoon back when I still owned a laptop. Except I don’t run a laptop anymore. I run this whole thing from my phone, standing in my kitchen in St. George most mornings, and the “afternoon” version of that idea turned into three days of me fat-fingering YAML on a six-inch screen and swearing at indentation.

The cron job worked. Sort of. It told me an agent was down. It just told me by dumping a line into a log file that — and this is the embarrassing part — no agent was watching. I had built a smoke detector and put it in a room nobody goes into. Same bug. Different room.

The thing that finally worked was pettier than I expected

So I stopped trying to build a monitor. I made the agents responsible for each other instead.

Every agent now has to check in. Not to me — to the next agent in line. The content agent pings the Slack agent. The Slack agent pings the research one. The research one pings the deploy watcher. It’s a little chain of “you still there?” messages, and if any link goes quiet for more than 90 minutes, the agent waiting on it does one thing: it messages me directly. On my phone. By name. “Haven’t heard from the content agent since 2:14. Last thing it said was that it was pulling the newsletter draft.”

My agents snitch on each other now. That’s the actual architecture. I gave up on a wise central overseer and built a neighborhood of nosy neighbors, and the nosy neighbors caught things the overseer never would have.

Why does this work better? Because the check-in isn’t a separate system that can also die quietly. It’s baked into the work the agents are already doing. If they’re alive enough to do their job, they’re alive enough to say hi to the next one. And if they’re not, the silence itself is the alarm. No extra process to forget about. No dashboard I have to remember to open.

Where it broke, because of course it broke

First week, I got pinged eleven times in one afternoon. Turns out 90 minutes is way too tight for an agent that’s doing a long research pull. It wasn’t dead. It was just thinking. I’d built a system that panicked every time one of my agents took a lunch break.

Fixed that by letting each agent declare how long its current task should take. “I’m starting a big scrape, don’t expect me for two hours.” A little status flag. Now the timeout flexes with the work. Should’ve thought of that first. Didn’t.

Then there was the weekend it all went silent for a good reason — I’d paused the whole pipeline myself and forgot to tell the check-in chain. So the agents dutifully noticed everyone was gone and started blowing up my phone at 7am Saturday to report that my entire company was, technically, dead. Which, fair.

I added a “we’re on break” mode after that. You’d think running a business from your phone would make you better at remembering to mute things. It has not.

What this actually changed

The real shift wasn’t technical. It was that I stopped trusting any single point in the system to know the truth. Including me. Especially me.

When I ran everything myself, the status of the business lived in my head, and my head is a bad database. It forgets. It rounds up. It tells me things are fine because I want them to be fine. Handing the “is everything okay” question to a bunch of agents that literally cannot want anything — that took the wishful thinking out of it. The system reports what’s true whether I like it or not.

And I’ll be honest, there’s something a little strange about it. I built a thing whose entire job is to notice when the things I built stop working. A watcher for the watchers. Somewhere down there is probably a philosophy problem I’m not qualified to solve. But it’s caught two more silent failures since I set it up, both before I’d have noticed on my own, both while I was doing something completely unrelated. One while I was at my kid’s baseball game. My phone buzzed. “Deploy watcher hasn’t reported since the last push.” I fixed it from the bleachers between innings.

That’s the part that still gets me. Not that the agents do the work. That they now tell me when they can’t.

The takeaway, if you want one

If you’re building anything with AI agents — even one — the question isn’t “can it do the task.” That part’s getting easy. The question is “will you know when it stops.” Those are completely different problems, and almost nobody builds the second one until the first one has already failed silently on them at least once.

Build the part that notices. Make the noticing something the system does automatically, not something you have to remember to check. And don’t put the smoke detector in the room nobody goes into. I’ve made that mistake twice now. Learn from the guy running his company from a phone in Utah, one panicked Saturday morning ping at a time.

Postscript. The same phone-run operation behind these agents is also how I launched my Canva Mastery Course, drawn from my book Canva For Dummies. If that is helpful, here is the launch announcement.

From the author of Canva For Dummies: my Canva Mastery Course is 67 dollars, down from the 149 list price. One payment, fifteen modules. See what is inside. My full Canva toolkit is here.

Email
Chat with us
Email
Livechat

Leave a Reply

Your email address will not be published. Required fields are marked *