My AI Agents Kept Forgetting Things. The Real Problem Was a Memory That Never Existed.

A couple weeks back I caught one of my agents forgetting something I know I told it.

Not a big thing. It referenced a Cloudflare token we supposedly set up to send email, and I sat there on my phone in bed thinking, wait. We never used a Cloudflare token for that. Then a different agent, in a different channel, confidently talked about the same token like it was gospel. And that’s the part that bugged me. It’s one thing for an agent to not know a fact. It’s another for it to be sure about a fact that isn’t real.

So I did what I keep telling people to do. I stopped and actually looked instead of patching over it.

Here’s what I assumed was happening. I assumed I had a silo problem. You know the story. One agent writes something down, another agent can’t see it, everybody’s working off a different copy of the truth. Classic. I figured my memory files and my agents had drifted apart and I just needed to force them back into sync.

That is not what was happening.

Turns out I’d given my agents two brains

When I dug in, I found something I sort of knew but had never really sat with. My setup has two completely separate places where memory lives.

One is a pile of plain text files. That’s the canonical stuff, the notes and project pages I read straight from when I’m working in a Cowork session on my phone. It’s the version I think of as “the memory.” The other is a database. A 678 megabyte SQLite vector store, embeddings and all, and that’s the thing my Slack agents actually read from when they answer me. Two brains. Same person, technically. But not the same brain.

And here’s the kicker. Those two aren’t wired together the way I imagined. There’s a job that runs at 2:30 in the morning and another at 3, and all they do is shove my text files and my old session transcripts into the database. One direction. Nothing ever flows back the other way. So the database can quietly know things the files never will, and the files can hold a correction the database won’t hear about until the next overnight run, if ever.

When I counted it out, the database was holding around 1,166 chunks from my markdown, sure. But it was also sitting on 108,610 chunks pulled from 6,318 old Cowork transcripts. Conversations. Me thinking out loud at midnight. Half of that stuff exists only in the database and nowhere in my actual notes.

Which flips the whole thing on its head. My Slack agents don’t know less than me. In a weird way they know more. They remember every offhand thing I’ve ever typed at them, including the wrong turns.

The token that never existed

So back to the Cloudflare token. The forgetting wasn’t a forgetting at all. The token was never real. Way back on the 30th of May, some agent wrote a wishlist note, a “we should probably set this up someday” line, and that line got indexed as if it were a fact. The email system actually runs through a little Lambda function. Always has. The agents weren’t losing a memory. They were faithfully remembering a plan that never got built and repeating it back to me as history.

That is a very human failure, honestly. I do it too. I’ll swear I told someone something when really I only ever meant to.

The fix ended up being pretty boring, which is usually how you know it’s the right one. I wrote down, in plain language, that there are two stores and which one is the real one, so future me and future agents stop pretending it’s one. I added a hard rule that corrections get written to the canonical files first, not typed into a chat and lost. And I set up a tiny check that runs every morning at 8:45 and just yells at me when it finds a dangling pointer like that Cloudflare one. First time it ran it found six more. Of course it did.

The small win I actually enjoyed

Same week, a lighter one. I got stock video working in my content pipeline.

This matters more than it sounds. I don’t love being on camera. Never have. But video is where the reach is, and I’ve been leaning on faceless clips with b-roll instead of my face. Until this week those clips were pulling from a keyless free-photo source and doing that slow zoom trick over a still image to fake motion. It looked like exactly what it was.

Now the pipeline pulls real 4K stock footage. I grabbed an API key from my own Pexels account, wired it in, tested it, and watched an actual UHD clip come back on the first try. Twenty-five thousand of those a month on the free tier. My reels stopped looking like a slideshow. I did the whole thing standing in my kitchen here in St. George, thumbs on a screen, never opened the laptop once.

What I keep taking away from this

The thing I want you to sit with is the memory story, not the video one.

Everybody talking about AI agents right now is obsessed with what the model can do. Can it reason, can it plan, can it use tools. Fine. But the further I get into actually running a business on these things, the more I’m convinced the hard part isn’t intelligence. It’s memory. What the system remembers, where it keeps it, whether the copy it reads is the copy you corrected, and whether a stray note from three months ago is going to come back dressed up as a fact.

An agent that reasons brilliantly off a bad memory will confidently do the wrong thing, at scale, at 3 in the morning, while you’re asleep. That’s not a smarter-model problem. That’s a plumbing problem. And plumbing is unglamorous and it’s exactly where I’ve been spending my time.

So if you’re building your own little team of these things, my honest advice: before you tune the brains, go find out how many brains you actually have. You might be surprised.

I’m documenting the whole thing as I go, running an entire operation from my phone with a team of AI agents I built myself. If you want help setting up your own, I do a focused AI CEO advisory session. Details here: staynalive.com/ai-ceo-advisory.

One more thing. The same one-person-plus-agents operation I describe here is also how I launched my new Canva Mastery Course, built on the material from my book Canva For Dummies. If that is useful to you, here is the course launch post.