I Audited My AI Team: Every Failure Was Memory, Not Reasoning

A week after the month-one report I stopped the agents mid-week and read everything they had produced since July. Not the summaries — the actual worklists, the experiment ledgers, the git history, the scheduled-task registry.

The analysis was consistently good. Several of the agents had reasoned their way to conclusions I would not have reached on my own, and one of them corrected me twice in a single afternoon using its own logs. That is not the part that needs fixing.

Every real failure I found was the system not knowing something that had already happened. Not a reasoning failure. A state failure. And they compound quietly, because an agent that lacks a fact does not report uncertainty — it just produces confident output built on the last thing it read.

Five weeks of good advice about a closed decision

Every weekly worklist since July 27 opened its "needs a human" section with the same item: apply to the fresh dog food affiliate programs. The reasoning was airtight. One of my five sites, WoofTested, has about a third of its articles on fresh dog food delivery. Amazon barely sells that category, so those articles earn nothing. The brands themselves — The Farmer's Dog, Ollie, Spot & Tango — run programs paying $25 to $40 per lead. The site links to all of them as plain, unmonetized links. It is, on the face of it, free money sitting in a form.

I had applied in July. I was denied by all of them.

The agents had no way to know that, because I told a browser, not a repository. So for five weeks the highest-priority human item in my system was a decision that had already been made and closed. Worse, it was crowding out real items by sitting at the top of every list.

The denials also make complete sense in hindsight, which the agents could have worked out if they had thought about it from the program's side rather than mine. MeowTested is my strongest site and it was applying at roughly 200 sessions a month. WoofTested, the one that actually needs a program, runs 64 sessions a month and zero search clicks. Nobody approves that.

The fix is not cleverness, it is a written record: a policy file with an application register, a gate (no reapplying until a site sustains ~1,000 organic sessions a month for two months), and a six-month cooldown per denied program. That file is now referenced from four of the agent playbooks, and the register has an open TBC on it — I need to write down which programs I applied to, or the cooldown clock is guesswork.

The broader lesson is the one I keep relearning: an agent system needs a record of your decisions, not just your data. Mine had four months of Search Console history and no memory of a single thing I had personally tried and abandoned.

The safety mechanism that starved the two best sites

My content writer has a governor: before drafting for a site, count how many drafts are already waiting for my approval, and skip the site if there are three or more. Sensible. My approval throughput is the real bottleneck, and I did not want a pile of unreviewed work.

It counted unmerged git branches. And in August I changed the publishing flow — approved drafts now get re-applied onto a fresh branch to dodge a recurring merge conflict, and that branch gets merged. Which means the original draft branch stays permanently unmerged even though the article is live on the site.

So every article I published left behind a permanent decoy. Both of my best sites had hit exactly three. The writer had been quietly skipping them, and would have skipped them the next morning too, because of work it had already successfully finished.

Eleven of those decoys across the portfolio. All deleted, all with the article verified live first. The rule now counts published slugs rather than branch state, which is what it was always trying to measure.

A safety mechanism that measures a proxy will eventually measure the wrong thing. This one ate the thing it was protecting.

Silence is the default failure mode

My opportunity scout runs Friday evenings. Since late July it has produced an output file on two of five runs.

Nothing was broken. Nothing alerted. The runs fired, did something, and wrote nothing, and because the absence of a file is not an event, no part of the system noticed. I only found it by listing the output directory and counting.

Every playbook now ends with a required run receipt: name the files you wrote, or say NO OUTPUT and give the reason. "Nothing worth reporting" is a completely valid outcome. Silence is not.

I also added a daily health check — the only thing in the fleet I increased the frequency of, and I will come back to why. It verifies all five sites are live and indexable, builds every repo, checks for regressions in page metadata, confirms no affiliate link has lost its tag, and flags any site down more than 40% week over week. It posts to Discord only when something is wrong. A daily green message trains you to ignore the channel, and then the one red day gets ignored too.

The automation kept stopping to ask permission because of a typo

This was my biggest day-to-day frustration and the least interesting root cause. Scheduled runs kept halting on permission prompts, so "automation" meant sitting there clicking approve.

There was an allowlist. Ten of its eighteen rules were written with a space-star instead of the :* prefix matcher the tool actually uses:

Bash(git *)      →  never matched anything
Bash(git:*)      →  what it needed to be

So the file looked like a solved problem and was not. Every git command still prompted, every run, for weeks. Two further causes stacked on top: the allowlist lived in the project directory, which scheduled runs do not reliably start in, and the user-level list that does always apply was missing most of the ordinary shell — no grep, no awk, no sed, no head. Which is most of what any run actually does between the headline commands.

Fixed at the level that applies everywhere, with a deny list underneath it. One of those deny rules is doing something I like: it blocks calling the Netlify CLI directly, which my deploy documentation has forbidden in prose for a month because the CLI's active login is often the wrong account and can silently create a duplicate site. That rule used to be trusted. Now it is enforced.

I did decline one instruction. The config had a blanket "skip all permission checks" mode and I removed it, because it also disarms the deny rules, and these agents deploy to five production sites. The explicit allowlist gets the same hands-off result with the safety net still live.

Being wrong twice in an afternoon

Two corrections worth recording, because the series is only useful if the misses go in too.

I recommended internal linking as the main ranking lever. My own SEO agent had already tested it twice and graded both as nulls: three in-body links from the strongest page moved position by 0.28 places, and six exact-match anchors to an orphan page left it on one impression after 24 days. Its written conclusion was that at these site sizes internal links are not a lever. I had recommended building an agent around it before reading its log. The log was right.

I claimed a page had no meta description and Google was writing its own. It had one, through a template fallback I had not checked, and the existing text was better than my replacement. I reverted it.

Both were caught the same way: by checking what the system had already recorded instead of trusting a fresh inference. Which is the same failure as the affiliate programs, pointed at myself.

The economics I had backwards

The audit's most useful output had nothing to do with bugs. Across the five sites, I have been optimizing the one that pays worst.

MeowTested sells cat products with an average order around $19. It produces 93% of my affiliate clicks. VanTested sells van air conditioners and battery systems where the products run $900 and up; BrewTested sells espresso machines around $400. A click on either is worth roughly eleven times a MeowTested click — and between them they got six clicks last month, because they get almost none of the work.

BrewTested has five pages with more modelled monthly upside than MeowTested's twenty.

The honest caveat, which I have written into the model itself: those per-site conversion rates are guesses. Van and Brew have produced five affiliate clicks between them in thirty days, which measures nothing. The whole case rests on assumptions I have flagged as assumptions, and it will stay that way until one of those sites produces enough volume to measure. There is a scheduled agent whose job is now to say so every week rather than quietly present modelled dollars as forecasts.

The numbers

Since launch: 185 clicks, 32 orders, $617.82 in ordered revenue, $17.36 in earnings. Plus €1.51 and £0.52 internationally.

Against the last checkpoint on July 29 — 55 clicks and $6.69 — that is 130 clicks and $10.67 added in 26 days, roughly doubling the daily rate.

The number I did not expect is the conversion rate: 17.3% of clicks become orders. That is around double a typical Amazon Associates rate, and it reframes the problem entirely. The pages are working. The traffic they send is high-intent and it buys. There just is not enough of it, and what there is arrives on the cheapest site.

MeowTested also had its best week on record: impressions 719 → 1,854, search clicks 20 → 29, affiliate clicks 16 → 28.

And one thing that looked alarming and was not: VanTested's weekly impressions fell 209 → 172 → 25. Not a penalty, not a broken deploy — all five sites verified healthy. Its content is almost entirely van air conditioning, and August is when that category goes dormant until spring. Verified against demand research rather than assumed, which meant deferring a site I had been about to invest in, and rescheduling it for January so it ranks ahead of the May surge.

What changed

A chief-of-staff agent that allocates the week by expected revenue instead of traffic, and assigns each writing slot to either a new article or an upgrade of an existing money page. A consolidation agent for a problem I did not know I had — MeowTested has 37 articles covering four topics, with ten pages competing over litter boxes and four over grooming gloves. A daily health check. Run receipts everywhere. A metadata standard, after an audit of rendered HTML found 57% of my pages had titles too long to display fully in search results.

Sixteen pages of metadata fixed and shipped today across four sites, plus a page upgrade and a lever I built and deliberately did not use yet.

None of that is the interesting part. The interesting part is that a system can be good at analysis and still walk confidently into a wall, because it has no idea the wall was there last week. Most of what I built today is not intelligence. It is memory: written records of what was tried, what shipped, what failed, and what I already decided.

Next report in a month. Either way, as always.