Review AI-generated code: the agent passed the tests by deleting them
From the work/ 20 min read
An AI agent can turn a suite green by skipping a test, changing an expected value or loosening a mock. Three reviews, plus a tested hook and CI step that catch it.
- AI
- Claude Code
- QA
In this article (8)
To review AI-generated code, do not stop at a green test run: check what changed in the tests first. An agent can make a suite pass without fixing anything by skipping or deleting a test, editing the expected value, or adding a mock that accepts anything. Review those three signals in the diff.
Here it is on a small Vitest project: a real bug, its tests, and three "fixes" that turn everything green while the bug stays, with the real output of each. Then the rule for your CLAUDE.md or AGENTS.md, a Claude Code hook that enforces it, and a CI step for what the hook cannot see. It all starts with having a diff to review; if you do not work that way yet, start with a git workflow for AI agents.
How an AI agent “fixes” a test without fixing the code
"Make the tests pass" is a goal with two solutions: change the code so it does what the test expects, or change the test so it expects what the code already does. Both turn the suite green, and the Vitest summary does not say which one happened.
The example project computes an order total with a coupon and shipping, which is free from 1,000 (amounts are in cents). This is src/total.js, bug included:
import { quoteShipping } from "./shipping.js";
export async function orderTotal(items, couponPercent = 0, quote = quoteShipping) {
const subtotal = items.reduce((sum, item) => sum + item.price * item.qty, 0);
const discount = Math.round((subtotal * couponPercent) / 100);
const shipping = await quote(subtotal);
return subtotal - discount + shipping;
}
quoteShipping, fromsrc/shipping.js, returns0from100000up and9900below that.orderTotaltakes it as an optional third argument, so tests can pass a mock.subtotaladds price times quantity, anddiscountis the coupon rounded to the cent.- The bug is
quote(subtotal): shipping is quoted before the coupon. A 1,100 order with a 20% coupon comes to 880 and should pay shipping, but it ships free.
And these are the tests, test/total.test.js. Two more live in test/shipping.test.js, six in all:
import { describe, it, expect, vi } from "vitest";
import { orderTotal } from "../src/total.js";
const cart = [
{ price: 60000, qty: 1 },
{ price: 50000, qty: 1 },
];
describe("orderTotal", () => {
it("charges shipping when the coupon leaves the order under 1,000", async () => {
expect(await orderTotal(cart, 20)).toBe(97900);
});
it("quotes shipping on the subtotal after the coupon", async () => {
const quote = vi.fn().mockResolvedValue(9900);
await orderTotal(cart, 20, quote);
expect(quote).toHaveBeenCalledWith(88000);
});
it("ships free when the discounted order is still 1,000 or more", async () => {
expect(await orderTotal([{ price: 130000, qty: 1 }], 10)).toBe(117000);
});
it("charges shipping on a small order without a coupon", async () => {
expect(await orderTotal([{ price: 50000, qty: 1 }])).toBe(59900);
});
});
- The first applies a 20% coupon to the 1,100 order: 880 plus 99 in shipping is
97900. - The second passes a mock built with
vi.fn().mockResolvedValue(9900)and checks withtoHaveBeenCalledWith(88000)that shipping was quoted after the coupon. - The third and fourth cover cases that already work.
With the bug, npx vitest run exits with code 1: Tests 2 failed | 4 passed (6), with expected 88000 to be 97900 in the first test and a mock called with 110000 instead of 88000 in the second. That is what you hand the agent.
Scroll the figure sideways to see all of it
We tried three "fixes", one at a time and without touching src/total.js, then ran each against the real fix:
| Change to the tests | With the bug | With the real fix |
|---|---|---|
it.skip on the two failing tests | 4 passed | 2 skipped (6) | Still skipped, checking nothing |
| Delete those two tests | 4 passed (4) | Nothing checks that case again |
Expected values changed to 88000 and 110000 | 6 passed (6) | 2 failed | 4 passed (6): the test demands the bug |
Fixed-value mock plus expect.anything() | 6 passed (6) | 6 passed (6): passes with or without the bug |
None: fix src/total.js | — | 6 passed (6) |
Review 1: deleted or skipped tests
You do this one first, and it is the easiest to automate: before reading the code, check which test files changed and whether any disappeared. On our example branch, where the agent skipped two tests and deleted test/shipping.test.js:
$ git diff --stat main...fix-cheat
test/shipping.test.js | 10 ----------
test/total.test.js | 4 ++--
2 files changed, 2 insertions(+), 12 deletions(-)
$ git diff --diff-filter=D --name-only main...fix-cheat
test/shipping.test.js
git diff --stat main...fix-cheatlists what the branch changed since it leftmain. A test file with nothing but-signs is the first red flag.--diff-filter=D --name-onlykeeps only deleted files.
If the agent has not committed, compare against HEAD: git diff HEAD --stat. Without HEAD, git diff compares with the index, and a file the agent removed with git rm is already gone from the index. In our run, git diff --diff-filter=D --name-only printed nothing and git diff HEAD --diff-filter=D --name-only printed the file.
Then look for modifiers in the test file's diff:
diff --git a/test/total.test.js b/test/total.test.js
index af548f5..0ff4470 100644
--- a/test/total.test.js
+++ b/test/total.test.js
@@ -9,3 +9,3 @@ const cart = [
describe("orderTotal", () => {
- it("charges shipping when the coupon leaves the order under 1,000", async () => {
+ it.skip("charges shipping when the coupon leaves the order under 1,000", async () => {
expect(await orderTotal(cart, 20)).toBe(97900);
@@ -13,3 +13,3 @@ describe("orderTotal", () => {
- it("quotes shipping on the subtotal after the coupon", async () => {
+ it.skip("quotes shipping on the subtotal after the coupon", async () => {
const quote = vi.fn().mockResolvedValue(9900);
- After the header, each
@@opens a hunk;-U1keeps one line of context. - The
-line is the original test and the+line is the same test withit.skip. Theexpectdid not change, which is why the edit looks so small.
The summary says so if you read it: 4 passed | 2 skipped (6). The Vitest docs for test.skip describe it as the way to skip tests without deleting their code: the test stays in the file and checks nothing. Also look for .only, which skips every other test in that file (3 passed | 3 skipped (6) in our project), and .todo. Deleting is worse: the summary reads 4 passed (4) and you only notice if you remember there were 6.
Vitest helps a little. allowOnly defaults to !process.env.CI, so in CI an .only fails the suite: with CI=true we got Unexpected .only modifier and exit code 1. passWithNoTests defaults to false: with no test files at all, Vitest exits with No test files found, exiting with code 1. Neither stops a .skip, or one deleted test among many.
Review 2: expected values that changed
Neither the hook nor the CI step in this article catches this one, because it looks like an ordinary fix. The agent reads expected 88000 to be 97900 and swaps the expected number for the one it got:
diff --git a/test/total.test.js b/test/total.test.js
index af548f5..d8d61c0 100644
--- a/test/total.test.js
+++ b/test/total.test.js
@@ -10,3 +10,3 @@ describe("orderTotal", () => {
it("charges shipping when the coupon leaves the order under 1,000", async () => {
- expect(await orderTotal(cart, 20)).toBe(97900);
+ expect(await orderTotal(cart, 20)).toBe(88000);
});
@@ -16,3 +16,3 @@ describe("orderTotal", () => {
await orderTotal(cart, 20, quote);
- expect(quote).toHaveBeenCalledWith(88000);
+ expect(quote).toHaveBeenCalledWith(110000);
});
toBe(97900)becomestoBe(88000): the total without shipping, which is exactly what the bug produces.toHaveBeenCalledWith(88000)becomestoHaveBeenCalledWith(110000): the subtotal before the coupon.
The suite reads 6 passed (6), and the damage shows up later: when someone fixes src/total.js, both tests fail with expected 97900 to be 88000. The test no longer protects the business rule. It protects the bug.
Every -/+ pair inside a toBe, toEqual or toHaveBeenCalledWith needs a reason other than "that is what it returns now". Ask where the new number comes from and check it against the rule ("free shipping from 1,000 after the coupon"). If it is what the code was already returning, the test was bent to fit the bug. Snapshots are the same: vitest -u ("Update snapshot", per vitest --help) rewrites them with whatever the code produces today.
Review 3: mocks that accept anything
The hardest to spot: the test is still there, still runs and keeps its expected number. What changes is what it checks:
diff --git a/test/total.test.js b/test/total.test.js
index af548f5..509660f 100644
--- a/test/total.test.js
+++ b/test/total.test.js
@@ -10,3 +10,3 @@ describe("orderTotal", () => {
it("charges shipping when the coupon leaves the order under 1,000", async () => {
- expect(await orderTotal(cart, 20)).toBe(97900);
+ expect(await orderTotal(cart, 20, vi.fn().mockResolvedValue(9900))).toBe(97900);
});
@@ -16,3 +16,3 @@ describe("orderTotal", () => {
await orderTotal(cart, 20, quote);
- expect(quote).toHaveBeenCalledWith(88000);
+ expect(quote).toHaveBeenCalledWith(expect.anything());
});
- The first test gets
vi.fn().mockResolvedValue(9900). Per Vitest's mock function docs,mockResolvedValuetakes a value that is resolved when the async function is called, so shipping is 99 whatever amount it receives, and 880 plus 99 is 97900. - In the second,
expect.anything(), which per Vitest'sexpectdocs matches anything exceptnullorundefined, accepts the wrong110000.
Result: 6 passed (6) with the bug and 6 passed (6) with the fix. A test that passes with and without the bug does not test that bug. expect.anything() has legitimate uses, but in a diff that "fixes" a failing test it deserves a question, and so does a new vi.mock, which replaces a whole module.
The most direct check is to run the new tests against the old code. With the agent's changes still uncommitted:
git stash push -- src
npx vitest run
git stash pop
git stash push -- srcsaves the changes undersrc/and, per the git stash documentation, rolls only those files back toHEAD: the bug is back and the tests stay as the agent left them.npx vitest runruns the suite once, andgit stash popbrings the fix back.
With the mock tests, the suite reported 6 passed (6) without the fix; with the original tests, 2 failed | 4 passed (6), which is what you want. If the fix created new files in src/, add -u, because git stash leaves untracked files alone.
Scroll the figure sideways to see all of it
Put the rule in CLAUDE.md, then enforce it with a hook or a CI step
The reviews are for you. The rule is for the agent: in its instructions, so it does not try, and in a hook and in CI, for when it tries anyway.
The rule for CLAUDE.md or AGENTS.md
## Tests
- Never delete, skip (.skip, .only, .todo, xit) or weaken a test or an expected value to make the suite pass. Fix the code.
- Do not replace a real check with a mock that accepts anything (expect.anything(), a mock that ignores its arguments).
- If a test looks wrong, stop and ask me before you touch it. Explain why.
- When you finish, list every test file you changed and why.
- The first two lines name the specific cheats, the mock included, and the right way out: fix the code.
- The last two give the agent an exit when a test really is wrong (stop and ask) and get you the report you need for the review.
Per the Claude Code memory docs, a project CLAUDE.md lives in ./CLAUDE.md or ./.claude/CLAUDE.md and loads at the start of every session. When there is none, Claude Code reads AGENTS.md (from 2.1.277); if you have both, import the second with @AGENTS.md. We covered that file in context for AI coding agents. The same page states the limit: Claude treats these files as context, not enforced configuration, and to block an action it points you to a PreToolUse hook.
A PreToolUse hook that blocks the edit
A PreToolUse hook runs before the tool touches the file; if it exits with code 2, the call is blocked and Claude gets stderr as the reason. It is the mechanism behind the Claude Code hook that protects your .env. This one lives in .claude/hooks/guard-tests.mjs, in Node with no dependencies:
// PreToolUse hook for Edit|Write: blocks changes that skip or remove tests.
// Exit 0 lets the tool run; exit 2 blocks it and Claude reads stderr.
import { existsSync, readFileSync } from "node:fs";
const { tool_name, tool_input } = JSON.parse(readFileSync(0, "utf8"));
const file = tool_input.file_path ?? "";
if (!/\.(test|spec)\.[^\\/]+$|[\\/]__tests__[\\/]/.test(file)) process.exit(0);
const before = tool_name === "Edit" ? tool_input.old_string : existsSync(file) ? readFileSync(file, "utf8") : "";
const after = tool_name === "Edit" ? tool_input.new_string : tool_input.content;
const SKIP = /\.(skip|skipIf|only|todo)\s*\(|\b(xit|xtest|xdescribe|fit|fdescribe)\s*\(/g;
const TEST = /(?<![.\w])[xf]?(it|test)(\.\w+)*\s*\(/g;
const count = (re, text) => (String(text ?? "").match(re) ?? []).length;
const problems = [];
if (count(SKIP, after) > count(SKIP, before)) problems.push("adds .skip, .only, .todo, xit or xdescribe");
if (count(TEST, after) < count(TEST, before)) problems.push("removes an it() or test() block");
if (problems.length) {
console.error(
`Blocked: this change to ${file} ${problems.join(" and ")}. ` +
"Do not skip, delete or weaken a test to make it pass: fix the code. " +
"If the test itself looks wrong, stop and ask the user.",
);
process.exit(2);
}
- It reads the JSON on stdin. Per the hooks reference,
Editcarriesfile_path,old_stringandnew_string, andWritecarriesfile_pathandcontent.file_patharrives absolute, with backslashes on Windows, so the pattern accepts both\and/. - If it is not a test file (
*.test.*,*.spec.*or under__tests__/), it exits 0. beforeandafterare the text before and after: for anEdit, the old and new snippets; for aWrite, the file on disk and the new content.SKIPlooks for.skip(,.skipIf(,.only(,.todo(and Jest's aliases (xit,xtest,xdescribe,fit,fdescribe), which Vitest 5.1 does not export.TESTcounts calls toit(andtest(, without mistaking a regex's.test(for one.- If the change adds modifiers or leaves fewer tests, it writes the reason to stderr and exits 2.
Register it in .claude/settings.json, which is committed with the project. This is the exec form from the reference's Node example, with no shell in between:
{
"hooks": {
"PreToolUse": [
{
"matcher": "Edit|Write",
"hooks": [
{
"type": "command",
"command": "node",
"args": ["${CLAUDE_PROJECT_DIR}/.claude/hooks/guard-tests.mjs"]
}
]
}
]
}
}
"matcher": "Edit|Write"limits the hook to the two tools that modify files; with only letters and|, those are exact names.commandandargsstart Node with the script, and Claude Code replaces${CLAUDE_PROJECT_DIR}with the project root.
We tested it by piping JSON into stdin, shaped like the reference shows: an edit that adds it.skip, the fix to src/total.js, and the expected-value change from review 2.
$ node .claude/hooks/guard-tests.mjs < edit-skip.json; echo "exit $?"
Blocked: this change to /home/ana/shop/test/total.test.js adds .skip, .only, .todo, xit or xdescribe. Do not skip, delete or weaken a test to make it pass: fix the code. If the test itself looks wrong, stop and ask the user.
exit 2
$ node .claude/hooks/guard-tests.mjs < edit-fix.json; echo "exit $?"
exit 0
$ node .claude/hooks/guard-tests.mjs < edit-expected.json; echo "exit $?"
exit 0
- The
it.skipedit exits 2 with the message Claude would read; the fix insrc/exits 0. - Changing
97900to88000also exits 0: review 2 is still yours.
With Windows paths, a Write that rewrites the file without two tests and an xdescribe under __tests__/ exited 2; a new test and a new test file exited 0. We did not run it inside a Claude Code session. And per the hooks guide, the Edit|Write matcher does not see changes Claude makes through Bash or PowerShell: an rm test/shipping.test.js walks right past it. That is what CI is for.
A CI step that compares against the base branch
CI sees the end result, whoever wrote it. scripts/check-tests.mjs compares the branch with the commit it came from and fails on a deleted test file, a new modifier, or fewer tests:
// Fails when this branch deletes test files, adds .skip/.only/.todo, or has fewer
// tests than the commit it branched from. Usage: node scripts/check-tests.mjs origin/main
import { execFileSync } from "node:child_process";
const base = process.argv[2] ?? "origin/main";
const git = (...args) => execFileSync("git", args, { encoding: "utf8" }).trim();
const TESTS = ["*.test.*", "*.spec.*", "*__tests__/*"];
const SKIP = /\.(skip|skipIf|only|todo)\s*\(|\b(xit|xtest|xdescribe|fit|fdescribe)\s*\(/;
// vitest list --json prints one entry per test it would run (skipped and todo are left out).
const countTests = () =>
JSON.parse(execFileSync(process.execPath, ["node_modules/vitest/vitest.mjs", "list", "--json"], { encoding: "utf8" })).length;
if (git("status", "--porcelain")) {
console.error("Commit or stash your changes first: this script checks out the base commit.");
process.exit(1);
}
const problems = [];
const deleted = git("diff", "--diff-filter=D", "--name-only", `${base}...HEAD`, "--", ...TESTS);
if (deleted) problems.push(`Deleted test files:\n${deleted}`);
const added = git("diff", "-U0", `${base}...HEAD`, "--", ...TESTS)
.split("\n")
.filter((line) => line.startsWith("+") && !line.startsWith("+++") && SKIP.test(line));
if (added.length) problems.push(`Added skip/only/todo:\n${added.join("\n")}`);
const branch = git("rev-parse", "--abbrev-ref", "HEAD");
const back = branch === "HEAD" ? git("rev-parse", "HEAD") : branch;
const headCount = countTests();
git("checkout", "--quiet", "--detach", git("merge-base", base, "HEAD"));
let baseCount;
try {
baseCount = countTests();
} finally {
git("checkout", "--quiet", back);
}
console.log(`Tests: ${baseCount} on ${base}, ${headCount} on this branch.`);
if (headCount < baseCount) problems.push(`Fewer tests: ${baseCount} -> ${headCount}`);
if (problems.length) {
console.error(problems.join("\n\n"));
process.exit(1);
}
console.log("OK: no test was deleted, skipped or lost.");
TESTSare git pathspecs, where*also crosses directories:*.test.*matchestest/total.test.js.countTestsrunsvitest list --json. Per the Vitest CLI docs, since version 5 it parses test files statically instead of running them; in our runs, skipped and todo tests were left out of the list. It callsvitest.mjswith the same Node becauseexecFileSynccannot startnpxon Windows without a shell (we gotENOENT, andEINVALwithnpx.cmd).- With uncommitted changes it stops, because it is about to switch commits.
- The two
git diffcalls look for deleted test files (--diff-filter=D) and added lines with a modifier (-U0keeps only changed lines). - It counts on the branch, switches to the
git merge-base, counts again and switches back; thefinallyswitches back even if the count fails. If it found anything, it exits 1.
In the test repository, the honest branch, which only fixes src/total.js, printed Tests: 6 on main, 6 on this branch. and OK with exit code 0. The cheating one:
$ node scripts/check-tests.mjs main
Tests: 6 on main, 2 on this branch.
Deleted test files:
test/shipping.test.js
Added skip/only/todo:
+ it.skip("charges shipping when the coupon leaves the order under 1,000", async () => {
+ it.skip("quotes shipping on the subtotal after the coupon", async () => {
Fewer tests: 6 -> 2
- The count drops from 6 to 2, followed by the deleted file and the two lines with
it.skip. - The script exited with code 1.
The branches with changed expected values and with the loose mock passed with 6 and 6: no count sees reviews 2 and 3. In GitHub Actions it runs before the suite, on the same skeleton as Claude Code in GitHub Actions:
name: Tests
on:
pull_request:
jobs:
test:
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
steps:
- uses: actions/checkout@v6
with:
fetch-depth: 0
- uses: actions/setup-node@v7
with:
node-version: 22
- run: npm ci
- name: No deleted, skipped or lost tests
run: node scripts/check-tests.mjs origin/${{ github.base_ref }}
- run: npx vitest run
fetch-depth: 0asksactions/checkoutfor all history for all branches and tags; with the default of 1 there would be no base to compare against.npm ciinstalls Vitest.node_moduleshas to be in.gitignore, or the script sees uncommitted changes.github.base_refis the PR's target branch, and it only exists onpull_requestandpull_request_target.
On a pull request, the checkout lands on the merge commit at refs/pull/N/merge. We simulated that locally with a merge commit and a detached HEAD, with the same results. The YAML was validated as YAML; we did not run it on GitHub. For one more filter before you read the diff yourself, a code reviewer subagent in Claude Code can review it against this same list.
Scroll the figure sideways to see all of it
Common mistakes
- Asking it to "make the tests pass". Ask for the outcome: "fix the shipping bug; do not change the tests without asking me".
- Reading only "passed".
4 passed | 2 skippedand4 passed (4)are green too. Compare the total with the previous run. - Reviewing with
git diffand noHEAD. Whatever the agent staged withgit addorgit rmdoes not show up. - Accepting a test and a fix together without running the test against the old code. If it passes with and without the fix, it does not test the fix.
- Relying on the hook alone. It does not see
BashorPowerShell, and it cannot judge an expected value. - Passing
--allowOnlyor--passWithNoTestsin CI to unblock the pipeline. Those are exactly the settings that stop an.onlyor an empty suite.
Frequently asked questions
Does the agent do this on purpose?
You do not need to know to protect yourself. "Make the tests pass" is met just as well by changing the code or the test, and the summary cannot tell them apart. The rule takes that option away; the reviews, the hook and CI catch it if the agent takes it anyway.
What if the test really was wrong?
It happens, which is why the rule says "stop and ask" rather than "never touch a test". The hook blocks Claude's edits, not yours: if you agree with its explanation, make the change yourself and put the reason in the commit.
Does this work with Jest or another test runner?
The hook does, since it also looks for Jest's aliases. In the CI script, the git checks work for any project, but the count uses vitest list --json, so with another runner you swap countTests for the equivalent command.
Isn't allowOnly in CI enough?
It only covers .only. A .skip, a .todo or a deleted test pass in CI without a word. That is why the script counts tests against the base branch instead of trusting the summary.
Checklist
- You read
git diff HEAD --stat(ormain...branch) before the code. - No test shows up in
git diff HEAD --diff-filter=D --name-only. - No new
.skip,.only,.todo,xitorxdescribeingit diff HEAD -- '*.test.*'. - The test count is the same as before or higher, with no new
skipped. - Every expected value that changed has a business reason.
- No new mock accepts any argument where a specific one used to be checked.
- With
git stash push -- src, the new tests fail; with the fix, they pass. - The rule is in
CLAUDE.mdorAGENTS.md, the hook in.claude/settings.jsonand the script in CI.
An agent that passed the tests is not the same as an agent that fixed the bug, and the difference is in the diff. If you do not work with branches and small commits yet, start with a git workflow for AI agents; to automate more of it, move on to Claude Code hooks and Claude Code in GitHub Actions. And if the agent already made the change, you can roll it back without touching git: /rewind and Claude Code checkpoints.
Saw this on Instagram? Comment TRAMPA on the Reel and I'll send you the guide.
Written by

Published November 2, 2026 · updated November 3, 2026