Claude Code vs Codex: compare them on the same bug in your repo
From the work/ 18 min read
A method to compare Claude Code and Codex in your own repo: same base commit, same prompt and four metrics, repeated on three bugs before you decide.
- AI
- Claude Code
- Git
- QA
In this article (9)
To run a fair Claude Code vs Codex comparison, give both the same bug from your own repo: two branches cut from the same commit, one prompt saved in a file, and four metrics measured the same way for each: time, diff size, untouched tests and cost. Repeat it with three different bugs before you decide.
This is a method, not a benchmark: we publish no results and pick no winner. Below are the commands for both tools, checked against their official documentation on November 11, 2026, and two scripts we tested in a practice repo with hand-made commits. We did not run either agent.
Why compare on your own code
The comparisons that circulate use repos, languages and tasks that are not yours. What you care about is different: whether the agent understands your layout, follows your conventions, runs your suite and fixes the bugs you actually get. You can only measure that in your repo.
What you compare also changes often. Both CLIs ship new versions frequently and each can run different models, so you are really comparing tool plus model. Write down the model each one used (Claude Code sets it with --model, per its CLI reference; Codex with -m, per its own) and each CLI's version. With scripts, repeating the test when something changes means rerunning the same commands.
For a fair comparison, pin down three things:
- The same starting point. The same commit, the same dependencies and the same failing test.
- The same prompt. One text file that both receive, with no extra help for either.
- The same measurements. The same commands and the same clock for both, and no stepping in mid-run.
Scroll the figure sideways to see all of it
Set up two identical branches
Start with a bug you can reproduce with a test. Write the failing test, commit it on a base branch, and create two worktrees from it, one per agent. A git worktree is another folder with its own branch that shares the repo's history, so each agent edits its own copy without touching the other. In the example, a store charges shipping on a subtotal of exactly $999 MXN even though shipping is free from $999; the new test asserts assert.equal(shippingCost(999), 0).
git switch -c cmp/base-1
# add the test that reproduces the bug to test/shipping.test.js, then:
git add test/shipping.test.js
git commit -m "test: free shipping from exactly 999 (fails)"
npm test # must fail, and for this reason
mkdir -p ../cmp/runs
git worktree add ../cmp/claude -b cmp/claude-1 cmp/base-1
git worktree add ../cmp/codex -b cmp/codex-1 cmp/base-1
(cd ../cmp/claude && npm ci) && (cd ../cmp/codex && npm ci)
git switch -c cmp/base-1creates the base branch from where you are. Start with a clean working tree.- The commit stores the failing test: it is the spec for the fix and what the test metric checks later.
npm testmust fail on the base, and because of the bug. If it fails for another reason, no agent can pass that test.../cmp/runsholds each run's output outside the worktrees, so it never shows up in the diff.- Each
git worktree addcreates a folder and a new branch fromcmp/base-1: the same commit for both. npm ciinstalls the same dependencies in each folder, because a new worktree only has the tracked files. Use your project's install command.
Each tool has its own worktree option, but they do not start from the same place. According to Claude Code's worktree documentation, claude --worktree starts from the remote's default branch by default, not from your local HEAD. Codex's command reference describes codex exec --worktree as a managed worktree, and its worktrees page, written for the desktop app, says it starts from the HEAD of the selected branch and applies your uncommitted changes. To control the exact commit, create the worktrees yourself with git.
Git is also the undo button both tools share: each run lives on its own branch and you discard it with git worktree remove. Inside an interactive session, Claude Code also has its checkpoints and /rewind, but this test does not need them. For the branch-and-commit routine itself, see a git workflow for AI coding agents.
The same prompt for Claude Code and Codex
Save the prompt as ../cmp/prompt.md and pass it unchanged to both. Describe the bug the way you would to a person (we covered how in how to report a bug to AI) and add the rules you are going to measure:
Bug: with a subtotal of exactly $999 MXN the store charges shipping.
Shipping is free from $999. The test "ships free from exactly $999"
in test/shipping.test.js reproduces it.
Fix the code so that npm test passes.
Do not modify, delete or skip files in test/: they are the spec.
Do not commit: leave your changes in the working tree.
When you finish, summarize in three lines what you changed and why.
- The first three lines are the bug: what happens, what should happen and which test reproduces it.
- "Fix the code so that npm test passes" gives the definition of done and the command the agent can run.
- The
test/rule is what the third metric measures. If it is not written down, editing a test breaks no rule. - "Do not commit" leaves both results in the same state; you make the commit after each run.
- The closing summary is not measured, but it tells you whether the agent understood the cause.
Also check which project instructions each one reads, because they are part of the prompt. Per its AGENTS.md page, Codex reads AGENTS.md files before doing any work. Per its memory documentation, Claude Code reads CLAUDE.md and, since version 2.1.277, reads AGENTS.md when there is no CLAUDE.md. If your repo only has a CLAUDE.md, Codex never sees those rules. Use one shared AGENTS.md (imported from your CLAUDE.md, if you have one) or none at all. What belongs in it is in AI coding agent context.
Now run each agent in its folder, without stepping in. Both commands use each CLI's non-interactive mode and save its output as JSON:
cd ../cmp/claude
start=$(date +%s)
claude -p "$(cat ../prompt.md)" --permission-mode acceptEdits \
--allowedTools "Bash(npm test *)" --output-format json > ../runs/claude-1.json
echo "claude-1: $(( $(date +%s) - start )) s" | tee -a ../runs/times.txt
git add -A && git commit -qm "claude: bug 1" --allow-empty
cd ../codex
start=$(date +%s)
codex exec --sandbox workspace-write --json "$(cat ../prompt.md)" > ../runs/codex-1.jsonl
echo "codex-1: $(( $(date +%s) - start )) s" | tee -a ../runs/times.txt
git add -A && git commit -qm "codex: bug 1" --allow-empty
start=$(date +%s)stores the start time in seconds, and theecholine appends the duration totimes.txt: the same clock for both.claude -pruns Claude Code non-interactively. Per its non-interactive mode documentation, in-pan action that needs approval is denied unless something approves it:--permission-mode acceptEditslets it edit files and--allowedTools "Bash(npm test *)"lets it runnpm test, with or without arguments, per the permission rule syntax.--output-format jsonprints one object with the result,duration_ms,num_turns,total_cost_usd,usageandmodelUsage.codex execruns Codex non-interactively. Per its non-interactive mode documentation, it works in a read-only sandbox by default and--sandbox workspace-writeallows edits. The same page notes that--full-autois no longer supported.--jsonprints JSON Lines events (thread.started,turn.started,item.*,turn.completed); eachturn.completedcarries the token usage.git add -Aand the commit freeze what each agent left, new files included.--allow-emptyalso records a run that changed nothing.
The permissions are not identical: Claude Code can only run the command you listed, while Codex runs its commands inside its sandbox. Write down what each one was allowed to do. Run them one after the other, not at the same time, and alternate the order: one goes first on bug 1, the other on bug 2.
The four metrics: time, diff, untouched tests and cost
Three of the four metrics use the same commands for both tools: the shell clock and git. The fourth, cost, depends on what each one reports:
| Metric | How to measure it | Claude Code | Codex |
|---|---|---|---|
| Time | date +%s before and after the run | Also reports duration_ms in its JSON | The sample event stream in its docs has no duration field: use the shell clock |
| Diff | git diff --shortstat between the base and the run's commit | Same command | Same command |
| Untouched tests | git diff --name-only on test/ is empty, and the suite passes with the original tests | Same command | Same command |
| Cost | Tokens, and what they cost under how you pay | total_cost_usd (an estimate) and per-model tokens in modelUsage | Tokens on each turn.completed: input_tokens, cached_input_tokens, output_tokens, reasoning_output_tokens |
Scroll the figure sideways to see all of it
This script measures the diff and the tests. Save it as ../cmp/agent-metrics.sh:
#!/usr/bin/env bash
# Diff size, test files touched and the original suite for one agent run.
# usage: bash agent-metrics.sh <base> <dir> "<test command>" <test path>...
set -uo pipefail
[ $# -ge 4 ] || { echo 'usage: agent-metrics.sh <base> <dir> "<test command>" <test path>...' >&2; exit 2; }
base=$1 dir=$2 cmd=$3
shift 3
git -C "$dir" rev-parse -q --verify "$base^{commit}" > /dev/null || { echo "unknown base: $base" >&2; exit 2; }
if [ -n "$(git -C "$dir" status --porcelain)" ]; then
echo "$dir has uncommitted changes: commit the agent's result first" >&2
exit 2
fi
echo "run: $(git -C "$dir" branch --show-current)"
stat=$(git -C "$dir" diff --shortstat "$base" HEAD)
echo "diff: ${stat:- no changes}"
touched=$(git -C "$dir" diff --name-only "$base" HEAD -- "$@")
if [ -n "$touched" ]; then
echo "tests: TOUCHED"
echo "$touched" | sed 's/^/ /'
else
echo "tests: untouched"
fi
# Put the original tests back, run the suite, then return to the agent's commit.
git -C "$dir" restore --source="$base" -- "$@" || exit 2
if (cd "$dir" && bash -c "$cmd") > /dev/null 2>&1; then suite=pass; else suite=FAIL; fi
git -C "$dir" reset -q --hard
git -C "$dir" clean -qfd -- "$@"
echo "suite: $suite (with the original tests)"
- It takes the base (
cmp/base-1), the run's folder, the test command and the test paths. Without all four arguments it prints the usage and exits with code 2. - It checks that the base exists and that the folder has no uncommitted changes: the script resets the folder at the end and must not throw away anything unsaved.
git diff --shortstatsummarizes the diff: files changed, lines added and removed, per the git diff documentation.git diff --name-only, limited to the test paths, lists every test the agent changed, deleted or added. If the list is not empty, the run is markedTOUCHED.git restore --sourceputs the test paths back as they were on the base. Per the git restore documentation, by default it also removes tracked files that do not exist on the base, so a test the agent added does not run.- It runs the suite with the original tests and records
passorFAIL. Thenreset --hardandcleanleave the folder as the agent left it.
From ../cmp, run it once per agent: bash agent-metrics.sh cmp/base-1 claude "npm test" test, then the same with codex. This is the output from our practice repo, with two branches of hand-made commits and no agent involved: agent-a changed > to >=, and agent-b changed the test's expected value and added a comment to the code.
run: cmp/agent-a
diff: 1 file changed, 1 insertion(+), 1 deletion(-)
tests: untouched
suite: pass (with the original tests)
run: cmp/agent-b
diff: 2 files changed, 2 insertions(+), 1 deletion(-)
tests: TOUCHED
test/shipping.test.js
suite: FAIL (with the original tests)
agent-achanged one line, left the tests alone, and the original suite passes: it fixed the bug.agent-balso leftnpm testgreen, but it editedtest/shipping.test.js, and with the original test the suite fails: the bug is still there.
That is why the test metric has two parts. In another run we deleted the test file: npm test stayed green, because node --test with no test files does not fail (measured with Node 22.22.3), and only the run with the original tests showed FAIL. The other ways to "fix" a test, such as skipping it or loosening a mock, are in how to review what an AI agent did to your tests.
For time, use times.txt. It includes network and model latency, and it varies between runs of the same tool, so treat small differences as a tie and, if you can, run each bug twice per tool. For the diff, smaller is not always better: look at which files it touched, whether it refactored things you did not ask for and whether it added dependencies, because that is what you will review.
For cost, this script reads the tokens from each saved output. Save it as ../cmp/usage.mjs:
// Token usage (and Claude Code's cost estimate) from one run's saved output.
// usage: node usage.mjs <claude-run.json | codex-run.jsonl>
import { readFileSync } from "node:fs";
const text = readFileSync(process.argv[2], "utf8").trim();
let events;
try {
events = [JSON.parse(text)]; // Claude Code: one JSON document
} catch {
events = text.split("\n").filter(Boolean).map((line) => JSON.parse(line)); // Codex: JSON Lines
}
const result = events.find((e) => e.type === "result");
if (result) {
// Claude Code, --output-format json: one result object
const sum = (key) => Object.values(result.modelUsage ?? {}).reduce((n, m) => n + (m[key] ?? 0), 0);
console.log({
tool: "claude",
models: Object.keys(result.modelUsage ?? {}),
input: sum("inputTokens"),
cache_read: sum("cacheReadInputTokens"),
cache_write: sum("cacheCreationInputTokens"),
output: sum("outputTokens"),
cost_usd_estimate: result.total_cost_usd,
error: result.is_error,
});
} else {
// Codex, exec --json: usage on every turn.completed event
const turns = events.filter((e) => e.type === "turn.completed");
const sum = (key) => turns.reduce((n, e) => n + (e.usage?.[key] ?? 0), 0);
console.log({
tool: "codex",
turns: turns.length,
input: sum("input_tokens"),
cached_input: sum("cached_input_tokens"),
output: sum("output_tokens"),
reasoning_output: sum("reasoning_output_tokens"),
failed: events.some((e) => e.type === "turn.failed" || e.type === "error"),
});
}
- It reads the file as a single JSON document (Claude Code's output); if that fails, it reads it line by line as JSON Lines (Codex's).
- For Claude Code, it adds up each model's tokens in
modelUsage, which per the SDK cost tracking documentation includes subagents, and printstotal_cost_usd. - For Codex, it adds up the
usageon everyturn.completedevent and flags anyturn.failedorerror. - Usage:
node usage.mjs runs/claude-1.jsonandnode usage.mjs runs/codex-1.jsonl. We tested it on sample outputs that follow each tool's documented format, not on real runs.
Before comparing costs, pick a common unit. Claude Code's documentation says total_cost_usd is an estimate computed on your machine at list price and can differ from your bill; its cost management page adds that for Pro, Max and Ultra subscribers the session cost figure is not relevant for billing, since usage is included in the plan. For Codex, per its pricing page, usage on a ChatGPT plan draws on the plan's limits (check them with /status or the usage dashboard), and usage through an API key is billed per token. If you pay by API for both, compare dollars: Claude Code's estimate against Codex's tokens times its model's price. On subscriptions, record the tokens and compare each tool only with itself across bugs: two vendors' tokens are not the same unit.
Repeat with three bugs
One bug is an anecdote. Pick three with different shapes:
- A one-line logic error, like the free shipping threshold.
- A bug that spans two files, for example different rounding in the cart and at checkout.
- A bug where the tempting shortcut is editing a test or a snapshot, because the test looks "stale".
The best candidates are bugs you already fixed: you know the right answer, you have the regression test, and you can compare each diff with yours. But in a worktree the fix is still visible: in our practice repo, git log --all from the worktree showed the fix commit. For those bugs, use clones that only carry history up to the base instead of worktrees:
fix=a1b2c3d # the commit where you fixed bug 2
git switch -c cmp/base-2 "$fix^"
git checkout "$fix" -- test/shipping.test.js
git commit -m "test: bug 2 (fails)"
git switch main
git clone --no-local --single-branch --no-tags --branch cmp/base-2 . ../cmp/claude-2
git clone --no-local --single-branch --no-tags --branch cmp/base-2 . ../cmp/codex-2
git -C ../cmp/claude-2 switch -c cmp/claude-2
git -C ../cmp/codex-2 switch -c cmp/codex-2
cmp/base-2starts from the commit before the fix ("$fix^").git checkout "$fix" -- test/…brings in only that commit's regression test, without the fix itself. Swap in the path to your test; the commit leaves the base with the test failing.- Per the git clone documentation,
--no-localuses the regular git transport instead of copying the objects folder,--single-branchbrings only that branch's history and--no-tagsskips tags. In our test, the fix commit did not exist in the clone. - Each clone gets its own branch, so the script's output says whose run it is.
Install dependencies in each clone, run the agents as before, and measure against the remote base: bash agent-metrics.sh origin/cmp/base-2 claude-2 "npm test" test. The prompt is the same template; only the bug description changes.
How to decide
Fill in the scoreboard with each tool's three runs and decide in this order:
- The gate comes first. A run that touched tests or fails the original suite does not count as a fix, however fast or cheap it was.
- Count the gates. How many of the three bugs each tool passed. This is the difference that matters most.
- If the gates are tied, look at the diff. It is what you will review and maintain: changes outside the bug's area weigh more than a few extra lines.
- Then cost in your unit, and time last, since it is the noisiest metric.
- If it is still close, call it a tie. Decide by what you already pay for and how each tool fits your workflow (terminal, editor, CI).
Scroll the figure sideways to see all of it
The result holds for your repo, your three bugs and the versions and models you tested. Keep the scoreboard and the runs/ files, and repeat the test when any of those changes. When you are done, remove the test folders with git worktree remove ../cmp/claude (and the codex one), and the branches with git branch -D.
Common mistakes
- Starting from different commits. If one run uses its tool's own worktree and the other your
HEAD, you are not comparing the same thing. Create both worktrees from the same base branch. - Giving them different context. A
CLAUDE.mdwithout anAGENTS.mdgives one tool rules the other never sees. - Helping one mid-run. Once you clarify something for one, it is no longer the same prompt. Fix
prompt.mdand rerun both. - Counting a green run with edited tests. A green
npm testis not enough: checktests:and the suite with the original tests. - Comparing tokens across vendors. Compare money if you pay by API, or each tool with itself on subscriptions.
- Deciding on one bug and one run. The same agent can solve the same bug differently on two runs. Three bugs is the minimum.
Frequently asked questions
Can I run both agents at the same time?
Each has its own folder, so they will not overwrite each other's files. But if their test suites compete for the same machine, time stops being comparable. To measure time, run them one after the other and alternate who goes first.
What if both fail the same bug?
Check the prompt before the tools: it may be missing information, or the test may ask for something ambiguous. Fix prompt.md, recreate both folders from the base, and rerun that bug with both.
Does this method work for other coding agents?
Yes. It only needs a non-interactive mode, git and a way to read token usage. Change the run command and, if the output has another format, adapt usage.mjs; agent-metrics.sh stays the same.
Which model should I use in each tool?
The one you would use in your daily work, because that is what you want to learn about. Pin it with --model in Claude Code and -m in Codex so it does not change between bugs, and write it on the scoreboard: if you switch models later, rerun the test.
Checklist
- The bug has a failing test, committed on the base branch.
- Both worktrees (or clones, if the bug was already fixed) start from the same commit with the same dependencies.
- Both tools read the same project instructions: one shared
AGENTS.mdor none. - The prompt lives in
prompt.mdand includes the rules not to touchtest/and not to commit. - Each run is timed with
date +%s, saves its JSON inruns/and is frozen with a commit. agent-metrics.shreported the diff, tests and original suite for every run.- Cost is in a common unit, and each CLI's model and version are written down.
- It was repeated on three bugs, alternating the order, and the decision started at the gate.
With this, the question stops being which agent is better and becomes which one fixes your bugs better. Before you accept any diff, check what the agent did to your tests; to undo inside a Claude Code session, see checkpoints and /rewind, and for everything else, your git workflow with agents.
Saw this on Instagram? Comment VERSUS on the Reel and I'll send you the guide.
Written by

Published November 11, 2026