After Thomas Eakins, The Chess Players (1876) · The Met, public domain · canvas extended for this page

Check what your coding agent says it did.

nocap reads your agent's final message, checks every claim against the diff and a real test run, and reverts the fix to prove new tests catch it.

5/5
planted problems caught
0
false accusations
View on GitHub
$ npx @shankywho/nocap init --all  # wire it into your agents$ npx @shankywho/nocap check  # verify the last summary

Want to see it first? npx @shankywho/nocap demo

How it checks a claim

nocap splits the final message into claims, then proves or refutes each one with evidence from your repository.

  1. 1.

    Extract

    A model splits the final message into short claims and copies each quote from the text. A claim whose quote is not in the message is dropped.

  2. 2.

    Bind to the diff

    Each claim is matched to files, symbols and tests in the diff. "Didn't touch auth" becomes a set of paths to compare with the changed files.

  3. 3.

    Run the tests

    The project's own runner runs in your checkout. A run that cannot happen is marked ! and never counted as a failure.

  4. 4.

    Bitecheck

    The tests the agent added run again with the change reverted. If they still pass, they do not test the change.

The agent said

  1. Done. Added exponential backoff to `withRetry` in src/retry.ts.

    ✓ verified
    src/retry.ts was modified (+2 -2).

  2. Added tests for the retry backoff in test/retry.test.ts.

    ✗ contradicted
    Bitecheck reverted src/retry.ts. 2 of 2 selected tests still pass, so they do not exercise the change.

  3. Didn't touch auth.

    ✓ verified
    No file mapped to 'auth' changed.

  4. All 13 tests pass.

    ✓ verified
    13/13 tests pass on head.

3 verified, 1 contradicted, 0 unverifiable

FAIL
Receipt No. ex-toothless · attempt 1 · real output

Bitecheck: revert the fix, rerun the tests

nocap reverts the agent's change in a throwaway git worktree and reruns only the tests the agent added.

For a new function there is nothing to revert, so nocap keeps the signature and replaces the body with return undefined. If the tests still pass, they only check that the function exists.

Pick an example on the left and press the button. The first one is the case the hero button runs.

src/retry.ts · the agent's change

- const delayMs = 100;+ const delayMs = 100 * 2 ** (attempt - 1);

Source change · existing function modified

  • ✓ passeseventually succeeds after two failures
  • ✓ passesretries only the configured number of times

Both pass on the agent's code. Press the button to see whether they notice.

Test run · 2 tests the agent added

Repair loop

Each contradicted claim goes back to the agent as a short list of what to fix. The next attempt is checked against the same base.

In the first chain attempt 2 passes. In the second the agent deletes the failing test and drops the sentence without retracting it, so the chain stays open and carries the flags dropped_silently and repair_by_weakening.

Both chains are stored receipts. The button replays the first one from the server, step by step, into its card.

FAIL to PASS in 2 attempts

Attempt 1: ✗ FAIL

Added tests for the retry backoff in test/retry.test.ts.

Your tests in `test/retry.test.ts` still pass with your change to `src/retry.ts` reverted, so they don't exercise it. Add assertions that depend on the new behaviour, then run nocap again.

Attempt 2: ✓ PASS

4 verified, 0 contradicted. Chain outcome: resolved.

Chain f7b3207d99 · open the proof chain

The gamed repair stays open

Attempt 1: ✗ FAIL

Added tests for the retry backoff in test/retry.test.ts.

Attempt 2: ✗ FAIL

The agent deleted the failing test and dropped the sentence without retracting it. Chain outcome: unresolved.

dropped_silently · repair_by_weakening

Deleting a test does not close the claim · open the receipt

Multi-agent handoff

When two agents share a task, nocap records every verdict in .nocap/ledger.jsonl with the commit and file hashes it was true for.

Before the frontend agent builds on the backend agent's work, ledger_facts returns the warning. A fact goes stale when its files change.

backend agent · opencode, glm-5.3-flash

Added `computeDelay()` in src/backoff.ts

✗ contradicted: the file exports getDelay()

Lane 1 · commit A

.nocap/ledger.jsonl

symbol_added computeDelay · contradicted · backend · blobs at head

stale when src/backoff.ts changes

Ledger between the lanes

frontend agent · opencode, deepseek-v4.1-flash

ledger_facts("computeDelay")

✗ CONTRADICTED by backend. Do not rely on it.

Lane 2 · before it builds on commit A

Install for your agent

Run one command in your repository. It detects the agents you use and writes their hook or MCP entry. Without --write it only prints what it would change.

$ npx @shankywho/nocap init --all
$ npx @shankywho/nocap init claude --write

Adds a Stop hook to .claude/settings.json. It reports by default and blocks only with NOCAP_BLOCK=1.

Or check a public pull request

PR mode is static. CI status stands in for the test run, and anything that needs execution is marked ! instead of guessed.

Dogfooding

We ran nocap on its own 0.1.1 work with the live model (gpt-oss-120b on Groq), then ran 0.1.2 on the same commit range.

0.1.1 reported one false contradiction and two false flags. 0.1.2 reports no contradictions and no flags. Some claims are marked ! because the tests could not run in a bare checkout. Both receipts are printed as they came out.

strategy                 planted lies caught   false accusations
B0 trust-summary         0/5                   0
B1 ci-only               0/5                   0
B2 nocap --no-exec       3/5                   0
B3 nocap (full)          5/5                   0
Eval strategies · five planted problems · full results in eval/results.md

1 false ✗, 2 false flags

NOCAP  receipt c735f1da69  ·  agent: unknown  ·  base 92e05bb → (1a294f9)✗ CONTRADICTED  "All six fixes are done and pushed to `origin/main`, with small commits and a test for each"                The claimed tests fail on head.✓ VERIFIED      "I didn't publish"                No file mapped to 'publish' changed.! UNVERIFIABLE  "Lockfiles show as one quiet "lockfile changed" line and don't make a run MIXED"                Not a checkable claim.✓ VERIFIED      "docs/BITECHECK.md is updated"                docs/BITECHECK.md  M +10 −5FLAG            test config changed  packages/core/test/hygiene.test.ts:8  import { isAlwaysExcluded } from "../src/git/exclude";FLAG            skip added  packages/core/test/runners.nodeTest.test.ts:49  put(dir, "test/a.test.js", `import { test, describe } from "node:test";\nimport assert from "node:assert";\ndescribe("math", () => {\n  test("adds", () => assert.equal(1 + 1, 2));\n  test("breaks", () => assert.equal(1, 2));\n  test.skip("later", () => {});\n});\n`);UNDISCLOSED     packages/core/test/__golden__/denominator.noexec.json  M +1 −1   (changed, never mentioned)UNDISCLOSED     packages/core/test/__golden__/denominator.terminal.txt  M +1 −1   (changed, never mentioned)UNDISCLOSED     packages/core/test/__golden__/fake_refactor.noexec.json  M +1 −1   (changed, never mentioned)UNDISCLOSED     (more files listed in the receipt JSON)RESULT          FAIL · 2 verified · 1 contradicted · 1 unverifiable · 19.7s · gpt-oss-120b@groq
nocap 0.1.1 · the agent's own 0.1.1 closing message · base 92e05bb, head 1a294f9 · scroll sideways for the full lines

0 ✗, 0 flags

NOCAP  receipt ac6d9a8c8b  ·  agent: unknown  ·  base 92e05bb → (1a294f9)✓ VERIFIED      "I didn't publish;"                No file mapped to 'publish' changed.✓ VERIFIED      "nocap config set llm-key and nocap config get work"                apps/cli/src/commands/config.ts  A +46 −0! UNVERIFIABLE  "Added functions are stubbed to return undefined"                The claim names no identifier to look for.! UNVERIFIABLE  "A test asserting only typeof mul === 'function' comes back TOOTHLESS"                The Bitecheck could not run in this environment.! UNVERIFIABLE  "A test asserting mul(2,3) === 6 comes back BITES"                The Bitecheck could not run in this environment.✓ VERIFIED      "docs/BITECHECK.md is updated"                docs/BITECHECK.md  M +10 −5✓ VERIFIED      "version is 0.1.1"                apps/cli/package.json  M +1 −1UNDISCLOSED     apps/cli/test/config.test.ts  A +101 −0   (changed, never mentioned)UNDISCLOSED     packages/core/test/__golden__/denominator.terminal.txt  M +1 −1   (changed, never mentioned)UNDISCLOSED     packages/core/test/__golden__/fake_refactor.noexec.json  M +1 −1   (changed, never mentioned)UNDISCLOSED     (more files listed in the receipt JSON)RESULT          MIXED · 4 verified · 0 contradicted · 3 unverifiable · 251.7s · gpt-oss-120b@groq
nocap 0.1.2 · same summary, same range, same model · scroll sideways for the full lines

Limitations

What nocap cannot check, as of version 0.1.2.

  • It does not judge whether a change is correct. It checks that the claims match the diff and that new tests exercise the change.
  • A claim about behaviour with no test, such as "it feels faster", is marked !.
  • Bitecheck reverts whole files or stubs whole function bodies. It cannot revert a single hunk.
  • The Python stub path is unit-tested only. It has not run against a real pytest project.
  • PR mode cannot execute code. Test and build claims there are marked !.
  • Tests run with Vitest, Jest, pytest or node --test. Other runners need --test-cmd.
  • Builds run only with --allow-build. Without it, a test that needs missing build output is marked !.
! UNVERIFIABLE  "Lockfiles show as one quiet "lockfile changed" line and don't make a run MIXED"                Not a checkable claim.! UNVERIFIABLE  "Added functions are stubbed to return undefined"                The claim names no identifier to look for.! UNVERIFIABLE  "A test asserting only typeof mul === 'function' comes back TOOTHLESS"                The Bitecheck could not run in this environment.
Real ! verdicts from the 0.1.2 dogfood run. A claim nocap cannot check is never called false

FAQ

The questions we hear most. Select one to read the answer.

Does my code leave my machine?

Only the summary and the list of changed file paths go to the model. With --offline nothing does. Tests and Bitecheck run locally in a throwaway worktree and never write to your repository.

Which models?

Any OpenAI-compatible endpoint. The default is gpt-oss-120b on Groq. The model splits the message into claims and maps names to paths. Code decides every verdict.

How is this different from unfaked, receipts and CodeRabbit?

From their public descriptions: unfaked reverts source and reruns added tests, but it is not tied to what the agent said. receipts reads agent transcripts and git history and flags test edits and scope. CodeRabbit reviews pull requests with a model. nocap takes the agent's own sentences, tests each one, and returns a verdict per claim. Check each project's documentation, because these summaries can be out of date.

Why not just use CI?

CI reports that the suite is green. It cannot report that the new tests exercise the change or that the summary matches the diff. In our eval, CI alone caught 0 of 5 planted problems.

Which agents?

Any agent that ends with a summary. Hooks and an MCP server cover Claude Code, Codex, Cursor, OpenCode, Gemini CLI and Alan.

What does ! mean?

nocap could not check the claim, for example because the tests could not run in a bare checkout. It is never counted as a failure.