AI coding agents are getting very good at finishing tasks.

They modify files.

Fix errors.

Write tests.

Run commands.

Then they end with something like:

All tests pass ✅

And most of us move on.

But there is an important question:

Did the agent actually run the tests it claims passed?

That sounds obvious.

It isn’t.

Because in AI-assisted development, we are starting to trust summaries instead of evidence.

“Tests Pass” Is a Claim

Imagine an agent says:

Implementation complete.

✓ Tests pass
✓ Build succeeds
✓ No lint errors

That looks reassuring.

But you still do not know:

• what command it ran
• whether it ran the full test suite
• whether some tests were skipped
• whether the command exited successfully
• whether the output came from the latest code
• whether it ran tests at all

The message is a summary.

It is not proof.

This Is a New Kind of Trust Problem

Before coding agents, developers usually ran commands directly:

npm test

You saw the output.

You saw the failures.

You saw the exit code.

With agents, the workflow can become:

Developer asks for feature
        ↓
Agent changes code
        ↓
Agent runs something
        ↓
Agent summarizes result
        ↓
Developer trusts summary

There is now another layer between you and the actual verification.

That layer can be wrong.

The Agent May Have Run Only Part of the Tests

Suppose your project has:

unit tests
integration tests
API tests
end-to-end tests

The agent runs:

npm run test:unit

Everything passes.

Then it reports:

All tests pass.

Technically, some tests passed.

But the full application was never verified.

That difference matters.

It May Be Reporting Stale Results

Another easy failure mode:

Agent runs tests
        ↓
Tests pass
        ↓
Agent changes code again
        ↓
Agent reports "tests pass"

The statement was true earlier.

It may no longer be true now.

Verification should happen against the final state of the code.

Passing Tests Can Still Mean the Wrong Tests

There is another problem.

AI writes the feature.

Then AI writes the tests.

Then AI runs those tests.

They pass.

Great.

Except both the implementation and the tests may share the same misunderstanding.

For example:

Wrong requirement interpretation
        ↓
AI writes code
        ↓
AI writes tests for that interpretation
        ↓
Tests pass

Everything is green.

The feature is still wrong.

So there are really two questions:

Did the tests run?

and

Were they the right tests?

Both matter.

Require Evidence, Not Confidence

I have started preferring a simple rule:

If an agent makes a verification claim, ask for the evidence behind it.

Instead of accepting:

Tests pass.

Ask for:

Show me:

1. The exact command you ran.
2. The exit code.
3. How many tests ran.
4. How many failed.
5. How many were skipped.
6. Whether this was run after the final code change.

Now the claim becomes inspectable.

Give Your Agent a Verification Contract

You can make this part of your normal coding-agent instructions.

For example:

Never say "tests pass" unless you actually ran the relevant test command.

When reporting verification, include:

- exact command
- exit code
- number of tests
- failures
- skipped tests
- build status
- lint/typecheck status

If you did not run something, say "not verified."

That last line is especially important:

If you did not verify it, say so.

“I did not run the integration tests” is much more useful than false confidence.

Separate Implementation From Verification

For important changes, do not let the same workflow both create and certify the result.

A stronger pattern is:

Agent implements feature
        ↓
Independent test command
        ↓
CI verifies
        ↓
Human reviews result

For higher-risk changes, you can go further:

Agent A → implementation

Agent B → adversarial review

CI → tests/build/security checks

Human → final approval

The important part is independence.

The system that produced the code should not be your only source of confidence that the code is correct.

CI Should Be the Source of Truth

The agent's message should be treated as useful context.

Not final authority.

For example:

Agent says:

All tests pass.

CI says:

3 integration tests failed.

Trust CI.

This is why traditional engineering systems still matter even when AI writes more of the code.

You still want:

• CI
• deterministic tests
• build checks
• type checks
• linting
• security scans
• deployment gates

AI does not replace those systems.

It makes them more important.

Be Careful With “Fixed”

The same principle applies to other agent claims.

For example:

“The bug is fixed.”

How was it verified?

“The build works.”

Which build command ran?

“There are no breaking changes.”

What compatibility checks were performed?

“The migration is safe.”

Was it tested against realistic data?

“This is secure.”

What security checks were actually run?

AI agents can sound extremely certain.

Certainty is not evidence.

A Better Completion Message

Instead of this:

Done.

Everything works and all tests pass.

I would rather see:

Implementation complete.

Verification performed:

npm test
Exit code: 0
128 tests passed
0 failed
3 skipped

npm run typecheck
Exit code: 0

npm run lint
Exit code: 0

Integration tests were NOT run.

Manual browser testing was NOT performed.

That is much more useful.

Now I know exactly what was verified.

And what was not.

Add “Not Verified” to Your Vocabulary

One thing AI-assisted development needs more of is:

Not verified.

There is nothing wrong with an agent saying:

Implemented, but not tested.

or:

Unit tests pass, integration tests not run.

or:

I could not verify this because the required service is unavailable.

Those answers are better than pretending certainty.

Good engineering is not about sounding confident.

It is about knowing what evidence you actually have.

My Simple Rule

For any important AI-generated change:

Never trust this:

“It works.”

Prefer this:

“Here is how I verified it.”

That one difference can prevent a lot of false confidence.

A Practical Checklist

Before accepting an agent's “tests pass” message, ask:

What command was executed?

You should know the exact command.

Was it the full relevant test suite?

Not just one subset.

What was the exit code?

Success should be measurable.

Were any tests skipped?

Skipped tests matter.

Was verification run after the final code change?

Not before the last edit.

Did CI confirm the result?

Prefer independent verification.

Can I inspect the output?

Evidence should be available.

Final Thought

AI coding agents are becoming very good at producing code.

But as they become more autonomous, developers need to become more careful about one thing:

verification.

The dangerous workflow is:

AI writes code
↓
AI says it works
↓
Human believes it
↓
Merge

The better workflow is:

AI writes code
↓
AI provides evidence
↓
Independent checks run
↓
Human verifies
↓
Merge

Because:

“Tests pass” is not evidence that tests passed.

It is a claim.

And in software engineering, important claims should come with receipts.