Profile
Back to NewsBack
Dev.to 5 min
Reader Mode
Tool Names Are Identifiers. Don't Autocorrect Them.

Tool Names Are Identifiers. Don't Autocorrect Them.

10 hours ago

An agent contract requires retrieve_policy. The trace seems to contain retrieve_policy. The evaluator says the tool is missing.

Before changing the contract, inspect the characters.

const expected = "retrieve_policy";
const observed = "retrieve_p\u043Elicy";

console.log(expected === observed); // false

The observed value contains a Cyrillic small letter o, U+043E, where the expected identifier contains Latin o, U+006F. In many fonts, the names look alike. They are different strings.

I maintain AgentInspect. This example was tested against 6.31.35, whose diagnostics can expose code points for certain suspicious names. The important design choice is not a clever Unicode trick: improve the explanation without silently changing the identifier being evaluated.

Equality and readability solve different problems

A tool name participates in a contract between a tool registry, an invocation, a trace adapter, and an evaluator. If those components disagree, forgiving display logic can hide a real integration defect.

Operation Appropriate purpose Risk if used for policy matching
Exact comparison Compare recorded identifiers May need better diagnostics for humans
Escaped/code-point display Explain visually ambiguous values Must not replace the raw identity
Friendly display label Make a UI readable Several identifiers may share a label
Case folding or normalization Explicitly designed naming schemes Can silently merge names under a different scheme

An allowlist is not a spellchecker. If the application authorizes one identifier, an adapter should not authorize a lookalike on its behalf.

This does not mean non-Latin tool names are invalid. The contract can explicitly allow a name in any supported naming scheme. The problem is disagreement between identities, not the language used to write them.

Two visually similar tool names have different code points, leading to exact mismatch and an explanatory diagnostic rather than automatic replacement.

Preserve identity; make the mismatch legible.

Reproduce the mismatch without an LLM

Install the pinned package and capture a synthetic invocation:

npm install --save-exact [email protected]
import { inspectRun, step } from "agent-inspect";

await inspectRun("unicode-name-fixture", async () => {
  return step.tool("retrieve_p\u043Elicy", async () => ({
    found: true,
  }));
}, { traceDir: ".agent-inspect", silent: true });

The callback succeeds. That does not make the name match the contract. After opening the resulting trace with openTraceFile, evaluate:

import {
  defineTraceContract,
  evaluateTraceContractRead,
} from "agent-inspect/checks";

const contract = defineTraceContract({
  tools: {
    required: ["retrieve_policy"],
    allowed: ["retrieve_policy"],
  },
});

const result = evaluateTraceContractRead(trace, contract);
console.log(result.status, result.findings);

The measured fixture fails. Its findings include the U+043E diagnostic. A positive control using the exact ASCII name passes.

The two constraints are complementary. required asks whether the expected tool occurred. allowed asks whether an unexpected tool occurred. Depending on your application, the meaningful report may include both a missing expected operation and an unrecognized observed one.

TraceContract remains a beta API. Keep the regression fixture pinned to the version used by your release gate.

Trace the mismatch back to its boundary

When a name differs, investigate the registry, serialized invocation, adapter mapping, and contract definition. A copied description may have become an identifier. A display label may have replaced a registered name. A normalization layer may have rewritten characters.

Do not assume an attack from this fixture alone. Copy/paste errors, inconsistent naming conventions, and adapter defects can produce the same symptom. Diagnostics should report the observed mismatch without making an unsupported claim about intent.

The same discipline applies to leading spaces, case changes, hidden controls, and aliases. First establish the application's naming contract. Then determine which boundary violated it.

If aliases are intentional, declare them in a reviewed mapping and version it. An undocumented “close enough” conversion makes later evidence harder to interpret.

Reduce opportunities for drift

For names you control, share constants between registration, invocation, and test expectations:

export const Tool = {
  retrievePolicy: "retrieve_policy",
  summarizePolicy: "summarize_policy",
} as const;

type ToolName = typeof Tool[keyof typeof Tool];

function acceptsRegisteredName(name: ToolName) {
  return name;
}

The type prevents certain accidental literals in your own TypeScript code. It does not validate an untrusted runtime string. Validate external values at the boundary using the registry's actual identifiers.

Also separate identifiers from descriptions. A readable label can change during localization without renaming an operation. Conversely, changing the operation's identifier should trigger a deliberate migration of contracts and historical comparisons.

Keep diagnostic output bounded

A helpful error report shows the expected name, the observed name, and enough escaped or code-point information to explain the difference. It should not dump unbounded input into a terminal or report.

The current implementation applies additional code-point detail to selected mixed-script or risky-control cases; it is not a universal spoofing detector. A name that receives no extra diagnostic is not thereby trustworthy. Likewise, a name receiving a diagnostic is not automatically malicious.

Keep the raw captured evidence, the comparison result, and the human-friendly explanation conceptually separate. Each has a different job.

Don’t repair the evidence to make the test green

The tempting fix is to normalize the observed name into the expected one. That may make a fixture pass while hiding the actual mismatch.

A better fix corrects the naming boundary or explicitly updates the contract. Then rerun both the exact-match positive control and the lookalike negative control.

When an agent tool crosses registries, SDKs, adapters, and evaluators, a tiny identifier change can become a large debugging detour. The solution is exact policy and understandable diagnostics—not a more forgiving guess.

Implementation references

Chat with me