Our deploy tool pushed a fresh API token to production. It reported success. The token was correct. Every request then failed with this:
Cannot convert argument to a ByteString because the character at
index 7 has a value of 65279 which is greater than 255.
Character 65279 is U+FEFF, a byte-order mark. Index 7 is the first character after Bearer.
Something had prefixed a BOM onto our token, between reading it out of the vault and storing it at the platform. The value was right. The bytes were not.
Why we could not just look
The obvious move is to open the dashboard and read the variable. You cannot. The variable is marked Sensitive, which means it is write-only: the dashboard shows nothing, the CLI shows nothing, and the API returns the row with the value omitted entirely.
That is good security. It is also a total audit blind spot, and it is the reason this took an hour instead of a minute. We had a credential that was wrong in a way nothing could show us, in a store designed so that nobody — including us — could read it back to check.
The repro is two lines
The push script read the secret from the vault and piped it to the platform CLI. That pipe is the bug. Windows PowerShell 5.1:
'SECRETVALUE' | node -e "process.stdin.on('data',d=>console.log(JSON.stringify(d.toString())))"
# "SECRETVALUE\r\n"
There it is. Piping a string to a native command prefixes U+FEFF, because [Console]::OutputEncoding is UTF-8 with a 3-byte preamble in a non-interactive session:
[Console]::OutputEncoding.GetPreamble().Length # 3
The CLI passed those bytes through faithfully. The platform stored them faithfully. Everything downstream was correct about a value that was already wrong.
Two fixes that do not work
The obvious fix is to set the output encoding to a UTF-8 that emits no preamble:
$OutputEncoding = New-Object System.Text.UTF8Encoding $false
'SECRETVALUE' | node -e "..."
# "SECRETVALUE\r\n" <-- still there
Still there. So try the console encoding instead:
[Console]::OutputEncoding = New-Object System.Text.UTF8Encoding $false
'SECRETVALUE' | node -e "..."
# "SECRETVALUE\r\n" <-- still there
Still there.
Both of those are the fixes you will find if you search for this, and we measured both of them failing before writing any code. That mattered: the first patch we wrote was the $OutputEncoding one, and it shipped a comment confidently explaining a fix that did nothing. A control run caught it.
The fix that does work
Stop using the pipe. Write the value to a file with an encoding you control, and redirect that file into the process's stdin:
$stdin = Join-Path $env:TEMP ([guid]::NewGuid().ToString() + '.in')
try {
[IO.File]::WriteAllText($stdin, $plain, (New-Object System.Text.UTF8Encoding $false))
$proc = Start-Process -FilePath $env:ComSpec `
-ArgumentList @('/c', 'thecli', 'env', 'update', $name, 'production', '--yes') `
-RedirectStandardInput $stdin -NoNewWindow -Wait -PassThru
if ($proc.ExitCode -ne 0) { throw "push failed ($($proc.ExitCode))" }
}
finally { Remove-Item $stdin -Force -ErrorAction SilentlyContinue }
Verified at the byte level, which is the only verification that means anything here:
(Get-Content $stdin -Encoding Byte | Select-Object -First 4) -join ','
# 83,69,67,82 <-- "SECR", no preamble
Two things we hit on the way, in case you are doing the same:
- The CLI needs
--yesto overwrite an existing variable, or it exits 1 withconfirmation_required— and you cannot answer the prompt, because stdin is carrying the secret. -
Start-Processcannot launch an npm shim directly (%1 is not a valid Win32 application), hence going through%ComSpec%.
What we actually got wrong
The bug is a one-character encoding quirk. The reason it cost an hour is not.
A write-only secret cannot be audited by reading it. Sensitive storage removes the only check most people have. If you use it — and you should — then the verification has to be behavioural, not textual: after any push, make a real call with the stored credential and assert the response. Not "the CLI said Updated". Not "the row's updatedAt moved". An actual request.
We had that backwards. We trusted a success message from a tool that was faithfully transmitting corruption, about a value we had deliberately made unreadable, and we only found out when something unrelated fell over with an error about character 65279.
The encoding bug will bite someone else. The blind spot is the part worth fixing.