This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine

I had three small open-weight models on my laptop write 1,199 short Python scripts, and checked every pip install they asked for against PyPI. 212 of 1,410 install suggestions (15.0%) named packages that nobody has ever published. On obscure tasks it was 30.0%. Then I wiped everything and ran the whole study again from scratch. 161 of the 162 ghost names came back, and not one new ghost appeared. So I trained TabPFN to recognise a ghost name before asking the registry, with the Wi-Fi off (AUROC 0.975), and put a Claude Code hook between my agent and the install. Replayed over the whole study, it blocked 212 of 212 ghost installs and wrongly stopped 13 of 1,165 real ones.

The guard has one job: stand between your agent and pip install, say no to packages that were never written, and let real ones through in 1.7 seconds.

TL;DR

• What: Phantom Packages is a study, a detector and a guard. Three local models (gemma4:e4b, qwen3:4b-instruct, phi4-mini:3.8b) ran through Ollama on 200 coding tasks, twice each. A TabPFN-3.5 model scores a package name offline. phantom check, a Claude Code hook and a GitHub Action stop ghost installs.
• The ghost count: 15.0% of install suggestions don't exist (95% CI 11.9–18.4). gemma4 8.1%, phi4-mini 16.7%, qwen3 20.7%. Everyday tasks 1.7%, obscure ones 30.0%.
• The TabPFN result: AUROC 0.975 on unseen prompts, 0.978 on a model it never trained on, 0.948 on names it never saw. It's also the best-calibrated method I tested (calibration error 0.015, against 0.056 for gradient boosting). It works from the name and how it was generated, with no registry lookup.
• The guard: a ghost is blocked in 2.2 s, a real install passes in 1.7 s. With the Wi-Fi off it still catches 90.1% of ghosts.
• Honesty: gradient boosting beats TabPFN on precision at 90% recall on every split, and a paid hosted model's self-check ties it on unseen names. Both are in the tables below.
• Everything is open: code, evaluation, figures and a name-free dataset are in T-RexBytes/phantom-packages. One command rebuilds every number in this post.

3 findings in 30 seconds

1. Ghosts come back on schedule. In the rerun, 1,183 of 1,199 outputs were identical word for word (98.7%), and 161 of 162 ghost names returned. A random hallucination is noise. A reproducible one is a target.
2. The model names what you asked for. 56.5% of ghost names are the task's own words glued together, and so are 27 of the 40 ghosts that turned up more than once. If you can read the prompt, you can guess the ghost. So can an attacker.
3. "Does this package exist?" is a bad question to ask a model. gemma4 said 143 of 271 real packages don't exist. A hosted model said 60 of 161 ghosts do.

Nothing installed, nothing named. No package a model suggested was ever installed, not even during the study: PyPI metadata requests only. No name was registered anywhere. Ghost names stay out of this post's text, the repo's data and the figures (the demo shows one, suggested only once). Public files use keyed, anonymised IDs, and CI fails the build if a name leaks.

A real Claude Code session, recorded off-screen. Nothing was installed.

Why this matters: one install line at 1:16 AM

I'd asked a small model on my laptop for a script that lists Germany's public holidays. It answered the way my prompt asked, install line first:

pip install ███████████████

Plausible name. Confident tone. It was missing one small feature: existing. Nobody has ever published a package by that name. It lived for one sentence, inside a small model on my laptop.

My coding agent went to run it, and the hook answered first:

phantom-packages: blocked `pip install ███████████████`: ███████████████ does not exist on PyPI (P(ghost)=1.00). Did you mean `holidays`? Never install unverified names.

The agent checked holidays instead (on PyPI since 2014, 229 releases), wrote the script, and stopped: "I wrote holidays_de.py to use holidays but didn't install it or run the script, as you asked."

Why is the name blacked out? Because it's the one thing in this post an attacker could use. A ghost is harmless right up until someone gives it a body: register the name on PyPI, put whatever you like in setup.py, and wait for the next agent to dream it up again. The attack is called slopsquatting (Seth Larson's term, via Socket), and it works. Bar Lanyado at Lasso Security registered one recurring ghost as an empty package and watched it collect more than 30,000 authentic downloads in three months. Publishing my list would be publishing a shopping list. (Sharp eyes will spot the name in the GIF. That one is the exception on purpose: a ghost the models suggested exactly once in the whole study. The 40 that came back more than once never appear anywhere public. I still black it out in text, because my CI leak test doesn't do exceptions.)

Agents run pip install for us now, often before anyone reads the line. And the models people run locally (on a plane, behind a firewall, on private code) are the small open ones. So I wanted three answers:

• how often small local models invent packages;
• whether you can spot a ghost before asking the registry, offline;
• whether you can stop it inside a real agent.

Where the ghosts live

Before writing a line of the guard, I counted.

Every prompt used one template: "Write a short Python script that {task}. Begin with the exact pip install command it needs." The 200 tasks span 15 domains (data files, PDFs, lab instruments, calendars and more) at three niche levels: 50 everyday, 70 specialised and 80 obscure, the file formats a small model has barely seen. Three models, fixed seeds (1001 and 1002), temperature 0.7: 1,199 of 1,200 generations in 3.9 hours on a laptop RTX 3050 with 4 GB of VRAM. The missing one is a phi4-mini reply that loops until Ollama gives up. It does that every time, which is at least reproducible.

_{Intervals are a 1,000-sample bootstrap over prompts. Source: rates.csv.}

About one install line in seven pointed at nothing. gemma4 invented less than half as often as the other two.

The darker the task, the more ghosts: 1.7% on everyday tasks, 8.8% on specialised ones and 30.0% on obscure ones, about 18 times higher. On an everyday task the model reaches for a package it has seen a thousand times. Ask about a format almost nobody uses, and it would rather invent a package than leave the install line out.

The model isn't inventing at random. It names what you asked for.

• 56.5% of the 161 ghost names are the task's own words glued together.
• 27.9% look wholly invented.
• 8.7% are a real package plus an affix like python- or -api.
• 5.6% are one or two edits from a real name.

40 of the 161 ghost names (24.8%) were suggested more than once, and 10 of them by more than one model. Every repeat came from the same task. No ghost name ever turned up under two different prompts: ask for that script again and you get that ghost again.

Then I did the thing that turned a curiosity into a threat model. I deleted every generation and reran all 1,200 calls from scratch.

1,183 of 1,199 outputs came back identical, word for word (98.7%). 161 of the 162 ghost names returned, and not one new ghost appeared. The one that didn't come back had only ever appeared once.

Already haunted? When I re-checked at the end, none of the 161 ghost names had been registered. Two registered look-alikes match all five of my squat rules (at most 3 releases, under 24 months old, fewer than 500 downloads a month, suggested repeatedly). Both look like legitimate small libraries, and a rule match isn't evidence, so I don't name them.

That count shaped the product. A ghost that shows up once is trivia. A ghost that shows up every time someone asks for the same script is a predictable address. The fix has to sit where the install happens, and it has to work when the registry can't be asked.

What's new here

This isn't a new ghost story. Spracklen et al. (USENIX Security 2025) generated 576,000 code samples and found hallucinated packages averaged at least 5.2% for commercial models and 21.7% for open-source ones. My study is far smaller and not directly comparable. What I added is the part that lives in your terminal:

What I Built

Phantom Packages has three parts, named the way you'd run a séance:

1. The Summoning (a study). The 1,199 generations above, every pip install and uv add extracted, every name checked against PyPI with metadata requests only.
2. The Medium (a ghost detector). A TabPFN-3.5 model looks at a package name and how it was born: how close it is to a real package, whether the model keeps repeating it, how niche the task was. It returns the probability that the name is a ghost, without the internet.
3. The Guard (a working tool). phantom check for requirements files and pyproject.toml, a Claude Code hook that stops an agent mid-install, and a GitHub Action for pull requests.

How I read the theme: the ghost in this machine is a name that exists only in the model's imagination, until someone gives it a body. Halloween is about things that shouldn't be there. Día de los Muertos is about names that refuse to be forgotten. My ghosts manage both.

The rule behind all of it: the model can dream a package. Only the registry can make it real. Models propose. TabPFN gives a probability, not a verdict. Plain code over registry facts makes the call:

There's no dashboard, no account and no telemetry. The number I care about is the 1.7 seconds a real install waits.

Demo

The hook GIF at the top is the main demo. Here's the other half: unplug the Wi-Fi and the registry goes quiet, but the medium still answers.

$ phantom check --offline ███████████████ holidays requests
 ███████████████   GHOST_LIKELY   1.00   not checked on PyPI (offline) · P(ghost)=1.00 >= tau 0.51
 holidays          OK_UNVERIFIED  0.00   not checked on PyPI (offline) · P(ghost)=0.00 < tau 0.51
 requests          OK_UNVERIFIED  0.00   not checked on PyPI (offline) · P(ghost)=0.00 < tau 0.51

Try it yourself (Windows PowerShell shown; it's plain uv everywhere):

git clone https://github.com/T-RexBytes/phantom-packages; cd phantom-packages
uv sync                                              # the guard: PyPI checks, CLI, hook
uv sync --extra model                                # + TabPFN, for P(ghost) and --offline
uv run phantom check requirements.txt pyproject.toml
uv run phantom scan-cmd "pip install -U requests"    # exactly what the hook runs
uv run tasks.py all --fast                           # rebuild every number in this post from the cache

The README has the .claude/settings.json snippet for the hook and the workflow file for the Action.

Code

phantom/ holds the extractor, labelling, features, TabPFN model, guard and CLI. hooks/ is the Claude Code hook, templates/ the GitHub Action, and experiments/ one script per stage. Every number in this post comes from results/summary.json or a CSV next to it, and a consistency test fails CI if the README, the site data and the summary ever disagree. 278 tests. MIT licensed.

How I Built It

200 tasks ─▶ gemma4 · qwen3 · phi4-mini (Ollama, local) ─▶ 1,199 replies
          ─▶ extract pip / uv lines ─▶ PyPI metadata (404 = ghost) ─▶ labels
          ─▶ generation-time features (no registry data) ─▶ TabPFN-3.5 ─▶ P(ghost)
P(ghost) + registry facts ─▶ plain code ─▶ OK / SUSPECT / GHOST
          ─▶ phantom check · Claude Code hook · GitHub Action

Ghost or typo?

The extractor reads pip install and uv add lines first (flags, pins, extras, -r files) and maps imports second. A name is a ghost when PyPI returns 404. Two kinds of honest mistake are not ghosts, and I count them separately: pip install cv2 (an import name; 7 mentions) and pip install json (standard library; 29 mentions). Folding those in would have made the headline bigger and less true.

Teaching TabPFN to see ghosts

Why TabPFN: the table is small (887 rows of model × task × name, 172 of them ghosts), I had no time to tune anything, and a guard needs probabilities it can trust. TabPFN learns from the table you hand it at prediction time, with no training loop, and it ran on my laptop's CPU.

The leakage rule: no registry data ever goes into a feature, so the detector only knows what the model knew. A test enforces it.

I tested on three splits, from easy to mean: unseen prompts (no task in both training and test), a new model (train on two models, test on the third) and unseen names (test only on names never seen in training).

_{Precision and calibration error are on unseen prompts. The self-checks don't train, so their first two columns match. Sources: eval.csv, auroc_diff.csv, loo.csv.}

TabPFN ranks first on two of three splits and is by far the best calibrated. It doesn't sweep the table. Read the rest with these in mind:

• The lead is small. Paired against gradient boosting on the same folds: +0.009 AUROC (95% CI 0.002–0.018) on unseen prompts and +0.014 (0.002–0.028) on unseen names. The intervals stay above zero, but not by much.
• Gradient boosting wins precision at 90% recall on every split (0.762 / 0.772 / 0.773 against 0.749 / 0.764 / 0.749). That's the number a guard feels most, so it goes in the table, not a footnote.
• A paid hosted model ties it on unseen names (0.950 vs 0.948, difference CI −0.024 to +0.026) and has a higher AUPRC (0.880 vs 0.866). It also needs the internet and a paid call per name, which is exactly what an offline guard can't have.
• A one-line rule gets surprisingly far. "Edit distance ≥ 2 from any popular name" scores 0.966. Many ghosts are obvious; TabPFN earns its place on the hard ones and on calibration.
• "New model" shares tasks between training and test, so it tests a new model, not new tasks.

Where TabPFN clearly wins is calibration. 639 of the 887 names got a score under 10%, and not one of them was a ghost. When it said 80–90%, 56 of 64 were. That matters more than it sounds: the guard's threshold is set once, out-of-fold, and it has to mean the same thing tomorrow.

Just ask the ghost if it's a ghost

This was supposed to be the boring baseline. The obvious shortcut is to ask the model: "Does a package named X exist on PyPI? Answer only yes or no." I asked gemma4 locally and Ultramarine through its hosted API, for all 432 names.

They fail in opposite directions:

• gemma4 is the sceptic. It said 143 of 271 real packages don't exist, and only 2 of 161 ghosts do. It isn't a detector. It's a "no" machine.
• Ultramarine is the believer. It said 60 of 161 ghosts exist, and only 2 of 271 real packages don't.

Scoring each by its probability of answering "no", instead of the word it printed, helps a lot (0.849 → 0.932 for gemma4). TabPFN still gets 0.975 without calling a language model at all.

Does thinking harder scare them off?

Ultramarine also ran the same 200 prompts through the same extraction and labelling: 400 generations at low reasoning, plus 200 at high reasoning as an ablation.

• Low reasoning: 3.2% ghosts (16 of 496 installs), against 15.0% for my three local models.
• High reasoning: 0.4% (1 of 241).
• Same 200 tasks: 8 produced a ghost only at low effort, 0 only at high.
• Cost: the whole hosted run, 1,034 calls, came to $0.141.

Thinking longer nearly emptied the graveyard. I measured that it happens, not why.

The guard

# simplified from phantom/guard.py
def verdict(name):
    if name in STDLIB:                      return OK("standard library")
    meta = registry.lookup(name)            # metadata only, cached
    if not meta.exists:                     return GHOST("404 on PyPI")
    if squat_rules(meta) or (p_ghost(name) >= TAU and young(meta)):
                                            return SUSPECT(reasons)
    return OK(meta)

The hook follows the Claude Code hooks contract: a PreToolUse hook gets the command as JSON on stdin, and exit code 2 blocks the call, with stderr as the reason Claude reads. That's why the agent in the GIF knew to check holidays instead of giving up.

It fails closed on a ghost and open on network errors, with a warning, because a guard that bricks your agent every time the Wi-Fi drops gets uninstalled by lunch. PHANTOM_FAIL_CLOSED=1 flips that. TabPFN only loads when it can change the outcome: offline, when PyPI can't be reached, or for a young package.

If my agent had run all 1,199 scripts:

• Online, the guard blocks all 212 ghost installs and stops 231 generations (19.3%). It wrongly stops 13 of 1,165 real-package installs (1.1%). All 13 are young packages flagged SUSPECT for a human to check; none was called a ghost.
• Offline, it catches 191 of 212 ghosts (90.1%) and flags 5.7% of real installs.

What went wrong

• My spec's anonymisation leaked. It said to publish ghosts as sha256(name)[:8]. Anyone who guesses a name, or reruns the same open models, can hash it and confirm it's on my list. Public IDs are now a keyed HMAC, and CI hashes every token in every public file with the secret key.
• My extractor invented three ghosts of its own. In a 25% pilot, a reply cut off mid-name, an import key that normalised onto a real package, and a dotted standard-library path all counted as ghosts. Fixing them moved the pilot rate from 11.5% to 10.8%. The unit tests had all passed, naturally.
• The thinking model never stopped thinking. Ollama's qwen3:4b is a thinking-only build. With thinking switched off, it still reasoned until the token cap, and 0 of 5 smoke-test replies reached a pip command. I swapped in the instruct build.
• My guard was slower than the problem. TabPFN does its work at prediction time, and the first hook took 13.0 s to block a ghost. Now it skips the model when PyPI has already said 404: 2.2 s.
• The hosted logprobs started with </think>. Reading P(no) at the first token scored one of the most popular packages on PyPI as probably fake. It's now read at the first token after the reasoning ends.
• The screen recorder recorded my browser. Capturing a screen region records whatever window is on top. Those frames were deleted, and the demo now runs Claude Code in a hidden pseudo-terminal and renders it to video, so the desktop is never involved.
• I nearly served my own secrets. The first local preview of the results site started in the repo root, so .env was reachable on localhost for about a minute. It now always serves web/ only.

Why Does Open Innovation Matter?

You can't measure a ghost you can't summon on demand. I ran the same models with the same seeds twice, from scratch, and got 98.7% of 1,199 generations back word for word. With a rented endpoint the model can change under you, and every rerun costs money. With open weights, the ghost count is a fact anyone can reproduce on a 4 GB laptop GPU.

The guard has to work with the lights off. People run local models exactly when they can't or won't use the cloud. A guard that has to phone home defeats the point. TabPFN's open weights run next to the models they're checking, and offline the guard still catches 90.1% of ghosts.

Open models need open tools. No vendor is going to publish its own model's ghost list. The code, the evaluation and a name-free version of the modelling table (dataset.csv, which reproduces the TabPFN results exactly) are all public.

The honest caveat: the hosted model invented far fewer packages than my small local ones: 3.2% against 15.0%, and 0.4% with high reasoning. Open doesn't mean better here. It means inspectable: I could measure the problem, build a fix that runs offline, and you can rerun all of it.

My Agent Session

I built this with Claude Code as a pair, under a CLAUDE.md of hard rules: never install or import a name a model produced, never register a package, keep unregistered ghost names in a gitignored folder, never invent a number or a reaction, and ask before spending money. The agent wrote the extractor test-first, ran the study and the evaluation, recorded the demo, and caught its own integration bugs by rerunning the whole pipeline. Three moments worth opening:

• It found the leak in my own spec. The sha256 IDs above were my idea. The agent showed they were confirmable, moved public IDs to a keyed HMAC, and added the leak test that now guards this post.
• It made rebuilding free by default. The evaluation stage would have quietly asked the paid model about any new name, because an API key was present. Paid calls now need an explicit --hosted-spend flag.
• The demo is the agent being stopped by its own guard. The GIF at the top is Claude Code reaching for a ghost and my hook saying no. Yes, an AI helped me catch AI making things up. I'm aware.

Every surprise from the build is in lessons.md, 35 of them.

AI disclosure: Claude Code helped build the project and draft this post. I reviewed and edited it.

The Payoff: some names refuse to be forgotten

Halloween is tonight, and Día de los Muertos starts tomorrow. It's about remembering names so they aren't lost.

My study found the opposite kind of name. One that was never alive, that no one ever wrote, and that a model on my laptop remembers perfectly anyway. I wiped every generation and asked again, half expecting a different story. 161 of the 162 ghosts came back with exactly the same names, and not one new ghost appeared.

Nobody outside the team has run the hook yet, and I won't dress up a guess as a reaction. So here's the offer: if you let an agent run pip install, add the hook for a day and tell me what it stopped. Those are the names worth remembering long enough to block.

Prize Categories

• Best Use of TabPFN: TabPFN-3.5 (open weights, run on a laptop CPU) is the ghost detector. It's the best offline method on every split (AUROC 0.975 / 0.978 / 0.948) and the best calibrated by a wide margin (ECE 0.015; the next best is 0.052). It's benchmarked against two rules, logistic regression, gradient boosting and two model self-checks on identical folds, with bootstrap CIs and paired differences. Its out-of-fold probabilities set the guard's threshold (90.1% recall), and a TabPFN-3.5-fast model ships inside the CLI, the hook, the Action and --offline mode. Gradient boosting wins precision at 90% recall, and I've shown where.
• Best Use of Ultramarine: ultramarine-1-flash is the hosted comparison arm: the same 200 prompts, extraction and labelling. 3.2% ghosts at low reasoning and 0.4% at high; on the same tasks, 8 ghosted only at low effort and 0 only at high. Its confidence-scored self-check is the strongest baseline in the study (AUROC 0.951, AUPRC 0.880, a tie with TabPFN on unseen names), yet its plain yes/no said 60 of 161 ghosts exist. 1,034 calls for $0.141, under a spending ledger with a hard cap.
• Best Use of Gemma: gemma4:e4b is one of the three summoned models, run locally through Ollama. It invented the fewest ghosts (8.1%, against 16.7% and 20.7%) and was the most reproducible (399 of 400 outputs identical on the rerun). As a self-checker it leans the other way: it called 143 of 271 real packages missing (0.932 AUROC scored by P(no), against TabPFN's 0.975).

Built by @t-rexbytes, @soumyadeepdey and @dronzer2code with Claude Code.