RefChecker

Validate reference accuracy in academic papers.
Catch citation errors, fabricated references, and metadata mismatches before they reach reviewers.

Quick Start • Features • Web UI • CLI • Hallucination Detection • The Paper • Deployment

_{Linux: .AppImage · .deb · all builds}

_{Native desktop builds powered by Tauri · Built and signed by GitHub Actions on every release tag.}

Overview

RefChecker verifies citations against Semantic Scholar, OpenAlex, CrossRef, DBLP, and ACL Anthology, and uses LLM-powered deep web search to flag likely fabricated references. When the LLM finds a more likely source than the first database match, RefChecker re-verifies the citation against the LLM-found metadata before deciding whether it is an error or a hallucination. It supports single papers, bulk batches, and automated scanning of entire OpenReview venues.

Use it three ways from one shared engine: a Web UI (also packaged as a native desktop app), a CLI for single papers and scripted runs, and an HTTP API. Start at Quick Start.

Built by Mark Russinovich with AI assistants (Cursor, GitHub Copilot, Claude Code). Watch the deep dive video.

📄 Bulk scanning, by example. RefChecker is the pipeline behind

Phantom References: Hallucinated Citations That Survive Peer Review at Top-Tier Conferences,

which used the same bulk scan to audit every accepted paper at ICLR, ICML,

NeurIPS, and USENIX Security — finding that in 2025 roughly one in twenty NeurIPS and

USENIX Security papers carried at least two likely hallucinated references, at about **$0.04

per paper. Read the summary →**

🆕 Recent updates

Release highlights have moved to docs/RELEASE-NOTES.md. For every build, see the full release list.

Contents

• The Paper: Phantom References
• Release Notes
• Quick Start
• Features
• Feature Matrix (Web / Desktop / CLI / API)
• Sample Output
• Install
• Web UI
• CLI
• Hallucination Detection
• AI-Generated Text Detection
• Bulk Checking
• OpenReview Integration
• Output & Reports
• Deployment

• Configuration
• Local Database
• Testing
• License

Quick Start

Web UI (Docker)

docker run -p 8000:8000 ghcr.io/markrussinovich/refchecker:latest

Open http://localhost:8000 in your browser.

Web UI (pip)

pip install academic-refchecker[llm,webui]
refchecker-webui

CLI (pip)

pip install academic-refchecker[llm]
academic-refchecker --paper 1706.03762
academic-refchecker --paper /path/to/paper.pdf

LLM extraction is generally more accurate, but PDFs can fall back to GROBID when no extraction LLM is configured. Deep hallucination checks require a hallucination-capable LLM provider: OpenAI, ChatGPT, Anthropic, Google, or Azure.

Tip: Set SEMANTIC_SCHOLAR_API_KEY for 1-2s per reference vs 5-10s without.

Features

| Category | What it does | |----------|-------------| | Input formats | ArXiv IDs/URLs, PDFs, LaTeX (.tex), BibTeX (.bib/.bbl), plain text | | Verification sources | Semantic Scholar, OpenAlex, CrossRef, DBLP, ACL Anthology | | LLM extraction | OpenAI, ChatGPT plan usage, Anthropic, Google, Azure, or local vLLM for parsing complex bibliographies | | Metadata checks | Titles, authors, years, venues, DOIs, ArXiv IDs, URLs | | Smart matching | Handles formatting variations (BERT vs B-ERT, pre-trained vs pretrained) | | Hallucination detection | Flags likely fabricated references using deterministic pre-filters, LLM deep web search, and metadata reverification when the LLM finds a better match | | AI-generated-text detection (opt-in) | Optionally analyzes the body text of each checked article for AI-generated-likelihood, returning a low/medium/high band plus advisory flagged passages. Three engines: a local calibrated model (offline, downloadable), an LLM judge (reuses your configured LLM), or an external API (Pangram/GPTZero). Advisory only — detection is unreliable on technical and non-native-English academic writing, so results are framed as a self-check and never as proof of misconduct. Enable under Settings → AI Detection. | | Bulk checking | Upload multiple files or a ZIP in the Web UI; use --paper-list or --openreview in the CLI | | OpenReview scanning | Fetch all accepted (or submitted) papers for a venue and scan them in one command | | Reports | JSON, JSONL, CSV, or text — with error details, corrections, and hallucination assessments | | Corrections | Auto-generates corrected BibTeX, plain-text, and bibitem entries for each error | | Visual analysis | 3D reference-library graph (Obsidian-style), real per-paper citation graph, and a native-feeling document viewer with zoom + in-document find | | Share & export | Self-contained HTML report, publish-to-web link (GitHub Gist), an animated video walkthrough, and RIS export for Zotero / EndNote / Mendeley | | Web UI | Real-time progress, history sidebar, batch tracking, split extraction/hallucination LLM settings, export (Markdown/text/BibTeX), dark mode | | Multi-user hosting | OAuth sign-in (Google, GitHub, Microsoft), per-user rate limiting, admin controls |

Feature Matrix (Web / Desktop / CLI / API)

RefChecker ships in four access methods that share one verification engine (ProgressRefChecker). The table below shows where each capability is available. The CLI column lists the exact flag — these match refchecker-webui check --help (CLI guide below). UI-interactive surfaces (in-app viewers, graphs, share video, author hovers) are web/desktop-only and are documented as such. For per-feature guides see docs/FEATURES.md; for multi-user setup see docs/MULTIUSER.md.

Legend: ✅ available · — not applicable to that surface · 🌐 needs a hosted/multi-user server.

| Capability | Web UI | Desktop (Tauri) | CLI | API | Notes | |---|:---:|:---:|:---:|:---:|---| | Reference verification (S2 / OpenAlex / CrossRef / DBLP / ACL) | ✅ | ✅ | ✅ | ✅ | Core engine; identical results across surfaces | | LLM extraction (Anthropic / OpenAI / ChatGPT / Google / Azure / vLLM) | ✅ | ✅ | ✅ --llm-provider | ✅ | --no-llm for regex/structural only | | Hallucination detection (deep web search) | ✅ | ✅ | ✅ --check-hallucinations | ✅ | Needs a web-search-capable provider; see Hallucination Detection | | Inline-citation numbering/ordering check | ✅ | ✅ | ✅ --check-citation-order | ✅ | Scheme-aware; abstains when unclear | | Retraction screening (OpenAlex) | ✅ | ✅ | ✅ --check-retractions | ✅ | Flags only references OpenAlex reports retracted | | Gap-finder / co-citation suggestions | ✅ | ✅ | ✅ --suggest-missing | ✅ | OpenAlex-resolved real works only | | Enrichment (counts · abstract · claim/TL;DR · funding · author metrics incl. ORCID · h-index) | ✅ | ✅ | ✅ on by default (--no-enrich) | ✅ | Mirrors the web/API default | | Add-to-reference-list (dedup + renumbered list + tracked PDF diff) | ✅ | ✅ | — | — | Interactive editing surface | | AI-generated-text detection (opt-in, advisory) | ✅ | ✅ | ✅ --ai-detection {local,api} + --ai-detection-consent | ✅ | Opt-in + consent required; never proof of misconduct | | Multi-detector compare + checkbox export (RAID-informed roster) | ✅ | ✅ | ✅ --detectors key1,key2 · --list-detectors | ✅ | Per-detector scores shown honestly; no synthetic ensemble; uninstalled ⇒ abstains | | Local databases for offline / faster verification | ✅ | ✅ | ✅ --database-dir / --s2-db / … | ✅ | Same resolver across surfaces | | Structured machine-readable output | ✅ | ✅ | ✅ --json | ✅ (JSON responses) | Progress to stderr, JSON to stdout | | Bulk / batch checking | ✅ | ✅ | ✅ (academic-refchecker --paper-list / --openreview) | ✅ | See Bulk Checking | | Native PDF viewers (find · in-PDF citation links · color coding · pinch-zoom) | ✅ | ✅ | — | — | Interactive UI surface (R02/R28/R42) | | Seen-library graphs (radial + Obsidian-style 3D) + per-paper citation graph | ✅ | ✅ | — | — | Interactive UI surface | | Similar papers + "Cites & Refs" + common-works view | ✅ | ✅ | — | — | Interactive UI surface | | Per-reference chat (full-text grounded, TL;DR fallback) + Summarize | ✅ | ✅ | — | — | Separate model selection per feature | | Share / export (HTML · Markdown · PDF · DOCX · RIS · video) | ✅ | ✅ | — | — | Interactive share surface; CLI uses --report-file/--report-format | | Author / journal hover cards (h-index · ORCID · guidelines) | ✅ | ✅ | — | — | Interactive UI surface | | Live token / $ telemetry per LLM flow (R47) | ✅ | ✅ | — | ✅ (per-request usage) | UI meter is web/desktop; usage is returned by the API | | Accounts · Teams · realtime shared-batch presence (R26/R27) | 🌐 | 🌐 | — | 🌐 | Opt-in multi-user mode; see Multi-User Server | | Support menu (email + open a GitHub issue) | ✅ | ✅ | — | — | In-app header menu |

Single-user vs multi-user. Web/Desktop/CLI all run **single-user/local by

default** — no login, no team, no presence. Accounts, Teams, and shared-batch

presence light up only when you enable multi-user mode (set REFCHECKER_MULTIUSER=true

and configure an OAuth provider, or use the in-app Accounts & Teams form

with hot-reload). The CLI is always single-user and never makes a team/collaboration

claim. Full setup: docs/MULTIUSER.md.

Sample Output

Web UI

A completed check — summary health, the 2×2 article-tools grid (retractions · gap-finder · citation-numbering · chat & summarize), and AI-generated-text detection with a per-page breakdown:

!RefChecker — completed check overview

Per-reference verification and enrichment — matched database, verified/DOI links, citation counts, and the Additional-Info bar (abstract · claim · topics · full link · add-to-library):

!RefChecker — reference verification detail

CLI — Startup banner

Running the CLI prints an environment + capabilities banner (colourised on a TTY, plain when piped). --help lists the full options and examples.

_{The banner prints to stderr (so machine-readable stdout like --report-format json stays clean). NO_COLOR=1 disables colour · FORCE_COLOR=1 forces it.}

📄 Processing: Attention Is All You Need
   URL: https://arxiv.org/abs/1706.03762
[1/45] Neural machine translation in linear time
       Nal Kalchbrenner et al. | 2017
       ⚠️  Warning: Year mismatch: cited '2017', actual '2016'
[2/45] Effective approaches to attention-based neural machine translation
       Minh-Thang Luong et al. | 2015
       ❌ Error: First author mismatch: cited 'Minh-Thang Luong', actual 'Thang Luong'
[3/45] Deep Residual Learning for Image Recognition
       Kaiming He et al. | 2016 | https://doi.org/10.1109/CVPR.2016.91
       ❌ Error: DOI mismatch: cited '10.1109/CVPR.2016.91', actual '10.1109/CVPR.2016.90'
============================================================
📋 SUMMARY
📚 Total references processed: 68
❌ Total errors: 55  ⚠️ Total warnings: 16  ❓ Unverified: 15

[5/7] Efficient Neural Network Pruning Using Iterative Sparse Retraining
      Shuang Li, Yifan Chen | 2019
      ❓ Could not verify
      🚩 Hallucination assessment: LIKELY
         A web search for the exact title and authors yields no results in any
         academic database. The paper does not appear in ICML 2019 proceedings,
         indicating it is probably fabricated.

Full CLI usage, flags, and examples are in the CLI section below.

Install

PyPI (recommended)

pip install academic-refchecker[llm,webui]  # Web UI + CLI + LLM providers
pip install academic-refchecker[llm]        # CLI + LLM providers; recommended for best extraction and hallucination checks
pip install academic-refchecker             # CLI only; PDFs can still fall back to GROBID when available

From Source (development)

git clone https://github.com/markrussinovich/refchecker.git && cd refchecker
python -m venv .venv && source .venv/bin/activate   # Windows: .venv\Scripts\activate
pip install -e ".[llm,webui]"
pip install -r requirements-dev.txt                  # pytest, playwright, etc.

Requirements: Python 3.11+. Node.js 20.19+ is only needed for Web UI frontend development.

Web UI

The Web UI provides real-time progress, check history, batch tracking, and one-click export of corrections.

The Summary Export menu offers the original bibliography, the full corrected bibliography, corrections only (entries with verified changes), and a side-by-side diff. Corrections use verifier-supplied fields even when no complete corrected citation was stored; unresolved findings remain visible in diff reports. Corrections-only files contain corrected citations, not verification reports or error details, and omit unchanged entries and ordinary metadata enrichment. Explicit issue fixes are included for older checks even when displayed metadata already contains the corrected value and no original-citation snapshot was saved. Unchanged fields retain the original citation values when available. Exports preserve existing BibTeX keys, keep corrected identifiers and their links consistent, and sort by the values in the selected bibliography mode. Author names prefer OpenAlex profile links when available. Google Scholar remains a secondary card link: a known profile when metadata provides one, otherwise a name search. Card statistics come from OpenAlex/Semantic Scholar, not Google Scholar. For PDFs, checked-paper authors are extracted from a recognizable title-page byline before consulting scholarly indexes, so unindexed papers can show author cards too. Affiliations, anonymous bylines, and ambiguous layouts are not treated as author names; unavailable scholarly profiles do not fabricate statistics.

LLM extraction is preferred, but PDF uploads and direct PDF URLs can fall back to GROBID. Hallucination checks use a separate hallucination LLM selection when one is configured; otherwise the UI falls back to the selected extraction LLM only if that provider supports web search. Local vLLM can be used for extraction, but hallucination checks require OpenAI, ChatGPT, Anthropic, Google, or Azure.

refchecker-webui                    # default: http://localhost:8000
refchecker-webui --port 9000        # custom port

Key features:

• Single check — paste an ArXiv URL/ID or upload a PDF/BibTeX/LaTeX file
• Bulk check — upload multiple files (up to 50) or a single ZIP archive; papers are grouped into a batch with a progress bar
• Bulk URL list — paste up to 50 URLs or ArXiv IDs (one per line) to check in a single batch
• Status dashboard — filterable badge counts for errors, warnings, unverified, and hallucinated references
• Reference cards — per-reference details with corrections, source links (Semantic Scholar, ArXiv, DOI), and hallucination assessment
• Export — download corrections as Markdown, plain text, or BibTeX
• History sidebar — browse and re-run previous checks; batches are grouped together
• Settings — separate extraction and hallucination LLM provider/model selection, API key management, Semantic Scholar key validation, local database directory, dark/light/system theme

Frontend Development

cd web-ui && npm install && npm start     # http://localhost:5173

Or run backend and frontend separately:

# Terminal 1 — Backend
python -m uvicorn backend.main:app --reload --port 8000
Terminal 2 — Frontend
cd web-ui && npm run dev

See web-ui/README.md for more.

CLI

# ArXiv (ID or URL)
academic-refchecker --paper 1706.03762
academic-refchecker --paper https://arxiv.org/abs/1706.03762
Local files (PDF, LaTeX, text, BibTeX)
academic-refchecker --paper paper.pdf
academic-refchecker --paper paper.tex
academic-refchecker --paper refs.bib
With LLM extraction (recommended for complex bibliographies)
academic-refchecker --paper paper.pdf --llm-provider anthropic
Save human-readable output
academic-refchecker --paper 1706.03762 --output-file errors.txt
Save structured report (JSON, JSONL, CSV, or text)
academic-refchecker --paper 1706.03762 --report-file report.json --report-format json
Bulk: check a list of papers
academic-refchecker --paper-list papers.txt --report-file report.json
OpenReview: fetch and scan an entire venue
academic-refchecker --openreview iclr2024 --report-file report.json
OpenReview: fetch the paper list only and save it to a custom path
academic-refchecker --openreview aistats2025 --openreview-list-only --openreview-output-file paper_lists/aistats2025.txt

All CLI Options

Input (choose one):
  --paper PAPER              ArXiv ID, URL, PDF, LaTeX, text, or BibTeX file
  --paper-list PATH          Newline-delimited file of paper specs (URLs, IDs, paths)
  --openreview VENUE         Fetch papers from a supported OpenReview venue (iclr, icml, aistats, uai, corl)
  --openreview-status MODE   accepted (default) or submitted
  --openreview-list-only     Fetch the OpenReview paper list and exit without scanning
  --openreview-output-file PATH
                            Custom path for the generated OpenReview paper list
LLM:
  --llm-provider PROVIDER    openai, chatgpt, anthropic, google, azure, or vllm
  --llm-model MODEL          Override the default model for the provider
  --llm-endpoint URL         Custom endpoint (e.g. local vLLM server)
  --llm-parallel-chunks      Enable parallel LLM chunk processing (default)
  --llm-no-parallel-chunks   Disable parallel LLM chunk processing
  --llm-max-chunk-workers N  Max workers for parallel LLM chunks (default: 4)
  --hallucination-provider PROVIDER
                            Separate provider for deep hallucination checks: openai, anthropic, google, or azure
  --hallucination-model MODEL
                            Override the hallucination-check model for the provider
  --hallucination-endpoint URL
                            Custom endpoint for the hallucination-check provider
Verification:
  --database-dir PATH        Directory containing local DBs: semantic_scholar.db, openalex.db, crossref.db, dblp.db, acl_anthology.db
  --s2-db PATH               Path to local Semantic Scholar database
  --openalex-db PATH         Path to local OpenAlex database
  --crossref-db PATH         Path to local CrossRef database
  --dblp-db PATH             Path to local DBLP database
  --acl-db PATH              Path to local ACL Anthology database
  --update-databases         Install/update configured local databases
  --openalex-since DATE      Only ingest OpenAlex partitions newer than YYYY-MM-DD during updates
  --openalex-min-year YEAR   Only ingest OpenAlex works published in YEAR or later during updates
  --db-path PATH             (Deprecated) alias for --s2-db
  --semantic-scholar-api-key KEY   Override SEMANTIC_SCHOLAR_API_KEY env var
  --disable-parallel         Run verification sequentially
  --max-workers N            Max parallel verification threads (default: 6)
Output:
  --output-file [PATH]       Human-readable output (default: reference_errors.txt)
  --report-file PATH         Structured report (includes hallucination assessments)
  --report-format FORMAT     json (default), jsonl, csv, or text
  --debug                    Verbose logging

refchecker-webui check — single-paper checker (web-parity flags)

The refchecker-webui command (installed with the [webui] extra) has two subcommands. With no subcommand it serves the Web UI / API (the historical behaviour); the check subcommand runs the same pipeline the web app uses (ProgressRefChecker) against a single paper from the terminal, exposing the web/API feature flags — hallucination check, inline-citation numbering/ordering, retraction screening, gap-finder suggestions, enrichment backfill, and opt-in AI-text detection. It reuses the real backend implementations (it never forks the verification, retraction, gap-finder, inline-citation, or AI-detection logic).

# Serve the Web UI / API (default — no subcommand needed)
refchecker-webui                     # http://localhost:8000
refchecker-webui serve --port 9000   # explicit subcommand form
Check a single paper from the terminal (examples match check --help)
refchecker-webui check --paper 2406.01234
refchecker-webui check --paper ./paper.pdf --json
refchecker-webui check --paper ./refs.bib --check-retractions --suggest-missing
refchecker-webui check --paper 2406.01234 --check-hallucinations \
    --llm-provider anthropic --llm-model claude-3-5-sonnet-latest
refchecker-webui check --paper ./paper.pdf --ai-detection api \
    --ai-detection-consent --ai-detection-key $PANGRAM_KEY
Multi-detector AI-text compare (only INSTALLED detectors run; rest abstain)
refchecker-webui check --list-detectors            # roster: installed vs. available
refchecker-webui check --paper ./paper.pdf \
    --ai-detection local --ai-detection-consent \
    --detectors desklib,e5-small-lora

Structured output (--json). A single JSON document is printed to stdout; all progress logging goes to stderr, so stdout stays machine-readable. The document always carries paper_title, paper_source, source_type, summary, and references, plus — only when you set the corresponding flag — citation_order (--check-citation-order), retractions (--check-retractions), suggestions (--suggest-missing), and ai_detection (--ai-detection).

Web/desktop-only — not on the CLI. The native in-app PDF viewers and in-PDF citation hyperlinks, the seen-library / similar-papers 3D graphs, the shareable per-check "video", and the author hover/pin profile cards are interactive UI surfaces, available only in the Web UI and the desktop (Tauri) build. The CLI makes no team / collaboration claim — it is always single-user/local.

Honesty notes (same as --help). No fabrication — every author / paper /

DOI / count comes from a real resolved source, and checks abstain rather than

emit a wrong badge. Cross-source enrichment backfill is on by default (pass

--no-enrich to opt out). AI-generated-text detection is **opt-in and advisory

only** (never proof of misconduct) — it requires --ai-detection plus an explicit

--ai-detection-consent flag.

Run refchecker-webui check --help for the full, authoritative flag list.

Hallucination Detection

RefChecker automatically evaluates suspicious references for potential fabrication using deterministic filters, LLM deep web search, and metadata reverification.

Stage 1 — Deterministic Pre-filter (no LLM needed)

References are flagged for deeper inspection when they exhibit:

• Unverified status — not found in Semantic Scholar, OpenAlex, CrossRef, DBLP, or ACL Anthology
• Author overlap below 60% — fewer than 60% of cited authors match any known paper (applies to references with 3+ authors)
• Identifier conflicts — DOI or ArXiv ID resolves to a different paper
• URL verification failure — cited URL is broken or points to a different paper

Stage 2 — LLM Deep Web Search

Flagged references are sent to the configured hallucination LLM for a mandatory web search. The LLM must look for a dedicated page for the cited work, not just a citation in another paper's reference list. It returns a short verdict plus the best link it found and any found title, authors, and year.

Supported hallucination-check providers are OpenAI, ChatGPT, Anthropic, Google, and Azure. The CLI can use the extraction provider when it is hallucination-capable, or you can pass --hallucination-provider / --hallucination-model to use a different model. The Web UI exposes the same split as separate extraction and hallucination selectors in Settings.

Stage 3 — Reverification Against LLM-Found Metadata

When the LLM says the reference is probably real (UNLIKELY) and provides found metadata, RefChecker re-runs its normal title, author, and year comparison against that LLM-found metadata. This catches cases where a database lookup matched the wrong edition, version, or similarly titled work. If the cited title/authors/year match the LLM-found source, stale unverified or wrong-match errors can be cleared and the LLM-found URL is added as an llm_verified source. If substantive mismatches remain, the reference stays an error rather than being blindly upgraded.

If the LLM cannot find an exact source, or finds only a similar paper with different authors or identifiers, the reference remains suspicious and can be marked as a likely hallucination.

Each reference receives a verdict:

| Verdict | Meaning | |---------|---------| | 🚩 LIKELY | Probably fabricated — no exact source was found, or the found source conflicts substantially with the citation | | ❓ UNCERTAIN | Inconclusive — may exist but could not be confirmed | | ✅ UNLIKELY | Probably real — found on a dedicated page with matching title/authors, then rechecked against the cited metadata |

Hallucination assessments appear inline in CLI output, in Web UI reference cards, and in structured reports (JSON/JSONL/CSV) via the hallucination_assessment field.

AI-Generated Text Detection

Opt-in and advisory only. AI-text detection is unreliable on academic, technical, and non-native-English writing, and on human text polished with AI. RefChecker frames every result as a low/medium/high likelihood band with a permanent disclaimer — never a binary verdict or proof of misconduct, and never a basis for an accusation, grade, or decision. Below ~300 words, or on equation/code/citation-heavy passages, it abstains (inconclusive).

When enabled (Settings → AI Detection), each checked article's body text is analyzed for AI-generated likelihood, in single and batch modes. Results include a confidence donut, AI / Mixed / Human probability pills, a page-by-page breakdown, Top AI / Top Human sentence lists, and advisory flagged passages you can open highlighted in the document — alongside the engine/model used and a permanent disclaimer.

Run mode. Settings → Run mode controls what a check actually runs:

| Mode | Reference checking | AI detection | |------|:---:|:---:| | References only (turn AI detection off) | ✅ | — | | Reference check + AI detection | ✅ | ✅ (runs in parallel) | | AI detection only | — (extraction & verification skipped) | ✅ |

Detection engines (pick one in Settings)

| Engine | What it is | Cost | Notes | |--------|-----------|------|-------| | Local model (default) | desklib/ai-text-detector (DeBERTa-v3, MIT) run offline via Transformers + PyTorch | Free | One-time model and runtime download, both installable from Settings → AI Detection; calibrated, reproducible; no data leaves your machine | | LLM judge | Reuses your configured LLM provider (OpenAI/ChatGPT/Anthropic/Google/Azure) with an anti-false-positive rubric | LLM tokens | Uncalibrated, so it is hard-capped at "medium" — it can never raise a standalone "high" | | External API | Pangram or GPTZero | Per-word $ | Requires an API key and explicit consent (your manuscript text is sent to a third party) |

The local model needs an inference runtime (torch + transformers) that is not bundled, to keep the desktop app small. Click Install runtime under Settings → AI Detection to fetch it on demand (installed into the app's data folder and used without a restart), or install it yourself with pip install torch transformers. The LLM-judge and External-API engines need no runtime.

Multi-detector compare (RAID-leaderboard-informed roster)

Beyond the default desklib model you can install one or more open-source detectors and run them side-by-side. Each detector's verdict is shown on its own — there is no synthetic "ensemble truth"; disagreement between detectors is surfaced as signal. Detectors are installed on demand (never bundled), and an uninstalled detector abstains — it never reports a number. Heavy Tier-2 metric/zero-shot detectors are listed for honesty but are opt-in and not runnable in this build (real size / RAM warnings are shown so you understand why).

| Key | Model | Arch | Tier | Size | License | Note | |---|---|---|:---:|---|---|---| | desklib (default) | desklib/ai-text-detector-v1.01 | DeBERTa-v3-large | 1 | ~870 MB | MIT | RAID leaderboard leader among open models | | superannotate | SuperAnnotate/ai-detector | RoBERTa-Large | 1 | ~1.4 GB | research/eval | #1 open-source on RAID (late 2024) | | e5-small-lora | MayZhou/e5-small-lora-ai-generated-detector | e5-small + LoRA | 1 | ~130 MB | MIT | tiny/fast/CPU-friendly (~89% acc) | | mage | yaful/MAGE | Longformer | 1 | ~570 MB | Apache-2.0 | "Detection in the wild" (ACL 2024) | | binoculars | paired causal LMs | metric zero-shot | 2 (heavy) | ~14 GB | see models | best at low FPR; opt-in, not runnable here | | fast-detectgpt | GPT-Neo-2.7B scorer | metric zero-shot | 2 (heavy) | ~11 GB | see models | 340× faster DetectGPT; opt-in, not runnable here | | radar | TrustSafeAI/RADAR-Vicuna-7B | adversarial classifier | 2 (heavy) | ~13 GB | see card | robust to paraphrase; opt-in, not runnable here |

Roster informed by the RAID benchmark (ACL 2024) (leaderboard · paper). In Settings → AI Detection you install/remove each detector (real size + license shown), run any subset, compare per-detector scores + per-sentence agreement, and checkbox-export only the detectors you select (MD / CSV / JSON). From the CLI:

refchecker-webui check --list-detectors            # roster: installed vs. available
refchecker-webui check --paper ./paper.pdf \
    --ai-detection local --ai-detection-consent \
    --detectors desklib,e5-small-lora              # only INSTALLED run; rest abstain

Usage & cost tracking

AI-detection work is metered in the same per-check token/$ badge under an "AI-generated-text detection" flow: the local model records the processed word count at $0; the API backends record words sent plus an estimated dollar cost; the LLM-judge records real input/output tokens and their cost.

Graph 2nd-degree expansion

In the Graph tab, the 2nd-degree expansion has a "Refs only" vs "+ AI-gen" toggle. With "+ AI-gen", each expanded article also gets an AI-likelihood ring (red = high, amber = medium), estimated locally from its abstract (free, offline). Abstracts are short, so most come back inconclusive — this is an advisory signal, never a full-text analysis.

Sources & credits

The detection engines build on these open-source projects and services:

• desklib/ai-text-detector-v1.01 — DeBERTa-v3 detector (MIT), the bundled local model
• harshaneel/humanize — the ai-check forensic rubric (MIT) adapted for the LLM-judge prompt
• distil-labs/distil-ai-slop-detector — the "small quantized classifier in-app" concept (Apache-2.0)
• Zero-shot research: Binoculars and Fast-DetectGPT
• API services: Pangram and GPTZero

Bulk Checking

Web UI

Upload multiple files or a ZIP archive to check up to 50 papers in a single batch. Alternatively, paste a list of URLs or ArXiv IDs (one per line). Batches track progress per paper and appear as a group in the history sidebar.

Supported file types: PDF, TXT, TEX, BIB, BBL, ZIP.

CLI

Create a text file with one paper per line (ArXiv IDs, URLs, or local file paths):

1706.03762
https://openreview.net/pdf?id=ZG3RaNIsO8
paper/local_sample.bib
/path/to/paper.pdf

Then run:

academic-refchecker --paper-list papers.txt --report-file bulk_report.json

The report includes per-paper rollups and a cross-paper summary with flagged reference counts.

OpenReview Integration

Scan all accepted (or submitted) papers for an OpenReview venue in one command:

# Scan accepted papers
academic-refchecker --openreview iclr2024 --report-file report.json
Scan all public submissions instead
academic-refchecker --openreview iclr2024 --openreview-status submitted --report-file report.json

Supported venues: ICLR, ICML, AISTATS, UAI, and CoRL.

Use shorthands like iclr2024, icml2025, aistats2025, uai2025, or corl2025.

The command fetches the paper list from OpenReview, writes it to output/openreview__.txt by default, and then runs a bulk scan. Use --openreview-list-only to generate the list without running verification, and --openreview-output-file to choose the output path. The structured report includes per-paper rollups with flagged record counts and error-type distributions, making it easy to triage an entire conference for citation problems.

Output & Reports

Result Types

| Type | Description | Examples | |------|-------------|----------| | ❌ Error | Critical issues needing correction | Author/title/DOI mismatches, incorrect ArXiv IDs | | ⚠️ Warning | Minor issues to review | Year differences, venue variations | | ℹ️ Suggestion | Recommended improvements | Add missing ArXiv/DOI URLs | | ❓ Unverified | Could not verify against any source | Rare publications, preprints | | 🚩 Hallucination | Likely fabricated reference | Unverifiable with rich metadata, identifier conflicts |

Structured Reports

Write machine-readable reports with --report-file and --report-format:

academic-refchecker --paper 1706.03762 --report-file report.json --report-format json

{
  "generated_at": "2026-03-15T19:50:52Z",
  "summary": {
    "total_papers_processed": 1,
    "total_references_processed": 7,
    "total_errors_found": 2,
    "total_warnings_found": 2,
    "total_unverified_refs": 4,
    "flagged_records": 3,
    "flagged_papers": 1
  },
  "papers": [
    {
      "source_paper_id": "local_hallucination_7ref_sample",
      "source_title": "Hallucination 7Ref Sample",
      "total_records": 6,
      "flagged_records": 3,
      "max_flag_level": "high",
      "error_type_counts": { "unverified": 3, "multiple": 2, "year (v1 vs v2 update)": 1 },
      "reason_counts": { "unverified": 3, "web_search_not_found": 3 }
    }
  ],
  "records": [
    {
      "ref_title": "Deep Residual Learning for Image Recognition",
      "ref_authors_cited": "Jian He, Xiangyu Zhang, Shaoqing Ren, Jian Sun",
      "ref_authors_correct": "Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun",
      "error_type": "multiple",
      "error_details": "- First author mismatch ...\n- Year mismatch ...",
      "ref_corrected_bibtex": "@inproceedings{he2016resnet, ... year = {2015} ...}",
      "hallucination_assessment": { "verdict": "UNLIKELY", "explanation": "..." }
    }
  ]
}

❌ Error: First author mismatch: cited 'Jian He', actual 'Kaiming He'
❌ Error: DOI mismatch: cited '10.5555/3295222.3295349', actual '10.48550/arXiv.1706.03762'
⚠️ Warning: Year mismatch: cited '2019', actual '2018'
ℹ️ Suggestion: Add ArXiv URL https://arxiv.org/abs/1706.03762
❓ Could not verify: Llama guard (M. A. Research, 2024)
🚩 Hallucination assessment: LIKELY — no matching paper found in academic databases

Each report record includes the original reference, error details, corrected metadata (BibTeX, plain text, bibitem), verified URLs, and hallucination assessment when applicable.

Deployment

Docker

Pre-built multi-architecture images are published to GitHub Container Registry on every release.

# Quick start
docker run -p 8000:8000 ghcr.io/markrussinovich/refchecker:latest
With LLM API key (recommended)
docker run -p 8000:8000 -e ANTHROPIC_API_KEY=your_key ghcr.io/markrussinovich/refchecker:latest
Persistent data
docker run -p 8000:8000 \
  -e ANTHROPIC_API_KEY=your_key \
  -v refchecker-data:/app/data \
  ghcr.io/markrussinovich/refchecker:latest

Other LLM providers:

docker run -p 8000:8000 -e OPENAI_API_KEY=your_key ghcr.io/markrussinovich/refchecker:latest
docker run -p 8000:8000 -e GOOGLE_API_KEY=your_key ghcr.io/markrussinovich/refchecker:latest

Docker Compose

git clone https://github.com/markrussinovich/refchecker.git && cd refchecker
cp .env.example .env   # Add your API keys
docker compose up -d

docker compose logs -f    # View logs
docker compose down       # Stop
docker compose pull       # Update to latest

| Tag | Description | Arch | Size | |-----|-------------|------|------| | latest | Latest stable release | amd64, arm64 | ~800MB | | X.Y.Z | Specific version (e.g., 2.0.18) | amd64, arm64 | ~800MB |

Multi-User Server (OAuth)

By default, RefChecker runs in single-user mode — no login required, and every request runs as a built-in local admin. Multi-user mode is opt-in: it turns on only when you both set REFCHECKER_MULTIUSER=true and configure at least one OAuth provider's client ID and secret (Google, GitHub, or Microsoft). Setting the flag alone — with no provider credentials — leaves the app in single-user mode and shows no login screen. Once at least one provider is configured, the Web UI gates behind a login page that renders a sign-in button only for the providers the server reports at /api/auth/providers, and every API route requires a valid session.

If the server has LLM provider environment variables such as ANTHROPIC_API_KEY, OPENAI_API_KEY, GOOGLE_API_KEY, or AZURE_OPENAI_API_KEY, the Web UI exposes those providers as selectable server-environment configs without revealing the secret. Users can still enter their own keys to override the server key for their browser session; user-entered keys are stored in the browser's localStorage and sent per-request — never stored on the server.

1. Generate a JWT Secret Key

python -c "import secrets; print(secrets.token_hex(32))"

2. Register an OAuth Application

Configure at least one provider:

| Provider | Registration URL | Callback URL | |----------|-----------------|--------------| | Google | Google Cloud Console | https:///api/auth/callback/google | | GitHub | GitHub Developer Settings | https:///api/auth/callback/github | | Microsoft | Azure App Registrations | https:///api/auth/callback/microsoft |

3. Configure Environment Variables

cp .env.example .env

REFCHECKER_MULTIUSER=true
JWT_SECRET_KEY=<output from step 1>
SITE_URL=https://<your-domain>
HTTPS_ONLY=true
At least one OAuth provider — only providers whose ID and secret are set
appear as login buttons. Microsoft uses the MS_* prefix.
GOOGLE_CLIENT_ID=...
GOOGLE_CLIENT_SECRET=...
GITHUB_CLIENT_ID=...
GITHUB_CLIENT_SECRET=...
MS_CLIENT_ID=...
MS_CLIENT_SECRET=...
Optional — by default the callback URL is derived from SITE_URL as
<SITE_URL>/api/auth/callback/{google,github,microsoft}. Override per provider
only if you registered a different redirect URI:
GOOGLE_REDIRECT_URI=https://<your-domain>/api/auth/callback/google
GITHUB_REDIRECT_URI=https://<your-domain>/api/auth/callback/github
MS_REDIRECT_URI=https://<your-domain>/api/auth/callback/microsoft
Optional
REFCHECKER_ADMINS=github:you  # comma-separated; first sign-in is auto-admin
MAX_CHECKS_PER_USER=3         # max concurrent checks per user (default: 3)

4. Launch

docker compose up -d

Or without Docker:

pip install "academic-refchecker[llm,webui]"
REFCHECKER_MULTIUSER=true JWT_SECRET_KEY=<secret> GOOGLE_CLIENT_ID=... GOOGLE_CLIENT_SECRET=... \
  refchecker-webui --port 8000

Verify:

curl http://localhost:8000/api/auth/providers
{"providers":["google","github"]}

Notes:

• The first user to sign in is automatically admin. Add more via REFCHECKER_ADMINS.
• Each user may run up to MAX_CHECKS_PER_USER concurrent checks (default 3). The 4th returns HTTP 429.
• The CLI is unaffected — academic-refchecker works without any auth configuration.
• Place the server behind a TLS-terminating reverse proxy (nginx, Caddy) for HTTPS.

Deploy to Render

RefChecker includes a render.yaml Blueprint for one-click deployment to Render:

1. Fork this repo (or connect your own copy).
2. On Render, click New + → Blueprint → select the repo.
3. Render reads render.yaml and creates the service with a persistent disk.
4. Set environment variables in the Render dashboard (Environment tab):

1. Register each provider's callback URL as https:///api/auth/callback/{google,github,microsoft}.

Note: The persistent disk at /data stores the SQLite database and uploaded files, so data survives redeployments. For other PaaS hosts (Railway, Fly.io), the same Docker image works — set PORT, REFCHECKER_DATA_DIR, and the auth env vars.

Local Semantic Scholar database on a server

Reference verification is far faster against a local Semantic Scholar DB than against the remote S2 API. When REFCHECKER_DB_PATH (or REFCHECKER_DATABASE_DIRECTORY) points at a location that does not contain the DB yet, the server starts a full bootstrap download in the background and then keeps it current with incremental refreshes. Until that first build finishes, checks fall back to the remote S2 API and are correspondingly slower.

| Variable | Default | Purpose | |----------|---------|---------| | REFCHECKER_DB_PATH | (unset) | Path to semantic_scholar.db (or a directory containing local DBs). Setting it opts the deployment into local lookups. | | SEMANTIC_SCHOLAR_API_KEY | (unset) | Required for the S2 datasets API used to build/refresh the local DB. | | REFCHECKER_DB_REFRESH_INTERVAL_HOURS | 24 | How often to refresh local DBs. 0 refreshes once per process start only. | | REFCHECKER_DB_AUTO_BOOTSTRAP | true | Set false to never auto-download a missing DB (admin UI download still works). | | REFCHECKER_S2_MIN_FREE_GB | 95 | Refuse to start a bootstrap download without at least this much free disk. | | REFCHECKER_S2_KEEP_ARCHIVES | false | Keep the downloaded .gz dataset shards after they are ingested (needs tens of GB extra). |

Size the disk for the database plus real working room (~150 GB for the ~90 GB slim DB). The downloader ingests one compressed shard at a time and deletes it, so the full compressed dataset never has to sit on the disk alongside the database; it also stops before filling the volume, because SQLite fails hard (and can leave the DB unusable) on a full disk. Budget well beyond the database size: incremental updates rewrite rows in place, so a diff needs free space to land before the superseded pages are released. A volume sized to just fit the database will refuse to apply updates and leave the snapshot frozen.

Admins can confirm what a deployment is actually using at any time:

curl -s https://<your-url>/api/databases/status -H "Cookie: <admin session>"
{"using_local_s2": true, "databases": [{"database": "s2", "exists": true,
  "snapshot": "2026-01-20", "size_bytes": 91234567890, "last_refresh": {...}}], ...}

using_local_s2: false means every reference lookup is going to the remote API — check that the disk has room, that SEMANTIC_SCHOLAR_API_KEY is set, and read last_refresh.detail for the failure reason.

A build that is killed mid-download (deploy, restart, OOM) strands a multi-GB staging directory on the data disk; enough of those fill the disk and break SQLite. The server sweeps those orphaned directories on startup, in the background so a slow disk can't delay the health check.

Configuration

LLM Providers

LLM-powered extraction improves accuracy with complex bibliographies. Hallucination detection is configured separately so you can use one model for extraction and another, web-search-capable model for deep hallucination checks. Claude Sonnet 4 performs best for extraction; GPT-4o may hallucinate DOIs.

| Provider | Env Variable | Example Model | |----------|--------------|---------------| | Anthropic | ANTHROPIC_API_KEY | claude-sonnet-4-6 | | OpenAI | OPENAI_API_KEY | gpt-4.1 | | ChatGPT | Browser sign-in (no API key) | Account model catalog | | Google | GOOGLE_API_KEY | gemini-3.1-flash-lite-preview | | Azure | AZURE_OPENAI_API_KEY | gpt-4.1 | | vLLM | (local) | meta-llama/Llama-3.3-70B-Instruct |

When running the Web UI, provider keys present in the server environment are added automatically

... (README truncated for length)