Argus

[!IMPORTANT]

Argus has many tunable command-line options. Please read Settings in full.

Updates

Overview

To my knowledge, Argus is the most advanced open-source robot learning annotation harness (as of 2026).

It is the only harness that combines native multi-format ingestion (LeRobot's annotation pipeline, for example, accepts only LeRobot), joint visual/sensor reasoning, dense annotations of actions, subgoals, outcomes, operator mistakes, recoveries and object-state changes, and execution and outcome scores for each episode, with execution scores for each subgoal. It reads multi-view RGB alongside state/action trajectories, depth, tactile and other recorded fields, and keeps their timing and measurement semantics. For tactile data, and for depth with --inspection all, the VLM can inspect additional samples or image regions to support a finding, and the viewer and JSON exports keep each finding's evidence, confidence and limitations.

Each request covers one recording, or one part of a long one. Argus samples a frame every 1.5s on teleop, 1s on UMI and 0.5s on human ego, plus the edges of still spans and the first and last frames, and tiles them into JPEG grids with one row per camera and one timestamped column per instant, four instants to a grid (six with one camera). Cells are 224 px wide on teleop, 320 px on UMI and 256 px on human ego. The first and last instants follow the grids at larger size, and the text adds the instruction as a claim to check, the recorded motion between instants and the sensor readings. A fixed prompt comes first, with rules for what counts as a data issue on each rig and variants for an instruction, timed subtasks and recorded state; human ego has its own output schema.

| Input support | Details | |---|---| | Recording types | Teleop, UMI and human ego | | Native formats | LeRobot v2.0/v2.1/v3.0, MCAP, ROS 2 bags stored as SQLite (.db3, read by the MCAP reader), HDF5, video files and folders | | Archives | ZIP and TAR, plain or compressed with gzip, bzip2 or xz | | Sidecar metadata | JSON, JSON Lines, CSV, TSV, text and Markdown notes, and LeRobot's Parquet and JSON Lines annotation tables | | Public dataset adapters | Ten, each downloading and preparing a public dataset |

| Available evidence | How it contributes | |---|---| | RGB | Timestamped multi-view frames provide visual context for task execution, object interactions, progress, failures and recoveries. | | Tactile, inspected by default | Pressure, normal/shear force, contact arrays and optical tactile images support analysis of grip, contact distribution and surface motion. The model compares whole arrays, single cells and regions with video and gripper state, and reads optical tactile images at the moments they change most. Array shape alone does not establish anatomy or force, and response changes alone do not establish slip. | | Depth, inspected with --inspection all | Metric depth supports camera-relative distance and separation when units, calibration and temporal alignment are known. Relative depth remains relative. | | Proprioception and actions | Joint states, end-effector poses, gripper states and action trajectories provide motion context. Training targets keep their declared semantics and are not assumed to be executed commands. | | Other episode fields | The prompt gives each force/torque, numeric, event flag, waveform or structured metadata channel as its range over the recording, plus a reading at each sampled instant when the sampling rate resolves the channel, and otherwise its window means or extremes. --inspection all also inspects a recording whose only extra fields are these. | | Task context | The prompt presents instructions, annotations, calibration and the uploader's own metadata fields as claims, and the VLM checks each one against the recording. The statistics, storage offsets and dataset totals LeRobot writes into its metadata stay in the episode's context.json and out of the prompt. |

Install and review

You need Python 3.11, uv, and ffmpeg/ffprobe 6.1 or newer. Board clips keep each frame's source timestamp with -enc_time_base demux, which older ffmpeg refuses, so python -m review, board clips and board static media stop at the start with a message that names the version they found.

git clone https://github.com/Pantheon-Industries-Inc/argus.git
cd argus
uv sync --frozen

Paid annotation requires OPENROUTER_API_KEYS (comma-separated keys) or OPENAI_API_KEY in your shell or secret manager. OpenRouter requests carry an app attribution header (OPENROUTER_APP_HEADERS). Gated Hugging Face datasets require accepting their terms and setting HF_TOKEN.

Annotate a dataset under a spending cap, then open its board. Add --free to build the base requests without model calls.

uv run python -m review --data path/to/dataset --rig teleop_arms \
  --dataset-note "Two arms arrange blocks on a tray." --out data/review/mine --cap 20
uv run python -m board serve --board data/review/mine --clips data/review/mine/clips

--data accepts a folder, file, archive or HTTP(S) URL. Set --rig to teleop_arms, handheld_gripper or ego_head for teleop, UMI or human ego, respectively. Argus detects the format and every stream a recording carries, and works from the video alone when a recording has nothing else.

We strongly recommend defisheyeing fisheye footage before labelling it. The model sees each sampled frame downscaled to one cell of a frame grid, a few hundred pixels wide, and a raw fisheye frame spends much of that cell on its black border and compressed edges.

Settings

| Setting | Default | Options | |---|---|---| | --field-dictionary | off |
Turn on for recordings with unusual sensors. on sends one text request per upload (at most $3, no frames) that asks a model what each recorded field beyond the cameras means, such as which signal is a touch sensor. Turn it on for tactile, audio, force or other custom streams whose names do not say what they are. Ordinary teleop, UMI and human ego recordings label well with it off, and fields are then labelled as recorded | | --model | openai/gpt-6-astra | openai/gpt-6.1-sol (Sol 6.1), or google/gemini-4-argon (Gemini 4 Argon) with an OpenRouter key, priced from OpenRouter's published rates (label/pricing.py) | | --reasoning | Medium for Astra, high for every other model | none, minimal, low, medium, high or xhigh | | --sample-fps | The rig's period in the Overview; Sol 6.1 samples every 1s on every rig, human ego included, and refuses sparser sampling | 2 catches subsecond fumbles and rapid recoveries, at the cost of more images and, on teleop and UMI, shorter request windows | | --inspection | auto, for recordings with a touch sensor or with fields the prompt does not carry | all also inspects depth, force/torque, arm signals, actions and source metadata, for runs whose findings need exact values; off makes no evidence selection call, though a tactile array still gets its separate regional pass | | --resolution-route | off, every teleop episode at the narrow cells (224 px) with contact detail views and no routing call | on asks a small model, from the episode's task text alone, whether judging it needs fine detail such as lettering or which side of an object is up, and sends those episodes at 448 px | | --service-tier | Unset, the provider's default tier | flex sends the same request at half the price with lower priority, and the provider can reject it when it has no capacity. It is independent of --model and --reasoning, the cap reserves flex rates, and each label records the tier that served it, with counts in .service_tier/counts.json in the labels folder | | --flex-fallback | off, and after about 4 minutes of retries a rejected unit stays unlabelled and unpaid, and a resume sends it again | standard sends it once at the standard tier, at twice the price, and the cap reserves that rate for every request. A flex request that timed out is never sent again at standard, since it may still be billed | | --allow-unverified-fees | Off, and no request goes to a model whose published search fee could apply, since no cap can limit that fee | On sends those requests with a printed warning. Normal runs do not need it, because a request that sends no search option cannot be charged for one, which covers OpenAI, Claude and Gemini models. On or off, a run through OpenRouter stops at the first reply whose usage counts a search | | --cap | $20 | The most labelling may spend, counting every call and outstanding reservation | | --max-minutes | All footage | Reads at most this many minutes of footage | | --dataset | The name of --data | The name shown on the board and told to the model | | --dataset-note | The source's own collection note, if any | A description of the recordings given to the model as context | | --max-tokens | 64,000 | The output limit of each labelling request; a smaller limit reserves less against the cap | | --timeout | 6060s | Idle limit for a labelling request's connection, at least 6060s (Cloudflare's 6000s idle limit plus a minute) | | --concurrency | All of them | The most labelling requests in flight at once. Argus sends every request at once (label/inflight.py), except that the first episode of each prompt variant goes alone, so the others find the shared instructions in the provider's prompt cache (cache_warmup). A request holds its encoded frames until the reply arrives, at most about 70 MB, so set a number only on a machine short of memory. | | --jobs | 2 | Decode processes per preparation step | | --grouping | Inferred from file names | JSON such as {"cam_folder": "cameras"} or {"take_folder": "takes"}, for folders whose file names cannot tell separate takes from cameras of one take | | --free | Off | Builds the base requests and stops before any model call | | --board-clips | on | off decodes each camera and writes no viewing copy | | --force | Off | Relabels every selected episode with new paid calls, keeping earlier replies and charges | | --retry-received EPISODE | None | Sends that episode's request again after an unusable reply the model wrote, or after a provider failure that has used its two automatic resends; repeat for each episode |

Rerunning a review with the same --out folder resumes it and reuses every output whose inputs are unchanged, matched by content hash (label/content_identity.py). --free previews only the base requests, without field interpretation, routing, inspection or sensor cards.

[!NOTE]

python -m review does all post-processing itself once the labelling requests return, and nothing else needs to be run. It stitches long recordings, merges data issues that several parts reported, and builds the review board. Before publishing a result, it removes the phrases the model writes about the labelling itself, such as "in this supplied window" (label/wording.py), and drops a sentence only when every word in it is about the labelling.

Labels

Below is part of the label of MolmoAct2 episode 017777 (configs/examples/example_teleop_arms.json), two of its 62 timeline steps and two of its 7 key events. The instruction asks for the scanned items to end in the basket, and the operator leaves them on the table. A subgoal is a key event of kind subgoal_complete.

{
  "completion": {
    "task_completed": "failure",
    "success_predicate": "Each of the four packaged items has been picked up and scanned, and all four scanned items are in the basket.",
    "reason": "The four items are scanned, but the demonstration deliberately empties the basket and leaves the scanned items on the right-hand tabletop instead."
  },
  "timeline": [
    {"start_s": 31.5, "end_s": 34.5, "arm": "right", "action": "rotate and present packet to scanner", "object": "green Kleenex tissue packet", "destination": "barcode scanner", "spatial_relation": "over scanning window", "contribution": "advancing", "progress": 0.3},
    {"start_s": 35.5, "end_s": 36.5, "arm": "right", "action": "set down and release", "object": "green Kleenex tissue packet", "destination": "far-right table", "spatial_relation": "beside orange packet", "contribution": "advancing", "progress": 0.3, "notes": "Release inferred between 36 and 37.5 s; scanner window glows cyan."}
  ],
  "key_events": [
    {"t_s": 22.5, "label": "All four items staged outside the now-empty basket", "kind": "subgoal_complete", "outcome": "success"},
    {"t_s": 72.0, "label": "All scanned items left on the right-hand table, not in the basket", "kind": "terminal_state", "outcome": "success"}
  ]
}

Task verdicts

Argus annotates every recording as the list of tasks it observes, and judges each task against what the dataset claims about it. Instructions and timed subtasks come from the dataset's task fields, and a LeRobot text column counts only when its name marks it as the task, such as language_instruction. A task text that spans the whole recording is never a subtask, and becomes the recording's instruction when it has none.

A teleop or UMI recording with an instruction is judged as a whole against it, and work the instruction does not ask for never raises that verdict. With timed subtasks, each task is judged against the subtask it covers. With neither, a recording of one task takes that task's outcome, and a recording of several reports each task separately.

A human ego clip of one task takes that task's outcome unless the dataset provides timed subtasks, and a clip of several tasks gets no overall verdict. When the annotation gives a goal, the verdict is whether the clip fully reached it. Without a goal, a task cut off by the end of the clip is named after the work the clip shows.

Long recordings

Argus cuts a recording too long for one request (450s at default sampling) at still moments, labels each part and stitches the parts back onto the recording's timeline. Neighbouring parts overlap by 15s, and the stitch joins a task or step to its counterpart from the other side when the two share at least half of the shorter one's seconds in the overlap, however differently the requests worded them. Labels made without the overlap have their tasks joined by a small seam check (python -m review seam).

A task still under way at a cut that cannot be joined gets the outcome unclear, and an instructed teleop or UMI recording labelled in parts gets an unclear overall verdict, since no part sees both its start and its end. A small text-only request merges data issues that several parts report (label/issue_groups.py). It spends from what --cap leaves after labelling, within its own $5 limit, and when that runs out the issues stay apart.

Data issue severity

Some severities are fixed minimums, independent of training impact. On handheld_gripper, the prompt rates the operator using their own phone, a gripper set down or taken off while the work goes on, and visible bare-hand work at least medium. It rates any face in any camera medium under face_visible, with one timed issue per stretch.

On every rig, the parser (idle_issues_low) rates an idle stretch low and keeps the model's own rating in severity_as_labelled. An idle item that describes other activity, such as phone use, another person handling the objects, bare-hand work or a gripper set down, keeps the model's severity, as does a frozen or black camera.

Results and further use

[!IMPORTANT]

The board is the canonical dashboard for Argus annotations, QC results and robot-data episodes. Serve the data with board serve or board static site, and extend the viewer where a view is missing rather than writing a new page.

Live and static boards share one viewer, which overlays behavioral and sensor annotations on the footage with the supporting measurements beside it. The caption over the video names the step under way, with one line per side when the arms, grippers or hands do different things at once (label/prompts.py). Each step holds one action, and a hand-over spans several steps.

The banner over the dense timeline gives the model's reading of the episode under the recording's instruction. On an inference rollout it also shows the policy's instruction, whether the rollout succeeded, and how long the policy held the robot still while it computed its action chunks. Below the footage, an episode page lists the outcome and scores, the problems, one table of subgoals and key events, the sensors and scene, then every check that ran. The same findings export as episode JSON, filtered JSON Lines and static sites.

For a run in progress, give a dataset's manifest entry a target such as {"episodes": 2682, "hours": 101.9}, and the board shows the episodes and hours labelled so far against it, even before its first episode is labelled. A page open on board serve checks for a rebuild every 30s and updates its list and totals in place, keeping the open episode.

[!TIP]

To count the tasks in your labelled data and the hours of each, run python -m label.catalog --label L --out OUT. It places every labelled task in a task family, a coarse task and a specific task. A specific task is one action on one kind of object, so picking up a rubber duck and picking up a stuffed animal are counted separately. Each row lists its hours, the hours in which the task succeeded, and how many recordings it appears in. When you label more data, --base adds it to your last catalog under the same task names.

Public exports (the static site, downloads and /api/export) name the uploader's own metadata (file attributes, sidecar fields, publisher fields) but leave out its values, because no filter can recognize every secret a person writes there; --publish-uploader-values publishes them. In every field, exports also redact access attributes, recognizable credential values and private access locations. A site for customers adds the rules in board/public_board.py: --public-header shows one title top left, --customer removes every data issue and fails the build if its audit still finds one, and --split-episodes gives a subtask split out of a longer recording its own prompt and every other card the recording's summary.

Field definitions from the dataset publisher or user override inferred meanings and layouts without changing recorded measurements. Changes to source data or field interpretations invalidate stale findings.

Labels of a LeRobot dataset can go back into that dataset for training. python -m board export-lerobot --dataset DS --labels L --out OUT copies the dataset to a new folder and writes the board's episode files into LeRobot's two language columns. The timeline becomes subtask rows in language_persistent, one active on every frame. Outcomes, operator mistakes, recoveries, key events and object state changes become language_events rows in five new styles of the same names (outcome, mistake, recovery, key_event, state_change), each holding the finding as JSON.

Every time is snapped to a frame timestamp read from the dataset's own data files, and a time more than one frame period outside the episode is refused. The report lists each refused time, each time written more than one frame period from its frame, each step that lost its only frame, and each repeated timestamp. The copy keeps each episode in its own row group, drops the older subtask_index and tools columns as LeRobot's own annotation writer does, and is checked against the rules of LeRobot's annotation validator.

A project that loads these columns registers the five styles in EXTENDED_STYLES and EVENT_ONLY_STYLES in lerobot.datasets.language. The exporter works only for recordings that started as LeRobot datasets.

!Two-camera UMI annotation walkthrough, with action phases, subgoals and goal reasoning over the footage.

Browse example annotations.

uv run python -m prepare folder prepare --root path/to/data --rig teleop_arms --out data/episodes/mine/all
uv run python -m checks data/episodes/mine/all
uv run python -m label --dataset mine --episodes data/episodes/mine/all --kind dry
uv run python -m label --dataset mine --episodes data/episodes/mine/all --kind full --cap 20

uv run python -m prepare molmo prepare --episodes configs/quickstart/teleop.txt \
  --out data/episodes/molmo/quickstart

uv run python -m board static media --board data/review/mine --clips data/review/mine/clips
uv run python -m board static site --board data/review/mine --clips data/review/mine/clips

uv run python -m compare prepare --selection configs/compare/main.json
uv run python -m compare label --selection configs/compare/main.json --models astra,sol61_high --kind full --cap 20
uv run python -m compare board --entries data/runs/compare/main.json --out data/boards/compare
uv run python -m board build data/boards/compare
uv run python -m gate prepare
uv run python -m gate label --kind full --cap 20

Citation

If you use Argus or the labels it produces, please cite the blog post. GitHub's "Cite this repository" button reads the same entry from CITATION.cff.

@article{li2026data2,
  author = {Eric Li},
  title = {Argus: A Dense Annotator for Robotics Data},
  journal = {Pantheon Research},
  year = {2026},
  month = {October},
  day = {1}
}

Credits

The capture checks in checks/vendor/public_dataset_adapter_qc.py are Sambhav Gupta's, from public-dataset-adapter (commit 99d0a9e), with the changes listed in that file's header. The hand pose model is ACE-Ego-Hand (Yufei Liu et al., arXiv:2608.20308), built on Wan2.2, VideoX-Fun and MANO (Romero, Tzionas and Black, 2017).

Licenses

The code is Apache-2.0 (LICENSE), and the few files that carry material under another license are listed in THIRD_PARTY_NOTICES.txt. Full datasets and optional components are downloaded at run time; the architecture chart includes attributed image excerpts. The labels Pantheon publishes on the data dashboard, in its downloads and in the release are licensed CC BY 4.0. Anyone may use them for any purpose with credit to Pantheon. DiverseUMI-sample100, labels included, is released under the Pantheon Commercial Use License, Version 1.0. The footage they describe keeps its dataset's license.