theHarvester

!theHarvester logo

theHarvester requires Python 3.14 and uv. The repository's .python-version lets uv select the required Python version automatically:

git clone https://github.com/laramies/theHarvester.git
cd theHarvester
uv sync
uv run theHarvester -d example.com -b crtsh,certspotter

This first run uses two P0 passive sources that need no API key. Passive means the target is not contacted directly, but the selected providers still receive the target string. The terminal reports each source outcome and any retained findings. A source can complete with zero findings, stop early with partial evidence, or fail without erasing evidence retained by other sources.

See the installation guide for packaged distributions and platform-specific setup.

Choose how to work

| Interface | Best for | Start with | | --- | --- | --- | | CLI | One-off reconnaissance and shell workflows | uv run theHarvester -h | | HarvestView | Local run history, evidence review, hostname changes, and schedules | uv run harvestview | | REST API | Authenticated local automation and integrations | http://127.0.0.1:5000/docs |

The CLI, REST API, and HarvestView share one finite-run engine and normalized evidence model. HarvestView provides the local browser interface.

Common workflows

Query several P0 passive sources:

uv run theHarvester -d example.com -b crtsh,certspotter,commoncrawl

Three discovery sources run at once by default. Use -j or --source-workers to change the worker count. Every selected source still runs.

Run every source that can contribute subdomains:

uv run theHarvester -d example.com -b subdomains

Combine capability selectors, or mix them with explicit source names:

uv run theHarvester -d example.com -b emails,urls,certspotter

Capability selectors form a union and choose which sources run. They do not discard other result types returned by those sources. Available selectors are subdomains, emails, ips, asns, urls, people, and breaches. -b all runs every cataloged P0 passive source. P1 DNS and P2 direct sources require explicit selection.

Pass --limit 0 to remove the shared per-source result cap and local page ceilings. Provider quotas and runtime safeguards still apply. If a provider or safety limit stops a source after retaining results, the run keeps them and records a partial outcome with the stop reason.

Exclude hostname results while retaining other result types:

uv run theHarvester -d example.com -b emails,ips,urls --no-hosts -f non-host-results

--no-hosts skips hostname-only sources and omits hostname results while keeping other result types. It cannot be combined with actions that depend on hostnames. HarvestView and the REST API expose the same option as no_hosts.

Save results as JSONL:

uv run theHarvester -d example.com -b crtsh,certspotter -f report

This writes report.jsonl for automation and interchange plus report.json and report.xml compatibility reports. Completed runs are also retained in the local SQLite evidence store.

Resolve discovered hosts for an authorized domain with the default resolver list:

AUTHORIZED_DOMAIN='replace-with-a-domain-you-control'
uv run theHarvester -d "$AUTHORIZED_DOMAIN" -b crtsh,certspotter -r

List every option and its current behavior:

uv run theHarvester -h

Activity and scope

P0, P1, and P2 describe observable network behavior, not confidence or importance.

| Class | Network behavior | Examples | | --- | --- | --- | | P0 | Queries an existing provider or dataset without directing traffic toward the target | Search, certificate transparency, code, and passive DNS providers | | P1 | Queries DNS about authorized names or addresses | Resolution, brute force, recursive DNS, and reverse lookup | | P2 | Contacts a target endpoint or causes equivalent direct interaction | HTTP, TLS, screenshots, takeover checks, virtual hosts, ports, and API paths |

P1 and P2 activity runs only when you select it. A discovered related hostname, network, ASN, or URL remains review evidence and does not expand the authorized target automatically. The exact hostname matters: www.example.test and example.test are different targets unless the engagement authorizes both.

Common active options include DNS resolution (-r), DNS brute force (-c), reverse DNS (-n), recursive DNS (--dns-recursive-depth), takeover checks (-t), API path scanning (-a), and screenshots (--screenshot). A takeover indicator is evidence for review, not proof that a provider resource can be claimed.

Read Responsible use and scope before active work. Operator workflows covers limits, resolvers, proxies, and action-specific behavior. With explicit proxy mode enabled, supported discovery and actions fail closed when no proxy is available instead of sending a direct request.

A practical engagement workflow

1. Record the exact targets and activity the engagement authorizes.
2. Start with a bounded P0 run and inspect every source outcome, including zero-result and partial sources.
3. Add only the P1 or P2 actions the engagement authorizes for that target.
4. Review normalized results, retained DNS evidence, stop reasons, and artifacts together.
5. Compare finalized runs with matching targets, source sets, limits, and resolvers collected in comparable time windows.
6. Export the required evidence, then pause or delete schedules and verify that no queued or running work remains when authorization ends.

Architecture at a glance

From authorized target to retained evidence

export THEHARVESTER_API_KEY='replace-with-a-long-random-value'
uv run harvestview

Open HarvestView to start runs and inspect history, results, producer outcomes, retained artifacts, and hostname changes. Schedules can persist a timezone-aware run template across one or many authorized targets. Each occurrence creates an ordinary finite run per target, and the single local worker executes them serially.

SQLAlchemy stores schedule control state in a separate mode-0600 SQLite database that is excluded from portable run exports. Pausing a schedule prevents future occurrences but does not cancel runs already queued or running. Delete or pause schedules and inspect their dispatch history before an authorization window closes.

The browser receives a derived HttpOnly session cookie and never stores the API key. Keep HarvestView on loopback unless you add TLS and network access controls. See the installation guide for local assets, screenshot support, and isolated deployments.

HarvestView can start screenshot and DNS brute-force runs from a retained hostname. Each action creates a separate finite run and leaves the parent evidence unchanged.

Track hostname changes

harvest-report hostname-changes compares finalized SQLite evidence without running discovery or DNS. Select a run to compare it with its previous comparable run, or select a target to see the comparisons over time:

uv run harvest-report hostname-changes --run-id 11111111-1111-4111-8111-111111111111
uv run harvest-report hostname-changes --target example.test
uv run harvest-report hostname-changes --target example.test --include-still-reported

The previous comparable run is the latest earlier finalized run with the same canonical target and exact source list. The comparison classifies hostnames as newly_reported, still_reported, no_longer_reported, or uncertain. When only one run reports a hostname, it counts as newly reported or no longer reported only if every source that reported it completed successfully in the other run. Otherwise the difference is uncertain and includes the incomplete source outcomes and reasons.

No longer reported means absent from comparable saved evidence. It does not prove that the hostname stopped existing or resolving. Repeated comparable runs provide a change history; the command does not create alerts, trigger actions, or allow mixed-target timelines. Read Results and local data for the complete pairing and interpretation rules.

REST API and Docker Compose

Open Swagger or ReDoc for the current automation contract. API clients send THEHARVESTER_API_KEY in the X-API-Key header; provider credentials remain on the server.

Docker Compose

The Compose service runs as an unprivileged user and binds only to host loopback. It stores runs in a named volume and reads the operator key from a file secret:

install -d -m 0700 .secrets
openssl rand -hex 32 > .secrets/operator-api-key
chmod 0444 .secrets/operator-api-key
docker compose up --build -d
docker compose ps

docker compose logs -f theharvester.svc.local
docker compose down

The REST API guide documents requests, hostname comparisons, imports, exports, schedules, and authentication.

Discovery sources

Select sources by name or by a capability listed below. -b all runs the P0 sources. P1 and P2 sources require explicit selection.

Result types in this table always appear in this order: subdomains, emails, ips, asns, urls, people, breaches. A result followed by only means the source contributes no other result type. The API key column refers to provider settings in api-keys.yaml; some providers require more than one value. Optional means the source can run without a key.

The shodan source contributes subdomains. Shodan host enrichment through -s or --shodan is a separate action and is not a source result route.

Each source name links to its provider's site or documentation for current plans, quotas, and terms. See Configuration and API keys for the required fields and setup instructions. Contributors can add a provider through the module guide. The CLI and API read their source inventory from the source catalog.

Configuration

On first use, theHarvester creates default configuration files under ~/.theHarvester/. It also reads system configuration from /etc/theHarvester/ and /usr/local/etc/theHarvester/.

• api-keys.yaml stores provider API keys and related values such as organization IDs.
• proxies.yaml configures HTTP and SOCKS5 proxies used with -p.
• The shodan source and -s / --shodan enrichment use Shodan's Host REST API. When -p is enabled, both send those requests through proxies.yaml.
• routeviews.key is optional and enables authenticated RouteViews access for PeeringDB-verified users.

For JSMON, set apikeys.jsmon.key in api-keys.yaml:

uv run theHarvester -d example.com -b jsmon -l 100

Use -l 0 to follow all pages reported by JSMON without a local result cap. HTTP 403 (plan quota) or 429 (rate limit) stops collection immediately without retrying; results from earlier pages are retained.

Output and local data

Terminal output is intended for interactive use. -f NAME also writes NAME.jsonl, NAME.json, and NAME.xml. Screenshots go to the directory passed to --screenshot, and completed runs are stored in ~/.local/share/theHarvester/stash.sqlite.

Treat collected OSINT as potentially sensitive. Keep report files, screenshots, and the local database out of source control and share them only within the authorized engagement. Screenshots can contain login pages, internal names, and other target content; the database retains their metadata but does not embed the image files.

JSONL

JSONL is the primary format for automation and one-run interchange. The first line summarizes run status, producer outcomes, counts, timestamps, and artifacts. Each remaining line is one sorted, deduplicated finding with its source and action provenance. The example below contains six findings from two sources.

{"action_executions":[],"artifacts":[],"completed_at":"2026-08-17T12:01:00Z","counts":{"asn":1,"breach":1,"email":1,"hostname":1,"ip":1,"url":1},"evidence_status":"complete","result_count":6,"run_id":"123e4567-e89b-12d3-a456-426614174000","source_executions":[{"duration_ms":127.4,"error_type":null,"result_count":1,"source":"haveibeenpwned","status":"completed","stop_reason":null},{"duration_ms":482.3,"error_type":null,"result_count":5,"source":"zoomeye","status":"completed","stop_reason":null}],"started_at":"2026-08-17T12:00:00Z","target":"example.com","type":"summary"}
{"sources":["zoomeye"],"type":"asn","value":"AS64500"}
{"sources":["haveibeenpwned"],"type":"breach","value":"Example breach"}
{"sources":["zoomeye"],"type":"email","value":"[email protected]"}
{"sources":["zoomeye"],"type":"hostname","value":"api.example.com"}
{"sources":["zoomeye"],"type":"ip","value":"192.0.2.10"}
{"sources":["zoomeye"],"type":"url","value":"https://api.example.com/login"}

Extract common result types with jq:

jq -r 'select(.type == "hostname") | .value' report.jsonl
jq -r 'select(.type == "ip") | .value' report.jsonl
jq -r 'select(.type == "asn") | .value' report.jsonl
jq -r 'select(.type == "email") | .value' report.jsonl
jq -r 'select(.type == "url") | .value' report.jsonl
jq -c 'select(.type == "person") | .value | fromjson' report.jsonl
jq -r 'select(.type == "breach") | .value' report.jsonl

Some result types carry structured evidence. person, infostealer, and recursive DNS values contain JSON strings and need a second fromjson step. Shodan hosts use details; virtual hosts, network prefixes, and ASN attribution use native observations. Takeover results keep the hostname in value and their DNS, wildcard, HTTP, rule, status, and error evidence in details.

jq -c 'select(.type == "dns-recursive-finding") | .value | fromjson' report.jsonl
jq -c 'select(.type == "shodan-host") | {ip: .value, services: .details.services}' report.jsonl
jq -c 'select(.type == "hostname" and .observations) | {hostname: .value, observations}' report.jsonl
jq -c 'select(.type == "asn" and .observations) | {asn: .value, observations}' report.jsonl

List every finding as tab-separated type and value columns:

jq -r 'select(.type != "summary") | [.type, .value] | @tsv' report.jsonl

The subdomains capability produces hostname records because a result can be the target hostname itself. Read Results and local data for the complete JSONL and evidence contract.

Compare saved runs

harvest-report reads completed runs from an existing SQLite results database. Its subcommands list saved targets, count each source's contribution, and compare hostname evidence between runs:

uv run harvest-report targets
uv run harvest-report contributions --target example.test
uv run harvest-report contributions --database results.sqlite --kind ip --format json
uv run harvest-report hostname-changes --target example.test

The command reads saved evidence without running discovery or DNS. Use --target or --run-id when a database contains several targets, or select --all-targets for a deliberate aggregate. For useful comparisons, keep the target, source list, limit, resolvers, release version, and collection window consistent. The results guide explains the fields, target scope, and comparison rules.

SQLite, JSON, and XML

CLI and API runs use the same SQLite evidence model. JSONL moves one run at a time. SQLite import and export handle completed runs in bulk but exclude queue and worker state. Screenshot files remain separate from their metadata.

JSON and XML are compatibility reports grouped by result type. They do not include the full provenance, lifecycle outcomes, or structured action evidence available in JSONL, SQLite, the API, and HarvestView.

Development and contributing

Read CONTRIBUTING.md for the development setup, required checks, testing expectations, and pull-request process.

Package versions and license

theHarvester is distributed under the GNU General Public License version 2 only. See LICENSE for the complete license text.

Support and credits

• Use GitHub Issues for reproducible bugs and focused feature requests.
• Report suspected vulnerabilities according to SECURITY.md, not in public issues.
• Christian Martorella (@laramies) created theHarvester. Contact: [email protected].
• Matt Brown (@NotoriousRebel1) and Jay "L1ghtn1ng" Townsend (@jay_townsend1) maintain and develop the project.
• Lee Baird (@discoverscripts) is a main contributor.
• Thanks to John Matherly for Shodan and Ahmed Aboul Ela for the bundled subdomain dictionaries.