Profile
Back to NewsBack
GitHub Trending 3 min
Reader Mode
ArchiveBox/ArchiveBox: 🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

ArchiveBox/ArchiveBox: 🗃 Open source self-hosted web archiving. Takes URLs/browser history/bookmarks/Pocket/Pinboard/etc., saves HTML, JS, PDFs, media, and more...

15 hours ago
GitHub Trending

ArchiveBox
Open-source self-hosted web archiving.


▶️ Quickstart | Demo | GitHub | Documentation | Info & Motivation | Community


   


ArchiveBox is a self-hosted app that lets you preserve content from websites in a variety of formats.

We aim to make your data immediately useful, and kept in formats that other programs can read directly. As output, we save standard HTML, PNG, PDF, TXT, JSON, WARC, SQLite, all guaranteed to be readable for decades to come. ArchiveBox also has a CLI, REST API, and webhooks so you can set up integrations with other services.

Without active preservation effort, everything on the internet eventually disappears or degrades.

ArchiveBox is an open source tool that lets organizations & individuals archive both public & private web content while retaining control over their data. It can be used to save copies of bookmarks, preserve evidence for legal cases, backup photos from FB/Insta/Flickr or media from YT/Soundcloud/etc., save research papers, and more...

➡️ Get ArchiveBox with pip install archivebox>=0.9.0rc0' on Linux/macOS, or via Docker ⭐️ on Linux/macOS.

Once installed, you can interact with it through the: Browser Extension, CLI, self-hosted web interface, Python API, or filesystem.



📥 You can feed ArchiveBox URLs one at a time, or schedule regular imports from your bookmarks or history, social media feeds or RSS, link-saving services like Pocket/Pinboard, our Browser Extension, and more. See Input Formats for a full list of supported input formats...


snapshot detail page

It saves snapshots of the URLs you feed it in several redundant formats. It also detects any content featured inside pages & extracts it out into a folder:

  • 🌐 HTML/Any websites ➡️ original HTML+CSS+JS, singlefile HTML, screenshot PNG, PDF, WARC, title, article text, favicon, headers, ...
  • 🎥 Social Media/News ➡️ post content TXT, comments, title, author, images, ...
  • 🎬 YouTube/SoundCloud/etc. ➡️ MP3/MP4s, subtitles, metadata, thumbnail, ...
  • 💾 Github/Gitlab/etc. links ➡️ clone of GIT source code, README, images, ...
  • 📑 Google Docs/Dropbox/Scientific Papers ➡️ PDF, DOCX, CSV, PPTX, ...
  • ✨ and more, see Output Formats below...
You can run ArchiveBox as a Docker web app to manage these snapshots, or continue accessing the same collection using the uv-installed CLI, Python API, and SQLite3 APIs. All the ways of using it are equivalent, and provide matching features like adding tags, scheduling regular crawls, viewing logs, and more...


🛠️ ArchiveBox uses standard tools like Chrome, wget, & yt-dlp, and stores data in ordinary files & folders. (no complex proprietary formats, all data is readable without needing to run ArchiveBox)

The goal is to sleep soundly knowing the part of the internet you care about will be automatically preserved in durable, easily accessible formats for decades after it goes down.


📦  Install ArchiveBox using your preferred method: docker / uv / brew / apt / etc. (see full Quickstart below).

  Expand for quick copy-pastable install commands...   ⤵️ 
# Option A: Get ArchiveBox with Docker Compose (recommended):
mkdir -p ~/archivebox/data && cd ~/archivebox
curl -fsSL 'https://docker-compose.archivebox.io' > docker-compose.yml
docker compose pull
docker compose up -d --wait                                                # initializes new collections automatically

open http://admin.archivebox.localhost:5797 to finish setup


docker compose run --rm archivebox add 'https://example.com'


docker compose run --rm archivebox help






Option B: Or use it as a plain Docker container:

mkdir -p ~/archivebox/data && cd ~/archivebox/data
docker run -d --name archivebox -v "$PWD:/data" -p 5797:5797 archivebox/archivebox:dev

open http://admin.archivebox.localhost:5797 to finish setup


docker run -it -v $PWD:/data archivebox/archivebox:dev add 'https://example.com'


docker run -it -v $PWD:/data archivebox/archivebox:dev help






Option C: Or install it with uv (see Quickstart below for apt, brew, and more)

uv tool install --python 3.13 --prerelease explicit --upgrade 'archivebox>=0.9.0rc0,<0.10'
mkdir -p ~/archivebox/data && cd ~/archivebox/data
archivebox init
archivebox install

archivebox add 'https://example.com'


archivebox help


archivebox server 0.0.0.0:5797






Option D: Or use the uv install shortcut for Option C

curl -fsSL 'https://get.archivebox.io' | bash

Open http://web.archivebox.localhost:5797 for the public UI and http://admin.archivebox.localhost:5797 for the admin UI ➡️
Set BASE_URL to change the public base domain. The default auto mode uses web. and admin. subdomains on *.localhost, but one host for ordinary DNS names. BIND_ADDR only controls the local listen address.



bookshelf graphic   logo   bookshelf graphic

Demo | Screenshots | Usage
. . . . . . . . . . . . . . . . . . . . . . . . . . . .

cli init screenshot cli init screenshot server snapshot admin screenshot server snapshot details page screenshot

Key Features



grassgrass

Quickstart

🖥  Supported OSs: Ubuntu, macOS, Docker   👾  CPUs: amd64 (x86_64), arm64


✳️  Easy Setup

Apple ArchiveBox Server.app (macOS)
  1. Download ArchiveBox Server.app, move it to Applications, and open it.
  2. Set up an admin account and generate an API Key.
  3. Use the Web UI or ArchiveBox.app, and optionally connect REST API clients, the mobile app, the browser extension, and other integrations to your server using your API Key.

Docker docker-compose (macOS/Linux)   👈  recommended   (click to expand)
👍 Docker Compose is recommended for the easiest install/update UX + best security + all extras out-of-the-box.

  1. Install Docker on your system (if not already installed).
  2. Download the docker-compose.yml file into a new empty directory (can be anywhere). 
    mkdir -p ~/archivebox/data && cd ~/archivebox
curl -fsSL 'https://docker-compose.archivebox.io' > docker-compose.yml
docker compose pull
  3. Start the server, which initializes a new collection automatically. 
    docker compose up -d --wait
  4. Open /admin/ on the hostname or IP used to reach ArchiveBox (local example: http://admin.archivebox.localhost:5797/admin/) to create the first admin. If BASE_URL is not configured yet, continue through the web setup wizard. 
    

    run CLI commands inside the server container started above
    
docker compose exec archivebox archivebox add 'https://example.com'
docker compose exec archivebox archivebox help
    For more info, see Install: Docker Compose in the Wiki. ➡️

Docker image channels: archivebox/archivebox:dev advances only after full CI passes. Development servers can opt into :dev-unstable, which follows the newest clean dev image after both architectures finish the Docker build and checks, without waiting for the rest of CI. The existing universal :sha-<12-character-commit> tag is published at the same time. Both channels include amd64 and arm64, and are also available at ghcr.io/archivebox/archivebox. The early dev-unstable and SHA images can contain changes that fail later checks; ordinary installations should keep using the fully tested release channels.

See below for more usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive.

Docker docker run (macOS/Linux)
  1. Install Docker on your system (if not already installed).
  2. Create a new empty directory and start the server, which initializes the collection automatically (can be anywhere). 
    mkdir -p ~/archivebox/data && cd ~/archivebox/data
docker run -d --name archivebox -v $PWD:/data -p 5797:5797 archivebox/archivebox:dev
  3. Open /admin/ on the hostname or IP used to reach ArchiveBox (local example: http://admin.archivebox.localhost:5797/admin/) to create the first admin. If BASE_URL is not configured yet, continue through the web setup wizard. 
    

    completely optional, CLI can always be used without running a server
    

    docker exec archivebox archivebox [subcommand] [--help]
    
docker exec archivebox archivebox help
    For more info, see Install: Docker Compose in the Wiki. ➡️

See below for more usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive.

curl sh uv install shortcut bash uv install shortcut (macOS/Linux/BSD)
  1. Run the shortcut for the uv install method. It installs uv first when needed. 
    curl -fsSL 'https://get.archivebox.io' | bash
    For more info, see Install: Bare Metal in the Wiki. ➡️

See below for more usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive.
See setup.sh for the source code of the uv install shortcut.
See "Against curl | sh as an install method" blog post for my thoughts on the shortcomings of this install method.


🛠  Package Manager Setup

uv uv (macOS/Linux/BSD)

  1. Install uv on your system (if not already installed).
  2. Install the ArchiveBox package using uv. 
    uv tool install --python 3.13 --prerelease explicit --upgrade 'archivebox>=0.9.0rc0,<0.10'
archivebox version
    See the Install: Bare Metal Wiki for full install instructions for each OS...
  3. Create a new empty directory and initialize your collection (can be anywhere). 
    mkdir -p ~/archivebox/data && cd ~/archivebox/data   # for example
archivebox init     # initialize a new collection
archivebox install  # install all the runtime dependencies (e.g. chrome, single-file, yt-dlp, etc.)
  4. Optionally start the server, then open /admin/ on the hostname or IP used to reach ArchiveBox (local example: http://admin.archivebox.localhost:5797/admin/) to create the first admin. If BASE_URL is not configured yet, continue through the same web setup wizard used by Docker installs. 
    archivebox server 0.0.0.0:5797

    completely optional, CLI can always be used without running a server
    

    archivebox [subcommand] [--help]
    
archivebox help

See below for more usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive.

See the uv tool documentation for more details about this installation method.

aptitude apt (Ubuntu/Debian/etc.)
  1. Add the third-party ArchiveBox apt repo and install archivebox. 
    echo 'deb [trusted=yes] https://archivebox.github.io/debian-archivebox dev main' | sudo tee /etc/apt/sources.list.d/archivebox.list
sudo apt update
sudo apt install archivebox
(cd /tmp && archivebox version)            # make sure the package is installed
  2. Create a new empty directory and initialize your collection (can be anywhere). 
    mkdir -p ~/archivebox/data
cd ~/archivebox/data
archivebox init
sudo archivebox install
archivebox add 'https://example.com'

  3. Optionally start the server, then open /admin/ on the hostname or IP used to reach ArchiveBox (local example: http://admin.archivebox.localhost:5797/admin/) to create the first admin. If BASE_URL is not configured yet, continue through the same web setup wizard used by Docker installs. 
    archivebox server 0.0.0.0:5797

    completely optional, CLI can always be used without running a server
    

    archivebox [subcommand] [--help]
    
archivebox help
See below for more usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive.
The apt package is a thin dev-channel wrapper around the normal Python install flow. sudo archivebox install uses apt for missing system dependencies while preserving ownership of the user-owned collection. See the debian-archivebox repo for details.

homebrew brew (macOS and Linux)
Run Homebrew as your normal non-root user on both macOS and Linux; do not use sudo brew.
  1. Install Homebrew on your system (if not already installed).
  2. Install the ArchiveBox package using brew. 
    brew tap archivebox/archivebox
brew trust archivebox/archivebox
brew install archivebox
archivebox version                         # verify the installed version
    See the Install: Bare Metal Wiki for more granular instructions for macOS... ➡️
  3. Create a new empty directory and initialize your collection (can be anywhere). 
    mkdir -p ~/archivebox/data && cd ~/archivebox/data
archivebox init
archivebox install
  4. Optionally start the server, then open /admin/ on the hostname or IP used to reach ArchiveBox (local example: http://admin.archivebox.localhost:5797/admin/) to create the first admin. If BASE_URL is not configured yet, continue through the same web setup wizard used by Docker installs. 
    archivebox server 0.0.0.0:5797

    completely optional, CLI can always be used without running a server
    

    archivebox [subcommand] [--help]
    
archivebox help

See below for more usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive.
See the homebrew-archivebox repo for more details about this distribution.

Arch pacman / FreeBSD pkg / Nix nix (Arch/FreeBSD/NixOS/more)

Warning: These are contributed by external volunteers and may lag behind the official uv and Docker channels.
See below for usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive.


🎗  Other Options

Docker docker + Electron electron Desktop App (macOS/Linux/Windows)
  1. Install Docker on your system (if not already installed).
  2. Download the latest installer for Windows, Linux, or Mac.
  3. Install and open ArchiveBox Desktop, then create your local administrator account to start archiving.

Desktop app guide · Screenshots

Self-hosting Platforms TrueNAS / UNRAID / YunoHost / Cloudron / etc. (self-hosting solutions)

Warning: These are contributed by external volunteers and may lag behind the official Docker and uv channels.
See below for usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive.

paid Third-party paid hosting options (cloud VPS)

For more discussion on third-party hosting options see here: Issue #531.


➡️  Next Steps


Usage

⚡️  CLI Usage

ArchiveBox commands can be run in a terminal directly on your host, or via Docker/Docker Compose. (depending on how you chose to install it above)

mkdir -p ~/archivebox/data   # create a new data dir anywhere
cd ~/archivebox/data         # IMPORTANT: cd into the directory

archivebox [subcommand] [--help]

archivebox version
archivebox help

equivalent: docker compose run --rm archivebox [subcommand] [--help]

docker compose run --rm archivebox help

equivalent: docker run -it -v $PWD:/data archivebox/archivebox:dev [subcommand] [--help]

docker run -it -v $PWD:/data archivebox/archivebox:dev help

ArchiveBox Subcommands

  • archivebox help/version to see the list of available subcommands / currently installed version info
  • archivebox setup/init/config/status/shell/manage to administer your collection
  • archivebox add/schedule to pull in fresh URLs from bookmarks/history/RSS/etc.
  • archivebox list/update/remove to manage existing Snapshots in your collection

curl sh automatic setup script CLI Usage Examples: non-Docker 


make sure you have uv-installed ArchiveBox and it's available in your $PATH first




archivebox [subcommand] [--help]

mkdir -p ~/archivebox/data && cd ~/archivebox/data
archivebox init
archivebox install
archivebox add 'https://example.com'
archivebox version           # get archivebox version info + check dependencies
archivebox help              # get list of archivebox subcommands that can be run
For more info, see our Usage: CLI Usage wiki. ➡️


Docker CLI Usage Examples: Docker Compose 


make sure you have docker-compose.yml from the Quickstart instructions first




docker compose exec archivebox archivebox [subcommand] [--help]

docker compose exec archivebox archivebox version
docker compose exec archivebox archivebox help
docker compose exec archivebox archivebox add 'https://example.com'

to start webserver: docker compose up
For more info, see our Usage: Docker Compose CLI wiki. ➡️


Docker CLI Usage Examples: Docker 


make sure the archivebox server container from the Quickstart is running first




docker exec archivebox archivebox [subcommand] [--help]

docker exec archivebox archivebox version
docker exec archivebox archivebox help
docker exec archivebox archivebox add 'https://example.com'
For more info, see our Usage: Docker CLI wiki. ➡️


🗄  SQL/Python/Filesystem Usage 

archivebox shell           # explore the Python library API in a REPL
sqlite3 ./index.sqlite3    # run SQL queries directly on your index
find ./archive/users -path '/snapshots////index.html'  # inspect snapshot data directly
For more info, see our Python Shell, SQL API, and Disk Layout wikis. ➡️


🖥  Web UI & API Usage 


Start the server on bare metal (uv/apt/brew):

archivebox server 0.0.0.0:5797                 # start the server



Or with Docker Compose:

docker compose up                              # start the server



Or with a Docker container:

docker run -v $PWD:/data -it -p 5797:5797 archivebox/archivebox:dev

Open http://admin.archivebox.localhost:5797/admin/ to create the first admin and finish web setup. Use http://web.archivebox.localhost:5797 for the public UI. ➡️
Advanced: archivebox manage createsuperuser remains available for creating accounts from the CLI.
Set BASE_URL to change the public base domain. The default auto mode uses web. and admin. subdomains on *.localhost, but one host for ordinary DNS names. BIND_ADDR only controls the local listen address.

For more info, see our Usage: Web UI wiki. ➡️

Optional: Change permissions to allow non-logged-in users


archivebox config --set PUBLIC_ADD_VIEW=True   # allow guests to submit URLs 
archivebox config --set PERMISSIONS=public     # make newly added snapshots public
archivebox config --set PUBLIC_INDEX=True      # allow guests to see list of all snapshots

or

docker compose run --rm archivebox config --set PERMISSIONS=public

restart the server to apply any config changes

Snapshot caching follows each snapshot's saved PERMISSIONS, including inherited defaults and snapshot overrides. Public content can be cached by browsers and shared proxies/CDNs; private and unlisted content uses Cache-Control: private so only the user's browser can cache it. Both retain normal artifact cache lifetimes and validators for fast repeat viewing. Authentication handoffs are not stored. Changing permissions does not revoke copies already cached under the previous policy; purge any affected shared cache when needed.


Apple Windows 🐧  Desktop / Mobile / Extension Usage




[!TIP]
Whether in Docker or not, ArchiveBox commands work the same way, and can be used to access the same data on-disk.
For example, you could run the Web UI in Docker Compose, and run one-off commands with uv-installed ArchiveBox.

Expand to show comparison...


archivebox add --depth=1 'https://example.com'                     # add a URL with uv-installed archivebox on the host
docker compose run --rm archivebox add --depth=1 'https://example.com'                  # or w/ Docker Compose
docker run -it -v $PWD:/data archivebox/archivebox:dev add --depth=1 'https://example.com'  # or w/ Docker, all equivalent

For more info, see our Docker wiki. ➡️


grassgrass

. . . . . . . . . . . . . . . . . . . . . . . . . . . .

DEMO: https://demo.archivebox.io
Usage | Configuration | Caveats


lego


Overview

Input Formats: How to pass URLs into ArchiveBox for saving

Provides realtime archiving of browsing history or selected pages from Chrome/Brave/Firefox/Safari/Edge browsers. Save directly from the Share menu or Siri/Shortcuts to your server. Save links from DMs, @mentions, and groups; use ArchiveBox AI Bot for multi-step archive tasks. Setup & screenshots →
  • From manual imports of URLs from RSS, JSON, CSV, TXT, SQL, HTML, Markdown, etc. files
ArchiveBox supports injesting URLs in any text-based format. Instructions: Chrome, Firefox, Safari, IE, Opera, and more... Provides realtime archiving of all traffic from any device going through the proxy.
  • From bookmarking services or social media (e.g. Twitter bookmarks, Reddit saved posts, etc.)
Instructions: Pocket, Pinboard, Instapaper, Shaarli, Delicious, Reddit Saved, Wallabag, Unmark.it, OneTab, Firefox Sync, and more... 
# archivebox add --help
archivebox add 'https://example.com/some/page'
archivebox add --depth=1 --plugins=parse_rss_urls "file://$HOME/Downloads/some_feed.xml"
archivebox add --depth=1 'https://news.ycombinator.com#2020-12-12'
echo 'http://example.com' | archivebox add
echo 'any text with <a href="https://example.com">urls</a> in it' | archivebox add

if using Docker, add -i when piping stdin:


echo 'https://example.com' | docker run -v $PWD:/data -i archivebox/archivebox:dev add


if using Docker Compose, add -T when piping stdin / stdout:


echo 'https://example.com' | docker compose run --rm -T archivebox add

See the Usage: CLI page for documentation and examples.

It also includes a built-in scheduled import feature with archivebox schedule, handled by the same orchestrator that powers archivebox server, so you can pull in URLs from RSS feeds and websites regularly without a separate cron container.


Output Formats: What ArchiveBox saves for each URL

For each web page added, ArchiveBox creates a Snapshot folder and preserves its content as ordinary files inside the folder (e.g. HTML, PDF, PNG, JSON, etc.).

It uses all available methods out-of-the-box, but you can disable extractors and fine-tune the configuration as-needed.


Expand to see the full list of ways it saves each page...

data/archive/users/{username}/snapshots/{YYYYMMDD}/{domain}/{Snapshot.id}/

  • Index: index.html & index.json HTML and JSON index files containing metadata and details
  • Title, Favicon, Headers Response headers, site favicon, and parsed site title
  • SingleFile: singlefile.html HTML snapshot rendered with headless Chrome using SingleFile
  • Wget Clone: example.com/page-name.html wget clone of the site with warc/TIMESTAMP.gz
  • Chrome Headless
    • PDF: output.pdf Printed PDF of site using headless chrome
    • Screenshot: screenshot.png 1440x900 screenshot of site using headless chrome
    • DOM Dump: output.html DOM Dump of the HTML after rendering using headless chrome
  • Article Text: article.html/json Article text extraction using Readability & Mercury
  • Archive.org Permalink: archive.org.txt A

    ... (README truncated for length)

You finished the article!

Want to go deeper or explore more stories?

Full Article on GitHub Trending More Tech News
Reading Mode

Adjust typography and theme to your preference.

View Original
Related Stories
I Tracked a Scammer For 72 Hours. It Got Personal.
Dev.to 2 days ago
The Internet of Things is Held Together With Duct Tape and Default Passwords[Badly]
Dev.to 2 days ago
Ten findings, ten answers — publishing the audit of our own black box
Dev.to 2 days ago
Scraping websites and bot-protection
Dev.to 3 days ago
Nine years, a £5 Pico, and 100 nanoseconds — the Xbox verification glitch
Dev.to 3 days ago
Related Stories

I Tracked a Scammer For 72 Hours. It Got Personal.

Dev.to 2 days ago

The Internet of Things is Held Together With Duct Tape and Default Pas...

Dev.to 2 days ago

Ten findings, ten answers — publishing the audit of our own black box

Dev.to 2 days ago

Scraping websites and bot-protection

Dev.to 3 days ago

Nine years, a £5 Pico, and 100 nanoseconds — the Xbox verification gli...

Dev.to 3 days ago
Chat with me