Distill

Learn more →

📖 Distill implements the 4-layer context engineering stack described in The Agentic Engineering Guide, a free open book on AI agent infrastructure.

RAG / tools / memory / docs
          ↓
        Distill
  (remember · dedupe · compress · cache)
          ↓
         LLM

The Problem

Agents forget. Every new session starts from zero — the same constraints, preferences, and facts have to be re-established. When context does persist, 30-40% of it is semantically redundant, and contradictory information sits side by side with no signal about which version is current.

This causes non-deterministic outputs, confused reasoning, and failures that only show up at scale. Better prompts don't fix it. The agent needs memory it can trust.

How It Works

No LLM calls. Fully deterministic. ~12ms overhead.

| Stage | What it does | |-------|-------------| | Remember | Persistent memory across sessions with write-time dedup, expiry, and sensitivity tagging | | Deduplicate | Cluster semantically similar chunks, keep one representative per cluster | | Compress | Extractive compression to remove noise and preserve signal | | Summarize | Progressively condense conversation history as turns age | | Cache | Annotate stable prefixes with cache_control, track TTL per prefix |

All stages chain together via POST /v1/pipeline or distill pipeline CLI. Memory is available via --memory flag.

Dedup pipeline

Query → Over-fetch (50) → Cluster → Select → MMR Re-rank (8) → LLM

1. Over-fetch - retrieve 3-5x more chunks than needed
2. Cluster - group semantically similar chunks (agglomerative clustering)
3. Select - pick the best representative from each cluster
4. MMR Re-rank - balance relevance and diversity

Installation

Binary (Recommended)

Download from GitHub Releases:

# macOS (Apple Silicon)
curl -sL $(curl -s https://api.github.com/repos/Siddhant-K-code/distill/releases/latest | grep "browser_download_url.*darwin_arm64.tar.gz" | cut -d '"' -f 4) | tar xz
macOS (Intel)
curl -sL $(curl -s https://api.github.com/repos/Siddhant-K-code/distill/releases/latest | grep "browser_download_url.*darwin_amd64.tar.gz" | cut -d '"' -f 4) | tar xz
Linux (amd64)
curl -sL $(curl -s https://api.github.com/repos/Siddhant-K-code/distill/releases/latest | grep "browser_download_url.*linux_amd64.tar.gz" | cut -d '"' -f 4) | tar xz
Linux (arm64)
curl -sL $(curl -s https://api.github.com/repos/Siddhant-K-code/distill/releases/latest | grep "browser_download_url.*linux_arm64.tar.gz" | cut -d '"' -f 4) | tar xz
Move to PATH
sudo mv distill /usr/local/bin/

Or download directly from the releases page.

Go Install

go install github.com/Siddhant-K-code/distill@latest

Docker

docker pull ghcr.io/siddhant-k-code/distill:latest
docker run -p 8080:8080 -e OPENAI_API_KEY=your-key ghcr.io/siddhant-k-code/distill

Build from Source

git clone https://github.com/Siddhant-K-code/distill.git
cd distill
go build -o distill .

Development

make build        # compile ./distill
make test         # go test ./...
make check        # fmt + vet + test
make test-cover   # test with coverage report
make bench        # run benchmarks
make distill-lock-demo # deterministic lock/build/verify demo
make lint         # golangci-lint (requires golangci-lint in PATH)
make docker-build # build Docker image
make help         # list all targets

Distill Lock v0: context is a build artifact

Runtime prompt assembly hides source drift. Distill Lock freezes a reviewed source inventory and configuration, then builds a portable context artifact without a model, provider, or network call:

distill lock path/to/config.json --output path/to/context.lock.json
distill build path/to/context.lock.json --output path/to/context-output
distill verify path/to/context-output
distill verify path/to/context-output --expected-lock-sha256 <trusted-digest>
Synthetic fixture: build, offline verify, repeatability, and mutation proof.
make distill-lock-demo

A configuration explicitly lists selected and excluded UTF-8 text/code files. lock records original and normalized SHA-256 identities, NFC/LF-normalized chunks, exact-duplicate decisions, the deterministic token budget, and every inclusion/exclusion reason. build refuses any missing, changed, newly unexpected, unsafe, or unsupported input; it never relocks automatically. verify checks the complete allowlisted file set, canonical JSON, schemas, lengths, hashes, relationships, and exact bundle regeneration offline. Lock output cannot overwrite its configuration. A post-publication directory sync failure is reported explicitly as published but durability unconfirmed; verify the named output instead of retrying blindly. Without the optional out-of-band lock digest it proves internal consistency, not authenticity; use --expected-lock-sha256 when the artifact directory may be attacker-replaceable.

The fresh output directory contains:

context.bundle.md
context.lock.json
context.manifest.json
SHA256SUMS

Guarantee: the same locked sources, configuration, and supported runtime/tool identities produce byte-identical bundle, manifest, lockfile, and digests on supported macOS/Linux systems with Go 1.24 through Go 1.26. Other Go runtime versions fail closed. Original CRLF/CR hashes remain distinct while normalized LF content can deduplicate exactly. Builds use native atomic no-replace publication into a nonexistent directory, so failures cannot replace a previous valid artifact.

Non-guarantees and non-goals: v0 does not provide semantic or near-duplicate matching, a vector database, agent framework, model adapter, decision replay, dashboard, Jev integration, or deterministic model output. The existing semantic Distill pipeline remains available and unchanged.

See the frozen Distill Lock v0 specification for the canonical JSON rules, algorithms, failure behavior, and a complete configuration example.

The separate research-only offline excluded-pilot harness generates synthetic, zero-provider-call protocol fixtures and is not wired into the public distill CLI.

Quick Start

1. Standalone API (No Vector DB Required)

Start the API server and send chunks directly:

export OPENAI_API_KEY="your-key"  # For embeddings
distill api --port 8080

Deduplicate chunks:

curl -X POST http://localhost:8080/v1/dedupe \
  -H "Content-Type: application/json" \
  -d '{
    "chunks": [
      {"id": "1", "text": "React is a JavaScript library for building UIs."},
      {"id": "2", "text": "React.js is a JS library for building user interfaces."},
      {"id": "3", "text": "Vue is a progressive framework for building UIs."}
    ]
  }'

Response:

{
  "chunks": [
    {"id": "1", "text": "React is a JavaScript library for building UIs.", "cluster_id": 0},
    {"id": "3", "text": "Vue is a progressive framework for building UIs.", "cluster_id": 1}
  ],
  "stats": {
    "input_count": 3,
    "output_count": 2,
    "reduction_pct": 33,
    "latency_ms": 12
  }
}

With pre-computed embeddings (no OpenAI key needed):

curl -X POST http://localhost:8080/v1/dedupe \
  -H "Content-Type: application/json" \
  -d '{
    "chunks": [
      {"id": "1", "text": "React is...", "embedding": [0.1, 0.2, ...]},
      {"id": "2", "text": "React.js is...", "embedding": [0.11, 0.21, ...]},
      {"id": "3", "text": "Vue is...", "embedding": [0.9, 0.8, ...]}
    ]
  }'

2. With Vector Database

Connect to Pinecone or Qdrant for retrieval + deduplication:

export PINECONE_API_KEY="your-key"
export OPENAI_API_KEY="your-key"
distill serve --index my-index --port 8080

Query with automatic deduplication:

curl -X POST http://localhost:8080/v1/retrieve \
  -H "Content-Type: application/json" \
  -d '{"query": "how do I reset my password?"}'

3. MCP Integration (AI Assistants)

Works with Claude, Cursor, Amp, and other MCP-compatible assistants:

# Dedup only
distill mcp
With memory and sessions
distill mcp --memory --session

Add to Claude Desktop (~/Library/Application Support/Claude/claude_desktop_config.json):

{
  "mcpServers": {
    "distill": {
      "command": "/path/to/distill",
      "args": ["mcp", "--memory", "--session"],
      "env": {
        "OPENAI_API_KEY": "your-key"
      }
    }
  }
}

See mcp/README.md for more configuration options.

Context Memory

Persistent memory that accumulates knowledge across agent sessions. Memories are deduplicated on write, ranked by relevance + recency on recall, and compressed over time through hierarchical decay.

Enable with the --memory flag on api or mcp commands.

CLI

# Store a memory
distill memory store --text "Auth uses JWT with RS256 signing" --tags auth --source docs
Recall relevant memories
distill memory recall --query "How does authentication work?" --max-results 5
Remove outdated memories
distill memory forget --tags deprecated
View statistics
distill memory stats

API

# Start API with memory enabled
distill api --port 8080 --memory
Store (with auto sensitivity classification)
curl -X POST http://localhost:8080/v1/memory/store \
  -H "Content-Type: application/json" \
  -d '{
    "session_id": "session-1",
    "entries": [{"text": "Auth uses JWT with RS256", "tags": ["auth"], "source": "docs", "auto_classify": true}]
  }'
Recall (with task-relevance boosting)
curl -X POST http://localhost:8080/v1/memory/recall \
  -H "Content-Type: application/json" \
  -d '{"query": "How does auth work?", "max_results": 5, "boost_tags": ["auth"], "min_relevance": 0.3}'
Expire a memory (soft delete)
curl -X POST http://localhost:8080/v1/memory/expire \
  -H "Content-Type: application/json" \
  -d '{"id": "memory-id"}'
Supersede with a newer version
curl -X POST http://localhost:8080/v1/memory/supersede \
  -H "Content-Type: application/json" \
  -d '{"id": "old-memory-id", "new_id": "new-memory-id"}'

MCP

Memory tools are available in Claude Desktop, Cursor, and other MCP clients when --memory is enabled:

distill mcp --memory

Tools exposed: store_memory, recall_memory, forget_memory, memory_expire, memory_supersede, memory_stats.

How Decay Works

Memories compress over time based on access patterns:

Full text → Summary (~20%) → Keywords (~5%) → Evicted
  (24h)        (7 days)         (30 days)

Accessing a memory resets its decay clock. Configure via distill.yaml:

memory:
  db_path: distill-memory.db
  dedup_threshold: 0.15
  conflict_threshold: 0.35

Session Management

Token-budgeted context windows for long-running agent sessions. Push context incrementally - Distill deduplicates, compresses aging entries, and evicts when the budget is exceeded.

Enable with the --session flag on api or mcp commands.

CLI

# Create a session with 128K token budget
distill session create --session-id task-42 --max-tokens 128000
Push context as the agent works
distill session push --session-id task-42 --role user --content "Fix the JWT validation bug"
distill session push --session-id task-42 --role tool --content "$(cat auth/jwt.go)" --source file_read --importance 0.8
Read the current context window
distill session context --session-id task-42
Clean up when done
distill session delete --session-id task-42

API

# Start API with sessions enabled
distill api --port 8080 --session
Create session
curl -X POST http://localhost:8080/v1/session/create \
  -H "Content-Type: application/json" \
  -d '{"session_id": "task-42", "max_tokens": 128000}'
Push entries
curl -X POST http://localhost:8080/v1/session/push \
  -H "Content-Type: application/json" \
  -d '{
    "session_id": "task-42",
    "entries": [
      {"role": "tool", "content": "file contents...", "source": "file_read", "importance": 0.8}
    ]
  }'
Read context window
curl -X POST http://localhost:8080/v1/session/context \
  -H "Content-Type: application/json" \
  -d '{"session_id": "task-42"}'

MCP

Session tools are available when --session is enabled:

distill mcp --session

Tools exposed: create_session, push_session, session_context, delete_session.

How Budget Enforcement Works

When a push exceeds the token budget:

1. Compress oldest entries (outside the preserve_recent window) through levels:

1. Evict entries that are already at keyword level
2. Lowest-importance entries are compressed/evicted first

CLI Commands

distill api        # Start standalone API server
distill serve      # Start server with vector DB connection
distill pipeline   # Run full optimisation pipeline (dedup → compress → summarize)
distill mcp        # Start MCP server for AI assistants
distill memory     # Store, recall, and manage persistent context memories
distill session    # Manage token-budgeted context windows for agent sessions
distill analyze    # Analyze a file for duplicates
distill sync       # Upload vectors to Pinecone with dedup
distill query      # Test a query from command line
distill config     # Manage configuration files
distill completion # Generate shell completion scripts (bash/zsh/fish/powershell)

Pipeline command

# Run full pipeline on a JSON chunk array
echo '[{"id":"1","text":"..."}]' | distill pipeline
From file, with stats
distill pipeline --input chunks.json --output optimised.json --stats
Tune individual stages
distill pipeline --dedup-threshold 0.2 --compress-ratio 0.4 --summarize --summarize-max-tokens 2000
Disable a stage
distill pipeline --no-compress

Shell completions

# Bash (one-time)
distill completion bash > /etc/bash_completion.d/distill
Zsh
distill completion zsh > "${fpath[1]}/_distill"
Fish
distill completion fish > ~/.config/fish/completions/distill.fish
PowerShell
distill completion powershell | Out-String | Invoke-Expression

API Endpoints

Interactive docs available at /docs (Swagger UI). Full OpenAPI 3.1 spec at /openapi.yaml.

| Method | Path | Description | |--------|------|-------------| | POST | /v1/dedupe | Deduplicate chunks | | POST | /v1/dedupe/stream | SSE streaming dedup with per-stage progress | | POST | /v1/pipeline | Full optimisation pipeline (dedup → compress → summarize) | | POST | /v1/batch | Submit async batch job | | GET | /v1/batch/{id} | Poll batch job status and progress | | GET | /v1/batch/{id}/results | Retrieve completed batch results | | POST | /v1/retrieve | Query vector DB with dedup (requires backend) | | POST | /v1/memory/store | Store memories with write-time dedup and sensitivity tagging (requires --memory) | | POST | /v1/memory/recall | Recall memories by relevance + recency (requires --memory) | | POST | /v1/memory/forget | Remove memories by ID, tag, or age (requires --memory) | | POST | /v1/memory/expire | Mark memories as expired without deleting (requires --memory) | | POST | /v1/memory/supersede | Replace a memory with a newer version (requires --memory) | | GET | /v1/memory/stats | Memory store statistics (requires --memory) | | POST | /v1/session/create | Create a session with token budget (requires --session) | | POST | /v1/session/push | Push entries with dedup + budget enforcement (requires --session) | | POST | /v1/session/context | Read current context window (requires --session) | | POST | /v1/session/delete | Delete a session (requires --session) | | GET | /v1/session/get | Get session metadata (requires --session) | | GET | /health | Health check | | GET | /metrics | Prometheus metrics |

Pipeline API

POST /v1/pipeline
{
  "chunks": [{"id": "1", "text": "..."}],
  "options": {
    "dedup":     {"enabled": true, "threshold": 0.15},
    "compress":  {"enabled": true, "target_reduction": 0.5},
    "summarize": {"enabled": false, "max_tokens": 4000}
  }
}

Response includes per-stage token counts, reduction ratios, and latency.

Batch API

# Submit
curl -X POST /v1/batch -d '{"chunks":[...],"options":{...}}'
→ {"job_id":"batch_1234","status":"queued"}
Poll
curl /v1/batch/batch_1234
→ {"status":"processing","progress":0.45}
Results (when completed)
curl /v1/batch/batch_1234/results
→ {"chunks":[...],"stats":{...}}

Logging

Distill uses structured log/slog logging. Default output is JSON to stderr.

import "github.com/Siddhant-K-code/distill/pkg/logging"
// JSON logger (production default)
logger := logging.New(logging.Config{Level: "info", Format: logging.FormatJSON})
// Text logger for local development
logger := logging.NewDebug()
// Attach request context
logger = logging.WithRequestID(logger, requestID)
logger = logging.WithTraceID(logger, traceID)

Log levels: debug, info (default), warn, error.

Configuration

Config File

Distill supports a distill.yaml configuration file for persistent settings. Generate a template:

distill config init              # Creates distill.yaml in current directory
distill config init --stdout     # Print template to stdout
distill config validate          # Validate existing config file

Config file search order: ./distill.yaml, $HOME/distill.yaml.

Priority: CLI flags > environment variables > config file > defaults.

Example distill.yaml:

server:
  port: 8080
  host: 0.0.0.0
  read_timeout: 30s
  write_timeout: 60s
embedding:
  provider: openai
  model: text-embedding-3-small
  batch_size: 100
dedup:
  threshold: 0.15
  method: agglomerative
  linkage: average
  lambda: 0.5
  enable_mmr: true
retriever:
  backend: pinecone    # pinecone or qdrant
  index: my-index
  host: ""             # required for qdrant
  namespace: ""
  top_k: 50
  target_k: 8
auth:
  api_keys:
    - ${DISTILL_API_KEY}
memory:
  db_path: distill-memory.db
  dedup_threshold: 0.15
  conflict_threshold: 0.35
session:
  db_path: distill-sessions.db
  dedup_threshold: 0.15
  max_tokens: 128000

Environment variables can be referenced using ${VAR} or ${VAR:-default} syntax.

Environment Variables

OPENAI_API_KEY      # For text → embedding conversion (see note below)
PINECONE_API_KEY    # For Pinecone backend
QDRANT_URL          # For Qdrant backend (default: localhost:6334)
DISTILL_API_KEYS    # Optional: protect your self-hosted instance (see below)

Protecting Your Self-Hosted Instance

If you're exposing Distill publicly, set DISTILL_API_KEYS to require authentication:

# Generate a random API key
export DISTILL_API_KEYS="sk-$(openssl rand -hex 32)"
Or multiple keys (comma-separated)
export DISTILL_API_KEYS="sk-key1,sk-key2,sk-key3"

Then include the key in requests:

curl -X POST http://your-server:8080/v1/dedupe \
  -H "Authorization: Bearer sk-your-key" \
  -H "Content-Type: application/json" \
  -d '{"chunks": [...]}'

If DISTILL_API_KEYS is not set, the API is open (suitable for local/internal use).

About OpenAI API Key

When you need it:

• Sending text chunks without pre-computed embeddings
• Using text queries with vector database retrieval
• Using the MCP server with text-based tools

• Sending chunks with pre-computed embeddings (include "embedding": [...] in your request)
• Using Distill purely for clustering/deduplication on existing vectors

• Converts text to embeddings using text-embedding-3-small model
• ~$0.00002 per 1K tokens (very cheap)
• Embeddings are used only for similarity comparison, never stored

• Bring your own embeddings - include "embedding" field in chunks
• Use Ollama locally - --embedding-provider ollama (no API key needed)
• Use Cohere - --embedding-provider cohere with COHERE_API_KEY

Parameters

| Parameter | Description | Default | |-----------|-------------|---------| | --threshold | Clustering distance (lower = stricter) | 0.15 | | --lambda | MMR balance: 1.0 = relevance, 0.0 = diversity | 0.5 | | --over-fetch-k | Chunks to retrieve initially | 50 | | --target-k | Chunks to return after dedup | 8 |

Self-Hosting

Docker (Recommended)

Use the pre-built image from GitHub Container Registry:

# Pull and run
docker run -p 8080:8080 -e OPENAI_API_KEY=your-key ghcr.io/siddhant-k-code/distill:latest
Or with a specific version
docker run -p 8080:8080 -e OPENAI_API_KEY=your-key ghcr.io/siddhant-k-code/distill:v0.1.0

Docker Compose

# Start Distill + Qdrant (local vector DB)
docker-compose up

Build from Source

docker build -t distill .
docker run -p 8080:8080 -e OPENAI_API_KEY=your-key distill api

Fly.io

fly launch
fly secrets set OPENAI_API_KEY=your-key
fly deploy

Render

tracker := metrics.NewCallSiteTracker()
// After each Anthropic API call:
tracker.Record("agent/planner.go:84", metrics.UsageRecord{
    CacheCreationInputTokens: resp.Usage.CacheCreationInputTokens,
    CacheReadInputTokens:     resp.Usage.CacheReadInputTokens,
    InputTokens:              resp.Usage.InputTokens,
})
// Inspect
s := tracker.Stats("agent/planner.go:84")
fmt.Printf("hit rate: %.0f%%  efficiency: %.1fx\n", s.HitRate()*100, s.WriteEfficiency())
// All call sites, worst hit rate first
for _, s := range tracker.AllStats() {
    fmt.Printf("%-40s %.0f%%\n", s.CallSite, s.HitRate()*100)
}

scrape_configs:
  - job_name: distill
    static_configs:
      - targets: ['localhost:8080']

telemetry:
  tracing:
    enabled: true
    exporter: otlp         # otlp, stdout, or none
    endpoint: localhost:4317
    sample_rate: 1.0
    insecure: true

export DISTILL_TELEMETRY_TRACING_ENABLED=true
export DISTILL_TELEMETRY_TRACING_ENDPOINT=localhost:4317

store.OnLifecycleEvent(func(e memory.MemoryEvent) {
    // e.Type, e.EntryID, e.TokensBefore, e.TokensAfter, e.CompressionLevel
})

result, _ := store.Recall(ctx, req)
if result.CacheHint != nil {
    // result.CacheHint.StableEntryIDs - IDs likely stable this turn
    // result.CacheHint.ConfidenceScore - mean relevance of returned entries
}

{
  "session_id": "task-42",
  "accepted": 2,
  "current_tokens": 4200,
  "budget_remaining": 123800,
  "cache_boundary": {
    "markers": [
      {"entry_id": "abc123", "tokens_up_to_here": 3800, "stable_since_turn": 1}
    ],
    "total_stable_tokens": 3800,
    "advanced": true,
    "retreated": false
  }
}

session:
  cache_boundary:
    enabled: true
    min_stable_turns: 2     # pushes before an entry is considered stable
    min_prefix_tokens: 1024 # Anthropic's minimum cacheable prefix size
    max_markers: 4          # Anthropic allows up to 4 simultaneous markers

POST /v1/dedupe
{
  "chunks": [
    {"id": "sys", "text": "You are a helpful assistant.", "cache_control": "ephemeral"},
    {"id": "tool1", "text": "Tool schema JSON...", "cache_control": "ephemeral"},
    {"id": "msg1", "text": "What is the capital of France?"},
    {"id": "msg2", "text": "What is the capital of Germany?"}
  ],
  "options": {"preserve_cache_prefix": true}
}

{
  "stats": {
    "input_count": 4, "output_count": 3,
    "cache_prefix_frozen": true,
    "cache_prefix_tokens": 320,
    "cache_prefix_hash": "a3f2c1d4e5b6",
    "suffix_input_count": 2,
    "suffix_output_count": 1
  }
}

tracker := cache.NewTTLTracker(0) // 0 = use AnthropicCacheTTL (5 min)
// After each request that carries a cache_control marker:
wasAlive := tracker.Touch(plan.PrefixHash)
if !wasAlive {
    log.Warn("cache cold start: first request or TTL expired")
}
// For batch workloads: latest safe time to send next request
deadline := tracker.ScheduleDeadline(plan.PrefixHash, 30*time.Second)
time.Sleep(time.Until(deadline))
// Inspect expiry state
entry := tracker.Entry(plan.PrefixHash)
fmt.Printf("hits: %d  misses: %d  alive: %v\n", entry.HitCount, entry.MissCount, entry.IsAlive())

validator := cache.NewStabilityValidator(cache.DefaultStabilityConfig())
// Runtime check, call on every request
issues := validator.Check("agent/planner.go:84", chunks)
for _, issue := range issues {
    log.Warnf("%s", issue) // "cache-prefix-unstable: stability=12%, likely dynamic interpolation: request id"
}
// Static pre-flight check
found := validator.ValidateText(systemPromptText)
// found = ["request id", "timestamp"] if dynamic patterns detected

detector := cache.NewPatternDetector()
plan := detector.AnnotateChunksForCache(chunks)
// plan.Markers lists which chunk indices should receive cache_control markers
// plan.ManualMarkersPresent is true if the caller already placed markers

┌──────────────────────────────────────────────────────────────────────┐
│                         Your App / Agent                             │
└──────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
┌──────────────────────────────────────────────────────────────────────┐
│                             Distill                                  │
│                                                                      │
│  Dedup Pipeline (shipped)                                            │
│  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌──────────┐  ┌─────────┐  │
│  │  Cache  │→ │ Cluster │→ │ Select  │→ │ Compress │→ │  MMR    │  │
│  │  check  │  │  dedup  │  │  best   │  │  prune   │  │ re-rank │  │
│  └─────────┘  └─────────┘  └─────────┘  └──────────┘  └─────────┘  │
│     <1ms          6ms         <1ms          2ms           3ms        │
│                                                                      │
│  Context Intelligence                                                │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────────────┐   │
│  │ Memory Store │  │ Impact Graph │  │ Session Context Windows  │   │
│  │  (shipped)   │  │  (#30)       │  │  (shipped)               │   │
│  └──────────────┘  └──────────────┘  └──────────────────────────┘   │
│                                                                      │
│  ┌──────────────────────────────────────────────────────────────┐    │
│  │  /metrics (Prometheus)  ·  OTEL tracing  ·  MCP server      │    │
│  └──────────────────────────────────────────────────────────────┘    │
└──────────────────────────────────────────────────────────────────────┘
                                  │
                                  ▼
┌──────────────────────────────────────────────────────────────────────┐
│                              LLM                                     │
└──────────────────────────────────────────────────────────────────────┘

# Use Ollama (local, no API key)
distill api --embedding-provider ollama --embedding-base-url http://localhost:11434
Use Cohere
distill api --embedding-provider cohere
Use OpenAI (default)
distill api --embedding-provider openai

import (
    "github.com/Siddhant-K-code/distill/pkg/embedding"
    _ "github.com/Siddhant-K-code/distill/pkg/embedding/openai"  // register OpenAI
    _ "github.com/Siddhant-K-code/distill/pkg/embedding/ollama"  // register Ollama
    _ "github.com/Siddhant-K-code/distill/pkg/embedding/cohere"  // register Cohere
)
provider, err := embedding.NewProvider(embedding.ProviderConfig{
    Type:      embedding.ProviderOllama,   // "openai" | "ollama" | "cohere"
    BaseURL:   "http://localhost:11434",   // optional override
    Model:     "nomic-embed-text",         // optional override
    CacheSize: 10000,                      // 0 = default (10k), -1 = disabled
})

embedding.RegisterFactory("my-provider", func(cfg embedding.ProviderConfig) (embedding.Provider, error) {
    return myProvider{apiKey: cfg.APIKey}, nil
})

| Feature | Issue | Status | Description | |---------|-------|--------|-------------| | Context Memory Store | #29 | Shipped | Persistent, deduplicated memory across sessions. Write-time dedup, hierarchical decay, token-budgeted recall. See Context Memory. | | Session Management | #31 | Shipped | Stateful context windows with token budgets, hierarchical compression, and importance-based eviction. See Session Management. | | PatternDetector cache_control annotations | #53 | Shipped | PatternDetector emits CacheAnnotation per chunk and AnnotateChunksForCache produces a CacheControlPlan with up to 4 Anthropic-compatible markers. | | Session-aware cache boundary manager | #51 | Shipped | Auto-advances cache_control placement as sessions grow. Stable entries (present ≥ 2 turns unmodified) are included in the cached prefix; boundary retreats when content changes. | | Cache write cost accounting | #52 | Shipped | 9 new Prometheus metrics covering Anthropic prompt cache token usage, hit rate, write efficiency, and boundary position. Feed API response usage via RecordCacheUsage. | | Memory decay lifecycle events | #54 | Shipped | DecayWorker emits EventCompressed and EventEvicted on each transition. RecallResult includes a CacheBoundaryHint for high-relevance entries. | | Cache-aware dedup | #50 | Shipped | preserve_cache_prefix option freezes chunks before the last cache_control marker so dedup cannot reorder them. Prefix hash and token count reported in stats. | | Prefix stability validator | #48 | Shipped | StabilityValidator tracks prefix hashes across requests and detects dynamic content (timestamps, request IDs, UUIDs) bleeding into cached prefixes. | | Per-call-site hit rate tracking | #47 | Shipped | CallSiteTracker records Anthropic cache usage per call site; AllStats() returns worst performers first. | | TTL-aware cache tracker | #49 | Shipped | TTLTracker monitors Anthropic's 5-minute cache TTL per prefix hash. ScheduleDeadline tells batch jobs the latest safe time to send the next request. | | Multi-provider embedding abstraction | #33 | Shipped | embedding.NewProvider factory supports OpenAI, Ollama, and Cohere via a unified ProviderConfig. Custom providers register via RegisterFactory. | | Memory expiry and supersession | #79 | Shipped | Soft-delete via expire, version replacement via supersede. Expired entries excluded from recall. | | Sensitivity classification | #82 | Shipped | Regex-based classifier detects PII, credentials, and internal IPs on store. Recall includes sensitivity metadata. | | Conflict detection | #77 | Shipped | Entries with cosine distance 0.15–0.35 from existing memories are flagged as conflicts on store. | | Task-relevance ranking | #78 | Shipped | Recall supports boost_tags, task_context, and min_relevance for task-aware re-ranking. | | Multi-provider embeddings (CLI) | #25 | Shipped | --embedding-provider flag on api, serve, and memory commands. Supports openai, ollama, cohere. |

Code Intelligence

| Feature | Issue | Status | Description | |---------|-------|--------|-------------| | Change Impact Graph | #30 | Shipped | pkg/graph: BFS blast-radius queries over a dependency graph built from Go imports. | | Semantic Commit Analysis | #32 | Shipped | pkg/commits: Conventional Commits parser, heuristic risk scoring, cosine similarity search over commit embeddings. |

Infrastructure

| Feature | Issue | Status | Description | |---------|-------|--------|-------------| | Multi-Provider Embeddings | #33 | Shipped | embedding.NewProvider factory: OpenAI, Ollama, Cohere via unified ProviderConfig. | | Unified Pipeline | #4 | Shipped | POST /v1/pipeline + distill pipeline CLI: dedup → compress → summarize in one call with per-stage stats. | | Batch API | #11 | Shipped | POST /v1/batch: async job queue with worker pool, progress polling, 24h result retention. | | Structured Logging | #27 | Shipped | pkg/logging: JSON/text slog logger with debug/info/warn/error levels, request_id and trace_id helpers. | | Shell Completions | #26 | Shipped | distill completion [bash\|zsh\|fish\|powershell] generates shell completion scripts. | | Benchmark Suite | #24 | Shipped | go test -bench=. ./... covers cluster, MMR, selector, and compress with deterministic synthetic data. | | Makefile | #28 | Shipped | 20+ targets: build, test, bench, lint, fmt, vet, docker, release. | | OpenAPI Spec | #23 | Shipped | OpenAPI 3.1 spec at /openapi.yaml, Swagger UI at /docs. | | v2.0 Documentation | #8 | Shipped | Guides, API reference, configuration reference, LangChain and RAG examples. See docs/. | | Python SDK | #5 | Planned | pip install distill-ai with LangChain/LlamaIndex integrations. |

See all open issues: github.com/Siddhant-K-code/distill/issues

Why not just use an LLM?

LLMs are non-deterministic. Reliability requires deterministic preprocessing.

| | LLM Compression | Distill | |---|---|---| | Latency | ~500ms | ~12ms | | Cost per call | $0.01+ | $0.0001 | | Deterministic | No | Yes | | Lossless | No | Yes | | Auditable | No | Yes |

Use LLMs for reasoning. Use deterministic algorithms for reliability.

Integrations

Works with your existing AI stack:

• Embedding Providers: OpenAI, Ollama (local), Cohere
• LLM Providers: OpenAI, Anthropic
• Frameworks: LangChain, LlamaIndex (SDKs planned: #5)
• Vector DBs: Pinecone, Qdrant
• AI Assistants: Claude Desktop, Cursor (via MCP)
• Observability: Prometheus, Grafana, OpenTelemetry (Jaeger, Tempo)

FAQ