Context:
* Debian Code Search: taming the latency tail (2014)- Debian Code Search: improving client-side latency (2016)
- Debian Code Search: positional index, TurboPFor-compressed (2019)
- Debian Code Search: OpenAPI now available (2021)
- Debian Code Search: Fast TurboPFor with Go SIMD (2026)
- At least 1 TB of SSD disk space.
- The more CPU, the better. Then, more IOPS (disks that saturate many reads).
- A Google Cloud N2 instance (
n2-standard-16) has 64 GB of RAM and a Cascade Lake CPU (with AVX512), at least in my test.
fio, 16K random reads): 180k IOPS, 2.95 GB/s
* DCS needs 4.6 ns CPU time per byte searched, about 0.22 GB/s per vCPU.
* 32 vCPUs are the recommendation if you want fastest queries. 16 vCPUs is fine, too.
* 32 GB of RAM, or 64 GB of RAM (better) are recommended.
* 128 GB of RAM holds the typical working set and can deliver 2x faster (warm) queries.
Example: Hetzner dedicated server
- A Hetzner AX52 (no longer available to order) has 64 GB of RAM, a Ryzen 7 7700 (8 cores / 16 threads) and two 1 TB Samsung PM9A1 NVMe disks.
fio, 16K random reads): 400k IOPS, 6.5 GB/s (RAID0)
* DCS needs 2.7 ns CPU time per byte searched, about 0.37 GB/s per Zen 4 thread.
* This machine is CPU-bound: To saturate the disk, we’d need a 16 or 32 core CPU.
* With 64 GB of RAM, about 44% of searched bytes came from page cache.
Guide to this repository:
cmd/dcsis the swiss-army knife tool for Debian Code Search, displaying index files in a variety of ways.cmd/dcs-localdcsruns a local development instance of Debian Code Search.
cmd/dcs-compute-rankinggeneratesranking.jsonbased on the Debian Popularity Contest (popcon).cmd/dcs-feederfeeds packages to multipledcs-package-importershards.cmd/dcs-webis the web server which runs searches on multipledcs-source-backendshards.
- A
cmd/dcs-package-importerto receive, unpack and index Debian packages for this shard. - A
cmd/dcs-source-backendto run actual searches over imported source code.