memtier_benchmark ================= !GitHub

memtier_benchmark is a command line utility developed by Redis (formerly Garantia Data Ltd.) for load generation and benchmarking NoSQL key-value databases. It offers the following:

• Support for both Redis and Memcache protocols (text and binary)
• Multi-threaded multi-client execution
• Multiple configuration options, including:

Read more at:

• A High Throughput Benchmarking Tool for Redis and Memcached
• Pseudo-Random Data, Gaussian Access Pattern and Range Manipulation

Getting Started

Installing on Debian and Ubuntu

Pre-compiled binaries are available from the packages.redis.io Redis APT repository for Debian 11 (bullseye), 12 (bookworm), and 13 (trixie), and for Ubuntu 20.04 (focal), 22.04 (jammy), 24.04 (noble), and 26.04 (resolute). To configure this repository, use the following steps:

sudo apt install lsb-release curl gpg
curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/redis.list
sudo apt-get update

Once configured, to install memtier_benchmark use:

sudo apt-get install memtier-benchmark

Installing on RHEL, Rocky Linux, and AlmaLinux

Pre-compiled binaries are available for RHEL 8/9/10 compatible distributions from the packages.redis.io YUM repository. To configure this repository, use the following steps:

sudo curl -fsSL https://packages.redis.io/gpg -o /etc/pki/rpm-gpg/redis-archive-keyring.gpg
sudo tee /etc/yum.repos.d/redis.repo <<'EOF'
[redis]
name=Redis
baseurl=https://packages.redis.io/rpm/el$releasever/$basearch
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/redis-archive-keyring.gpg
EOF

Once configured, to install memtier_benchmark use:

sudo dnf install -y memtier-benchmark

Installing on Amazon Linux 2023

Amazon Linux 2023 uses a separate repository path because $releasever on AL2023 is 2023, not an el value:

sudo curl -fsSL https://packages.redis.io/gpg -o /etc/pki/rpm-gpg/redis-archive-keyring.gpg
sudo tee /etc/yum.repos.d/redis.repo <<'EOF'
[redis]
name=Redis
baseurl=https://packages.redis.io/rpm/amzn2023/$basearch
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/redis-archive-keyring.gpg
EOF

Once configured, to install memtier_benchmark use:

sudo dnf install -y memtier-benchmark

Installing on MacOS

To install memtier_benchmark on MacOS, use Homebrew:

brew install memtier_benchmark

Installing from source

For detailed instructions on building from source, running tests, and using sanitizers for development, see DEVELOPMENT.md.

Using Docker

Use available images on Docker Hub:

# latest stable release
$ docker run --rm redislabs/memtier_benchmark:latest --help
master branch edge build
$ docker run --rm redislabs/memtier_benchmark:edge --help

Or, build locally:

$ docker build -t memtier_benchmark .
$ docker run --rm memtier_benchmark --help

Using Docker Compose

$ docker-compose -f docker-compose.memcached.yml up --build

$ docker-compose -f docker-compose.redis.yml up --build

Using memtier_benchmark

See the included manpage or run:

$ memtier_benchmark --help

for command line options.

Numeric options that take a count, size, timeout or port (for example -n, -c, -t, --test-time, --pipeline, -d, --max-reconnect-attempts, --prometheus-port) reject negative, empty, out-of-range and malformed values instead of silently wrapping them. Leading whitespace and a + are still accepted. Note that an empty value such as --max-reconnect-attempts= used to mean 0 (unlimited) and is now an error. The same applies to --max-retries, which takes -1 (unlimited) up to 2147483647: an empty --max-retries= used to mean 0 (retries disabled) and is now an error, and values above 2147483647 are rejected instead of wrapping (4294967295 used to become -1, unlimited). Empty values are also rejected for --data-offset, --select-db, --thread-conn-start-min-jitter-micros, --thread-conn-start-max-jitter-micros, --scan-incremental-max-iterations, --replica-clients and --replicas-per-shard, where they used to mean 0; pass 0 explicitly. -0 (and -00) still parse as 0 on every option, since strtoul() never wrapped them; every other negative value is rejected. --data-offset above 536870911 is now rejected when the option is parsed, with "must be <= 536870911", instead of later with "data-offset too long". --pipeline, -d and --reconnect-interval now print "must be a positive integer" for -1, empty values and trailing junk (they used to print "must be greater than zero"), and --rate-limiting errors now name the option ("rate-limiting must be ..." instead of "rate must be ..."). --multi-key-get is now capped at 16777214, the largest value whose preallocated per-client key buffer (256 bytes per key) stays below 4 GiB (above it, keylist's old 32-bit size wrapped). It is a memory sanity limit: each client preallocates about 272 bytes per key, so size memory accordingly.

Using monitor input files

You can replay real command streams by pointing memtier_benchmark to a monitor log file with the --monitor-input=/path/to/file option. Special commands such as __monitor_line1__ pick a specific entry from the file, while __monitor_line@__ selects commands at runtime (optionally combined with --monitor-pattern and --command-ratio). For example, the following command replays the first command from the file on each request:

$ memtier_benchmark --monitor-input=monitor.txt --command=__monitor_line1__

This lets you mix synthetic workloads with realistic captured traffic. Monitor input works in both single endpoint and cluster mode (--cluster-mode); in cluster mode each command is sent to the shard that owns its first argument's key. Monitor lines may use the __key__ and __data__ placeholders, and text around __key__ (for example user:__key__ or {tag}__key__) is kept. Each __key__ argument gets its own generated key, so in cluster mode a __monitor_line@__ line with more than one __key__ argument must give them all the same {hash tag} (for example "MSET" "{t}a:__key__" "v" "{t}b:__key__" "v"), or memtier_benchmark refuses to start; without a tag the keys could land in different slots. The check looks only at __key__ arguments: a literal key next to a generated one (for example "MSET" "fixed" "v" "k:__key__" "v") is not detected, and Redis answers CROSSSLOT unless both share a hash tag. --failed-keys-file and --retry-on-error use the first argument as the request's key, the same one the command is routed by.

When the number of requested operations exceeds the number of commands in the monitor file, commands are reused:

• Sequential mode (--monitor-pattern=S): Commands wrap around and replay in order. For example, with 10K commands and 1M operations, the commands replay 100 times (0, 1, 2, ..., 9999, 0, 1, 2, ...).
• Random mode (--monitor-pattern=R, default): Each operation picks a random command from the file.

To generate monitor logs, you can use the Redis MONITOR command from redis-cli, which prints all commands received by the server. For example:

$ redis-cli MONITOR
OK
1460100081.165665 [0 127.0.0.1:51706] "set" "shipment:8000736522714:status" "sorting"
1460100083.053365 [0 127.0.0.1:51707] "get" "shipment:8000736522714:status"

You can pipe this output and filter specific patterns with tools such as grep, then save it to a file and use it as a --monitor-input source. For more details, see the official Redis documentation on monitoring commands executed in Redis.

What's new in 2.4

> Binary payloads — the monitor capture loader preserves binary payloads (including embedded NUL bytes) end-to-end into monitor_command_list. However, dispatch via arbitrary_command(const char *) currently re-truncates at the first NUL — full binary blob support through replay is tracked as #439.

> CR-only line endings (classic Mac \r, and some Windows export tools that emit bare \r without a following \n) parse correctly; previously the whole file would be read as one giant line and no commands were replayed.

Command statistics breakdown

By default, when using arbitrary commands (--command), statistics are aggregated by command type (e.g., all SET commands are grouped together, all GET commands are grouped together). You can control this behavior with the --command-stats-breakdown option:

• --command-stats-breakdown=command (default): Aggregate statistics by command name (first word, e.g., SET, GET)
• --command-stats-breakdown=line: Show each command line separately

$ memtier_benchmark --command="SET foo __data__" --command="SET bar __data__" --command="GET foo"

With the default command breakdown, you'll see aggregated stats for "Sets" and "Gets". With --command-stats-breakdown=line, you'll see separate rows for each command line.

JSON object keys must be unique, so when several command lines share a name their per-line JSON entries are numbered in output-row order (--command order for user lines), e.g. "Sets #1", "Sets #2", and "SET #1", "SET #2" for the latency histograms and Per-Key Misses. The per-command --hdr-file-prefix files use a shell-safe _N suffix instead (e.g. _GET_1_command_run_1.hgrm, _GET_2_command_run_1.hgrm), so they do not overwrite each other. Names that appear only once keep their plain key ("Gets") and file name.

Note: When using --command-stats-breakdown=command, the JSON output's Time-Serie percentiles (p50, p99, etc.) for aggregated command types are approximate. They reflect only one of the underlying commands rather than a true merge of all commands of that type. Totals, counts, bytes, and average latency are correctly aggregated.

Cursor iteration

--scan-incremental-iteration follows the cursors returned by SCAN, SSCAN, HSCAN, and ZSCAN, restarting when the cursor returns zero. Use one --command, --pipeline=1, and a standalone Redis endpoint. For example:

memtier_benchmark --command="SSCAN myset 0 COUNT 100" --scan-incremental-iteration --pipeline=1
memtier_benchmark --command="HSCAN __key__ 0 MATCH field:* COUNT 100" --scan-incremental-iteration --pipeline=1
memtier_benchmark --command="ZSCAN myzset 0 COUNT 100" --scan-incremental-iteration --pipeline=1

Each client keeps its own cursor and generated arguments (including __key__ affixes) throughout a walk. A new walk selects new generated arguments. For existing SCAN workloads, this changes generated MATCH arguments: SCAN 0 MATCH __data__ now reuses the initial pattern until the walk ends, instead of regenerating it for each continuation. Likewise, SCAN 0 MATCH __key__ advances the key generator once per walk rather than once per request, so the same request budget can visit fewer generated patterns. MATCH, COUNT, and command-specific options such as HSCAN NOVALUES are preserved; option availability depends on the Redis server version. --scan-incremental-max-iterations=N caps the number of continuation requests per walk (zero means unlimited). Statistics separate initial requests, such as HSCAN 0, from continuations, such as HSCAN . For compatibility, the initial statistics label remains 0 even when the command specifies a nonzero starting cursor; each new walk restarts at that supplied cursor.

End of a --test-time run

When --test-time expires, memtier stops sending and waits for the replies to requests already in flight. It waits at most 10 seconds (a fixed grace period). If the server has stopped replying, each client still waiting after that is ended: its unanswered requests are counted as errors, written to --failed-keys-file (status no-reply-after-test-time) when set, and reported as "Unanswered requests" in the JSON Runtime section. A warning is printed to stderr and the run reports the --test-time window as usual, with exit code 0. Commands that legitimately take longer than 10 seconds to reply after the deadline (for example a slow --command Lua script) are reported the same way.

With --retry-on-error, requests that get a retryable error after the deadline are still retried, so a slot migration or a failover in progress can complete. A client that is still retrying when the grace period is over (for example against a server that rejects every write with -OOM, with the default unlimited --max-retries) is ended the same way: its requests held for retry are not resent, are counted as errors and written to --failed-keys-file with status run-ended-before-resend, and stderr shows warning: N client(s) were still retrying M request(s) 10 seconds after the test time ended. They are not part of "Unanswered requests", which only counts requests that were waiting for a reply.

Interrupting a run

Ctrl+C (SIGINT) and SIGTERM (sent by kill, timeout, docker stop, Kubernetes and most CI runners) stop a run gracefully: memtier stops the threads, prints the results for the time it ran, and writes a complete --json-out-file with "Interrupted": "true" in the Runtime section. With --run-count, no further run starts; the summary covers the runs that happened ("Runs completed" in the JSON run information section).

A second signal sent more than about 1 second after the first ends memtier immediately, without results, as a way out of a stuck shutdown. Signals within that second count as one: Ctrl+C under timeout, or a kill of the whole process group, delivers the same signal twice.

Exit status:

• after a graceful interrupt (first SIGINT or SIGTERM): 0. Check

• after a second signal: memtier dies by that signal (128 + signal number in

• with --run-count, the exit status covers every run, not only the last

Output write errors

If the results cannot be written to --out-file, --json-out-file or stdout (for example on a full disk, an exceeded quota or an I/O error), memtier prints error: failed writing --out-file PATH: REASON (or --json-out-file PATH, or results to stdout) and exits with status 1, also after a graceful interrupt. A connection-stage abort still exits with 2 when an output write fails too.

One exception: when stdout is a pipe or socket and its reader has gone away (e.g. memtier_benchmark ... | head), the run is not failed, since the reader chose to stop reading. This also applies to an --out-file that is the same file as stdout, such as /dev/stdout. A write that fails while the reader is still there (for example a full non-blocking pipe) still fails the run. Any other --out-file (for example a FIFO whose reader exits) and --json-out-file always fail the run when a write fails.

Not covered yet: write errors on the HDR histogram files (--hdr-file-prefix), the --client-stats CSV files and --failed-keys-file are still not checked.

A stdout that was already closed when memtier started is handled separately, see Closed stdin, stdout or stderr.

Closed stdin, stdout or stderr

If a benchmark run is started with stdin, stdout or stderr closed (for example memtier_benchmark ... >&-), memtier reopens that descriptor on /dev/null before it opens any file. So --out-file, --json-out-file and the other files never share a descriptor with stdin, stdout or stderr. Then two checks run at startup, before connecting:

• Right after option parsing, before any file is opened or read: a path

• After all other option checks and the data import, just before the

Otherwise:

• A closed stdin or stderr does not by itself fail the run. With stderr

• Any spelling of /dev/null itself is accepted, and so is another

• --hdr-file-prefix and --client-stats are prefixes: memtier always

Crash Reporting

memtier_benchmark includes built-in crash handling that automatically generates detailed bug reports when the program crashes. If you encounter a crash, the tool will print a comprehensive report including:

• Stack traces for all threads
• System and build information
• Active client connection states
• Instructions for generating core dumps

Cluster mode

Connections

When using the cluster-mode option, each client opens one connection for each node. So, when using a large number of threads and clients, the user must verify that he is not limited by the maximum number of file descriptors.

Using sequential key pattern

When there is an asymmetry between the Redis nodes and user set the --requests option, there may be gaps in the generated keys.

Also, the ratio and the key generator is per client (and not connection). In this case, setting the ratio to 1:1 does not guarantee 100% hits because the keys spread to different connections/nodes.

Read Preference (cluster mode)

--read-preference selects which class of node receives read commands when running against a Redis Cluster with replicas. Four modes are available:

• primary (default) - reads land on the slot owner (primary). Legacy

• secondary - all reads go to a replica; if no live replica owns the

• secondaryPreferred - prefer a replica; on absence of a live replica

• nearest - EWMA-based selection. Cold replicas are seeded round-robin

When the target node class is unavailable, the fallback is controlled by --read-preference-fallback:

• error (default) - return an error.
• primary - fall back silently to the primary.
• queue - reserved; currently treated as error. Request queuing is

Built-in read commands (GET, MGET) and arbitrary commands marked with --command-is-read participate in read-preference routing. Reads on arbitrary keyless commands (e.g. DBSIZE, CLUSTER NODES) are routed to the same node class.

--read-server=HOST:PORT lets you declare additional replica endpoints for standalone (non-cluster) runs. The flag is parsed in all modes, but read dispatch via the --read-preference policy currently fires only in cluster mode.

--replica-clients=N and --replicas-per-shard=K are parsed but not yet wired into the connection topology in this release; see the open limitations in the PR.

Read-preference also extends the JSON output (--json-out-file) with two new top-level blocks (emitted only when --cluster-mode is on AND --read-preference != primary):

• Read Routing - Ops from Primary, Ops from Replica, and

• Endpoints - one entry per distinct shard connection (address + role)

Testing limitations

The OSS-CLUSTER + replicas: read-preference CI matrix cell exercises the RLTest harness with --use-slaves, which starts replicas using --slaveof without --cluster-enabled yes. The resulting slave nodes are not part of cluster gossip - CLUSTER SLOTS returns empty replica arrays from each master, so memtier_benchmark cannot discover them. The test helper's get_cluster_replica_connections() gate detects this and skips the tests with a stderr warning.

The production read-routing code was verified empirically against real redis-cli --cluster create clusters with cluster-aware replicas under both RESP2 and RESP3. See PR #456 round-18 reviewer reports for the measurements: all four modes (primary, secondary, secondaryPreferred, nearest) route reads correctly to replicas, with zero leakage to primaries.

A follow-up fixture that bootstraps a real redis-cli --cluster create cluster within the test harness is tracked in issue #462 (filed by this PR).

Using rate-limiting for informed benchmarking

When you impose a rate limit on your benchmark tests, you're essentially mimicking a controlled production environment. This setup is crucial for understanding how latency behaves under certain throughput constraints. Here's why benchmarking latency in a rate-limited scenario is important:

1. Realistic Performance Metrics: In real-world scenarios, systems often operate under various limitations. Understanding how these limitations affect latency gives you a more accurate picture of system performance, than simply running benchmarks at full stress level.

1. Capacity Planning: By observing latency at different rate limits, you can better plan for scaling your infrastructure. It helps in identifying at what point increased load leads to unacceptable latency, guiding decisions about when to scale up.

1. Quality of Service (QoS) Guarantees: For services that require a certain level of performance guarantee, knowing the latency at specific rate limits helps in setting realistic QoS benchmarks.

1. Identifying Bottlenecks: Rate-limited benchmarking can help in identifying bottlenecks in your system. If latency increases disproportionately with a small increase in rate limit, it may indicate a bottleneck that needs attention.

1. Comparative Analysis: It enables the comparison of different solutions, configurations or hardware in terms of how they handle latency under simmilar benchmark conditions.

Using rate-limiting in memtier

To use this feature, add the --rate-limiting parameter followed by the desired RPS per connection.

memtier_benchmark [other options] --rate-limiting=<RPS>

Note: When using rate-limiting together with cluster-mode option, the rate-limit is associated to the connection for each node.

Rate limited example: 100% writes, 1M Keys, 60 seconds benchmark at 10K RPS

memtier_benchmark --ratio=1:0 --test-time=60 --rate-limiting=100 -t 2 -c 50 --key-pattern=P:P --key-maximum 1000000

Full latency spectrum analysis

For distributions that are non-normal, such as the latency, many “basic rules” of normally distributed statistics are violated. Instead of computing just the mean, which tries to express the whole distribution in a single result, we can use a sampling of the distribution at intervals -- percentiles, which tell you how many requests actually would experience that delay.

When used for normally distributed data, the samples are usually taken at regular intervals. However, since the data does not obey to a normal distribution it would be very expensive to keep equally spaced intervals of latency records while enabling large value ranges. We can apply algorithms that can calculate a good approximation of percentiles at minimal CPU and memory cost, such as t-digest or HdrHistogram. On memtier_benchmark we’ve decided to use the HdrHistogram due to its low memory footprint, high precision, zero allocation during the benchmark and constant access time.

By default Memtier will output the 50th, 99th, and 99.9th percentiles. They are the latency thresholds at which 50%, 99%, and 99.9% of commands are faster than that particular presented value. To output different percentiles you should use the --print-percentiles option followed by the comma separated list of values ( example: --print-percentiles 90,99,99.9,99.99 ).

Saving the full latency spectrum

Sample Visual Feel of the full latency spectrum using an online formatter: ![alt text][sample_visual_histogram]

[sample_visual_histogram]: ./docs/sample_visual_histogram.png "Sample Full Latency Spectrum Histogram"

Real-Time Metrics Visualization

memtier_benchmark can stream live metrics to a StatsD server for real-time visualization in Grafana. This is useful for monitoring performance during long-running tests or observing behavior during scaling events.

Quick Start

# Start the included Graphite + Grafana stack
docker-compose -f docker-compose.statsd.yml up -d
Run a benchmark with metrics streaming
./memtier_benchmark -s <redis-host> --statsd-host=localhost --test-time=60
Open Grafana at http://localhost:3000 (admin/admin)

StatsD Options

| Option | Default | Description | |--------|---------|-------------| | --statsd-host=HOST | (disabled) | StatsD server hostname. Enables metrics when set. | | --statsd-port=PORT | 8125 | StatsD server UDP port. | | --statsd-prefix=PREFIX | memtier | Prefix for metric names. | | --statsd-run-label=LABEL | default | Label to identify/compare benchmark runs. |

For detailed setup instructions, dashboard usage, and troubleshooting, see Real-Time Metrics Guide.

Prometheus /metrics exporter

memtier_benchmark can also expose live metrics as a Prometheus /metrics endpoint, scrapable by Prometheus or any OpenMetrics-compatible collector. The exporter is built in by default (it requires libevent 2.1.1 or newer; see DEVELOPMENT.md for the --disable-prometheus build flavor). It is off at runtime until you pass --prometheus-port:

# Bind the exporter on a fixed port (loopback by default)
./memtier_benchmark -s <redis-host> --test-time=60 --prometheus-port=8080
Scrape it:
curl http://127.0.0.1:8080/metrics
Or let the OS pick an ephemeral port; the bound URL is announced on stdout:
./memtier_benchmark -s <redis-host> --test-time=60 --prometheus-port=0
-> "Prometheus exporter listening on http://127.0.0.1:43217/metrics"

| Option | Default | Description | |--------|---------|-------------| | --prometheus-port=PORT | (disabled) | TCP port for the /metrics HTTP endpoint. 0 selects an ephemeral port (announced on stdout). Enables the exporter when set. | | --prometheus-bind-addr=ADDR | 127.0.0.1 | Address to bind the exporter to. Loopback by default; a non-loopback address prints a one-shot warning. | | --prometheus-run-label=K=V | (none) | Repeatable label applied to every exported sample, e.g. to compare runs. | | --prometheus-latency-buckets=L | (26 defaults) | Comma-separated histogram bucket upper bounds in seconds. |

The exporter exposes cumulative counters (ops, bytes, hits/misses, errors, connection errors, retries), gauges (connections, threads, current/configured run), and a latency histogram (memtier_latency_seconds), plus build info. By design it echoes no connection identity (no server address, password, URI, or filesystem path) into the scrape body. The end-of-run output remains the authoritative latency source; the histogram counts every operation but trails memtier_ops_total by about one second of operations (longer while the server is stalled or under a low --rate-limiting rate), and never exceeds it. See the Real-Time Metrics Guide for the full metric mapping, threat model, and operational notes.