This README provides a quick overview of the Benchmark package. For more detailed information, please see the documentation.

Overview

Performance is a key feature for many apps and frameworks. Benchmark helps make it easy to measure and track many different metrics that affects performance, such as CPU usage, ARC traffic, memory/malloc usage and use of operating system resources such as threads and system calls, as well as completely custom metric counters.

Benchmark works on both macOS and Linux and supports several key workflows for performance measurements:

• Automated Pull Request performance regression checks by comparing the performance metrics of a pull request with the main branch and having the PR workflow check fail if there is a regression according to absolute or relative threshold tolerances specified per benchmark
• Manual comparison of multiple performance baselines for iterative or A/B performance work by an individual developer
• Export of benchmark results in several formats for analysis or visualization

Benchmark is suitable for both smaller ad-hoc benchmarks focusing on execution time and more extensive benchmarks that care about several additional metrics such as memory allocations, syscalls, thread usage, context switches, ARC traffic, and more. Using Histogram it’s especially suitable for capturing latency statistics for large number of samples.

Documentation

Documentation on how to use Benchmark in your Swift package can be viewed online or inside Xcode using Build Documentation.

Additionally the command plugin provides help information if you run swift package benchmark help from the command line.

Adding dependencies and getting started

There are just a few steps required to get started benchmarking:

1. Add a dependency to the Benchmark project
2. Add benchmark executable targets with swift package benchmark init
3. Add the snippet or code you want to benchmark
4. Run swift package benchmark

Step 1: Add a package dependency to Package.swift

.package(url: "https://github.com/ordo-one/benchmark", .upToNextMajor(from: "1.4.0")),

Step 2: Add benchmark exectuable targets using benchmark init

swift package --allow-writing-to-package-directory benchmark init MyNewBenchmarkTarget

This will perform the following steps for you:

• Create a Benchmarks/MyNewBenchmarkTarget directory
• Create a Benchmarks/MyNewBenchmarkTarget/MyNewBenchmarkTarget.swift benchmark target with the required boilerplate
• Add a new executable target for the benchmark to the end of your Package.swift file

After you’ve created the new target, you can directly run it with e.g.:

swift package benchmark --target MyNewBenchmarkTarget

Step 2 (optional approach): Add benchmark exectuable targets manually

.executableTarget(
      name: "My-Benchmark",
      dependencies: [
          .product(name: "Benchmark", package: "benchmark"),
      ],
      path: "Benchmarks/My-Benchmark",
      plugins: [
          .plugin(name: "BenchmarkPlugin", package: "benchmark"),
      ]
),

Step 3: Writing benchmarks

Sample benchmark code

import Benchmark
let benchmarks : @Sendable () -> Void = {
    Benchmark("Minimal benchmark") { benchmark in
      // measure something here
    }
func defaultCounter() -> Int {
        10
    }
    
    func dummyCounter(_ count: Int) {
        for index in 0 ..< count {
            blackHole(index)
        }
    }
    
    Benchmark("All metrics, full concurrency, async",
              configuration: .init(metrics: BenchmarkMetric.all,
                                   maxDuration: .seconds(10))) { benchmark in
        let _ = await withTaskGroup(of: Void.self, returning: Void.self, body: { taskGroup in
            for _ in 0..<80  {
                taskGroup.addTask {
                    dummyCounter(defaultCounter()*1000)
                }
            }
            for await _ in taskGroup {
            }
        })
    }     
}

Step 4: Running benchmarks

``swift package benchmark

Please see the documentation for more detail on all options.

Sample output benchmark run
<img width="1005" alt="image" src="https://user-images.githubusercontent.com/8501048/225311258-1247f8e9-c1fd-4598-a4b8-2b41a9b9a8e7.png">
Sample output benchmark grouped by metric 
<img width="1089" alt="image" src="https://user-images.githubusercontent.com/8501048/225281786-411530de-25c2-47b5-b99f-0d7bac3209a7.png">
Sample output delta comparison
<img width="1173" alt="image" src="https://user-images.githubusercontent.com/8501048/225282373-e7ba9fa1-1a2a-4028-b053-9f3aa82361b0.png">
Sample output threshold deviation check
<img width="956" alt="image" src="https://user-images.githubusercontent.com/8501048/225282982-95c9c641-9455-4df2-81bc-6aee43721223.png">
Sample usage of YouPlot
Install YouPlot

<img width="523" alt="image" src="https://user-images.githubusercontent.com/8501048/225284254-c1349494-2323-4460-b18a-7bc2896b5dc4.png">
JMH Visualization
Using jmh.morethan.io
<img width="1262" alt="image" src="https://user-images.githubusercontent.com/8501048/225313246-4369da1f-0890-4856-8fd8-b28d56d842aa.png">
<img width="1482" alt="image" src="https://user-images.githubusercontent.com/8501048/225313559-33014755-797f-4ddf-b536-24c1a618f271.png">
Finding allocation sites
An allocation stack is the chain of function calls captured when heap memory is allocated, showing which code path caused the allocation.
To see where a benchmark allocates, run it with --allocation-stacks. The stack trace of every allocation in the measured region is recorded and the unique stacks are printed sorted by allocation count:

### Sample output of benchmark runed with allocation stacks recording

!Terminal output showing an allocation stack report
Warmup iterations and code outside startMeasurement() / stopMeasurement() are excluded. By default, up to 64 frames are captured per stack and the 20 most frequent stacks are printed. Use --allocation-stack-depth <depth> to change the capture depth and --allocation-stack-limit <limit> to change the number printed (0 prints all).

Recording slows allocations down, so this diagnostic mode measures only mallocCountTotal and mallocBytesCount and is available with the run command. It requires the MallocInterposer trait (enabled by default) and a toolchain providing the Swift Runtime module. On macOS, it requires macOS 26 or later.

Exporting folded stacks
Use --export-allocation-stacks to capture, print, and export allocation stacks. It implies --allocation-stacks and defaults to the folded format, which can be opened in speedscope or rendered with FlameGraph:

This writes a <target>.<benchmark>.allocations.folded file in the current directory for each benchmark that records allocations. Folded files contain every captured stack, regardless of --allocation-stack-limit, weighted by exact allocation counts across all measured iterations.

Set --allocation-stacks-export-path <directory> to choose another destination. Grant the plugin write permission with --allow-writing-to-package-directory for output inside the package, or --allow-writing-to-directory <directory> for an external destination. T

Visualization of the folded file with Flamelens:

!Allocation flame graph displayed in Flamelens
Exporting JSON
Select --allocation-stacks-export-format json to export a full report as <target>.<benchmark>.allocations.json, including allocation counts, byte totals, and frames:

JSON also includes every captured stack and writes reports for benchmarks with no allocations. Both --allocation-stacks-export-format and --allocation-stacks-export-path require --export-allocation-stacks; --allocation-stacks alone prints reports without creating files.

To pipe the selected format to another tool, use --allocation-stacks-export-path stdout. This prints only the exported data. Select one benchmark when you need a single JSON report:

See Finding Allocation Sites for report examples and more details.

Swift 6 support
The package supports Swift 6.0 benchmark targets as well as Swift 5.10 targets (for Swift 5.9 support, need to use version 1.28.0 exactly).
For Swift 6 targets, use the following signature for the benchmarks closure:

The default text output from Benchmark is oriented around the five-number summary percentiles, plus the last decile (p90) and the last percentile (p99) - it's thus a variation of a seven-figure summary with the focus on the 'bad' end of results (as those are what we typically care about addressing). We've found that focusing on percentiles rather than average or standard deviations, is more useful for a wider range of benchmark measurements and gives a deeper understanding of the results. Percentiles allows for a consistent way of expressing benchmark results of both throughput and latency measurements (which typically do not have a standardized distribution, being almost always multi-modal in nature). This multi-modal nature of the latency measurements leads to the common statistical measures of mean and standard deviation being potentially misleading.

The benchmark internal baseline representation (stored in .benchmarkBaselines`) is not stable and is not viewed as public API and may break over time.

For those wanting to save benchmark data over time, it's recommended to export data in e.g. HDR Histogram representations (percentiles, average, stddev etc) or simply post processing the histogramSamples format (which is raw data) to your desired representation.

PR:s for additional standardized formats are welcome, as the export formats are the intended stable interface for saving such data.