OpenTelemetry eBPF Profiler

This repository implements a whole-system, cross-language profiler for Linux via eBPF.

Core features and strengths

• Implements the Alpha OTel Profiles signal
• Very low CPU and memory overhead (1% CPU and 250MB memory are our upper limits

• Support for native C/C++ executables without the need for DWARF debug

• Support profiling of system libraries without frame pointers and **without

• Support for mixed stacktraces between runtimes - stacktraces go from Kernel

• Support for native code (C/C++, Rust, Zig, Go, etc. without debug symbols on

• Support for a broad set of HLLs, like Hotspot JVM, Python, Ruby, PHP, Node.JS,

• 100% non-intrusive: there's no need to load agents or libraries into the

• No need for any reconfiguration, instrumentation or restarts of HLL

• ARM64 support for all unwinders.
• Support for native inline frames, which provide insights into compiler

Supported Linux kernel version

The minimum required Linux kernel version has increased with certain commits. Specifically:

• Commit 8047150e was the last to support kernel version 5.4. Subsequent changes may require a minimal Linux kernel version of 5.10 or greater.
• Commit 7ddc23ea was the last to support kernel version 4.19. Subsequent changes may require a minimal Linux kernel version of at least 5.4.

Updating the supported Linux kernel version

The project maintains its minimum supported kernel version in line with the lowest kernel version currently provided by actively maintained major Linux distributions, which include Debian stable, Red Hat Enterprise Linux, Ubuntu LTS, Amazon Linux and SUSE Linux. The minimum requirement may be increased when all such distributions no longer ship a specific kernel version. This approach enables the codebase to utilize newer eBPF features and avoids the need to maintain compatibility shims for obsolete kernels.

It should be noted that certain distributions incorporate eBPF features from newer kernels into their supported versions. When this occurs, the distribution's stated kernel version does not accurately reflect its true eBPF capabilities and will not prevent us from increasing the minimum supported version. On such kernels, the no-kernel-version-check configuration option can be used to bypass the checks and allow the profiler to execute.

Building

We have integrated the profiler into the OTel Collector as a receiver, and this is the supported configuration going forward.

To aid with development, testing and debugging, we also offer a standalone profiling agent binary named ebpf-profiler, and a local build of an OTel Collector profiling receiver binary (otelcol-ebpf-profiler). These binaries are not supported in any way, can be dropped in the future and should not be deployed in production.

Platform Requirements

For Linux, the following steps apply: 1. Build the agent for your current machine's architecture:

make agent

make agent TARGET_ARCH=arm64

Other OSes

Alternative Build (Without Docker)

make

Building otelcol-ebpf-profiler locally (Without Docker)

make otelcol-ebpf-profiler

make otelcol-ebpf-profiler TARGET_ARCH=arm64

See local.example.yaml for an example configuration.

Running

You can start the agent with the following command:

sudo ./ebpf-profiler -collection-agent=127.0.0.1:11000 -disable-tls

To start the OTel Collector profiling receiver, run:

sudo ./otelcol-ebpf-profiler --feature-gates=+service.profilesSupport --config cmd/otelcol-ebpf-profiler/local.example.yaml

The agent comes with a functional but work-in-progress / evolving implementation of the recently released Alpha OTel Profiles signal.

The agent loads the eBPF program and its maps, starts unwinding and reports captured traces to the backend.

Open Source Backends

As the OTel Profiles signal is still in development, mature production-ready backends have yet to emerge. The following open source projects can be used as backends:

• devfiler — to speed up development and

• Pyroscope — an open source continuous

Contributing

We welcome contributions of all kinds: bug reports, feature requests, documentation improvements and code. Everyone is welcome to open issues and send pull requests.

Before contributing, please read the contributing guidelines. They cover the development prerequisites, how to build and test your changes, the pull request workflow, and how PRs get reviewed and merged.

To understand how this project works and learn more about profiling, check out Profiling internals.

This project is developed by the OpenTelemetry Profiling SIG, which meets regularly. See the OpenTelemetry community repository for the meeting schedule, and join us in the #otel-profiles channel on the CNCF Slack for discussions and questions.

Maintainers

• Felix Geisendörfer, Datadog
• Christos Kalkanis, Elastic
• Timo Teräs

Approvers

• Roger Coll, Elastic
• Florian Lehner, Elastic
• Tim Rühsen

Emeritus

• Damien Mathieu, Approver
• Dmitry Filimonov, Maintainer
• Joel Höner, Approver

Become an Approver or a Maintainer

See the community membership document in OpenTelemetry community repo.

Legal

Licensing Information

This project is licensed under the Apache License 2.0 (Apache-2.0). Apache License 2.0

The eBPF source code is licensed under the GPL 2.0 license. GPL 2.0

Licenses of dependencies

To display a summary of the dependencies' licenses:

make legal