HUATUO 华佗

Kernel-wide Insight, Instant Observability, AutoTracing, Continuous Profiling

中文文档 · Quick Start · Documentation ·

What is HUATUO

HUATUO is a cloud-native operating system observability project open-sourced by DIDI and incubated under the CCF. It delivers kernel-level observability for general-purpose cloud-native computing, AI computing, and bare-metal infrastructure services. By integrating Linux kernel dynamic tracing technologies like kprobe, tracepoint, ftrace, and eBPF, HUATUO provides kernel-wide insights: finer-grained metrics, automatic context capture from kernel runtime, and intelligent tracing. Deployed at scale in Didi’s production environment, HUATUO plays a key role in troubleshooting system failures, enhancing the high availability and performance of cloud-native operating systems.

HUATUO is now listed in the CNCF Landscape.

Key Features

• Kernel-Wide Insight: Leverages BPF to maintain performance overhead below 1%, delivering full-stack, low-level observability insights into Linux kernel subsystems like MM, CPU scheduling, networking, and block I/O.
• Instant Observability: An event-driven runtime context capture mechanism that instruments kernel slow paths. It automatically triggers on events such as page faults, scheduling delays, generating detailed data for immediate analysis.
• AutoTracing: Employs automated snapshot retention to resolve performance jitters typical in cloud‑native and AI infrastructure environments, tackling issues such as CPU idle drops, CPU sys spikes, I/O surges, and Loadavg spikes.
• Continuous Profiling: A comprehensive and continuous performance profiling of the operating system and applications, covering CPU, Memory, I/O, and Locks. This drives business innovation and plays a key role in Chaos, HA and Stability Engineering.
• Heterogeneous Computing Hardware: Encompassing core system components such as CPUs, memory, PCIe interconnects, network adapters, and storage, alongside specialized AI accelerators including GPUs and NPUs.
• Ecosystem Integration: Integration with mainstream open-source observability stacks like Prometheus, Grafana, Pyroscope, and Elasticsearch. It automatically associates K8s container labels/annotations. Achieved through zero-instrumentation, kernel-level programming with eBPF, ensuring broad compatibility across hardware platforms and Linux distributions.

Big Picture

Ecosystem

Getting Started

• Quick Run

$ docker run --privileged --pid=host --cgroupns=host --network=host -v /sys:/sys -v /run:/run huatuo/huatuo-bamai:latest

To pull metrics from another terminal:

$ curl -s localhost:19704/metrics

• Quick Setup

$ docker compose --project-directory ./build/docker up

Once running, access the monitoring dashboard at http://localhost:3000.

• NOTE

Go Client

The public Go client is in client/. Node client files use the node_ prefix.

package main
import (
	"context"
	"encoding/json"
nodeapi "github.com/ccfos/huatuo/apis/v1/node"
	"github.com/ccfos/huatuo/client"
)
func updateNodeConfig(ctx context.Context, address client.NodeAddress) error {
	nodeClient, err := client.NewNode(&client.NodeConfig{
		BearerToken: "node-token",
	})
	if err != nil {
		return err
	}
	return nodeClient.UpdateConfig(ctx, address, &nodeapi.UpdateConfigRequest{
		Config: map[string]json.RawMessage{
			"Runtime.CPULimitCores": json.RawMessage("1.5"),
		},
	})
}

Kernel Versions

The project supports kernel version 4.18 and later. The following kernel and OS distribution are primarily tested.

| HUATUO | Kernel | OS | | :--- | :--- | :--- | | 1.0.0 | 4.18.x | CentOS 8.x | | 2.4.0 | 4.18.x | Rocky Linux 8.10 | | 1.0.0 | 5.4.x | OpenCloudOS V8 | | 1.0.0 | 5.4.x | Ubuntu 20.04.6 LTS | | 1.0.0 | 5.10.x | Anolis OS 8.10 | | 1.0.0 | 5.10.x | openEuler 22.03 LTS-SP4 | | 2.4.0 | 5.14.x | Rocky Linux 9.8 | | 1.0.0 | 5.15.x | Ubuntu 22.04.5 LTS | | 2.4.0 | 6.1.x | Debian 12 (bookworm) | | 1.0.0 | 6.6.x | Anolis OS 23.3 | | 1.0.0 | 6.6.x | OpenCloudOS V9 | | 1.0.0 | 6.6.x | openEuler 24.03 LTS-SP4 | | 1.0.0 | 6.8.x | Ubuntu 24.04.4 LTS | | 2.4.0 | 6.12.x | Debian 13 (trixie) | | 1.0.0 | 6.14.x | Fedora 42 | | 2.4.0 | 6.19.x | Fedora Linux 44 | | 2.3.0 | 7.0.x | Ubuntu 26.04 LTS |

Documentation

For more information, visit https://docs.huatuo.tech

Co-Building

• ❇️ We welcome all users, developers, companies, and organizations to use Huatuo, report bugs, submit feature requests, share best practices, and help us build a professional and vibrant open-source community.
• ❤️ Contributors

Contact Us

• WeChat Group and Official Account:

License

This project is open source under the Apache License 2.0. The BPF code is licensed under the GPL license.