LF Edge eKuiper - An edge lightweight IoT data analytics software
Overview
LF Edge eKuiper is a lightweight IoT data analytics and stream processing engine running on resource-constraint edge devices. The major goal for eKuiper is to provide a streaming software framework (similar to Apache Flink) in edge side. eKuiper's rule engine allows user to provide either SQL based or graph based (similar to Node-RED) rules to create IoT edge analytics applications within few minutes.
!arch
User scenarios
It can be run at various IoT edge user scenarios, such as,
- Real-time processing of production line data in the IIoT
- Gateway of connected vehicle analyze the data from CAN in IoV
- Real-time analysis of wind turbines and smart bulk energy storage data in smart energy
Features
- Lightweight
- Cross-platform
- Data analysis support
- Highly extensible
Source, Functions and Sink with Golang or Python.
- Source: allows users to add more data source for analytics. - Sink: allows users to send analysis result to different customized systems. - UDF functions: allow users to add customized functions for data analysis (for example, AI/ML function invocation)
- Management
- Integration with EMQX products
Quick start
Community
Join our Slack, and then join ekuiper or ekuiper-user channel.
Meeting
Subscribe to community events calendar.
Weekly community meeting at Friday 10:30AM GMT+8:
Contributing
Thank you for your contribution! Please refer to the CONTRIBUTING.md for more information.Security
Please report security vulnerabilities privately. Do not open a public GitHub issue. See SECURITY.md.Performance test result
MQTT throughput test
- Using JMeter MQTT plugin to send IoT data to EMQX Broker, such as:
{"temperature": 10, "humidity" : 90}, the value of temperature and humidity are random integer between 0 - 100. - eKuiper subscribe from EMQX Broker, and analyze data with SQL:
SELECT * FROM demo WHERE temperature > 50 - The analysis result are wrote to local file by using file sink plugin.
Ubuntu18.04 | 10k | sys+user: 25% | 20M |
EdgeX throughput test
- A Go application is written to send data to ZeroMQ message bus, the data is as
{
"Device": "demo", "Created": 000, …
"readings":
[
{"Name": "Temperature", value: "30", "Created":123 …},
{"Name": "Humidity", value: "20", "Created":456 …}
]
}- eKuiper subscribe from EdgeX ZeroMQ message bus, and analyze data with
SELECT * FROM demo WHERE temperature > 50. 90% of data will be filtered by the rule.
- The analysis result are sent to nop sink, so all the result data will be
ignored.
| | Message # per second | CPU usage | Memory usage |
|------------------------------------------------|----------------------|---------------|--------------|
| AWS t2.micro( 1 Core * 1 GB)
Ubuntu18.04 | 11.4 k | sys+user: 75% | 32M |
Max number of rules support
- 8000 rules with 800 message/second in total
- Configurations
- 2 core * 4GB memory in AWS
- Ubuntu
- Resource usage
- Memory: 89% ~ 72%
- CPU: 25%
- 400KB - 500KB / rule
- Rule
- Source: MQTT
- SQL: SELECT temperature FROM source WHERE temperature > 20 (90% data are filtered)
- Sink: Log
Multiple rules with shared source instance
- 300 rules with a shared MQTT stream instance.
- 500 messages/second in the MQTT source
- 150,000 message processing per second in total
- Configurations:
- 2 Core * 2GB memory in AWS
- Ubuntu
- Resource usage
- Memory: 95MB
- CPU: 50%
- Rule
- Source: MQTT
- SQL: SELECT temperature FROM source WHERE temperature > 20, (90% data are filtered)
- Sink: 90% nop and 10% MQTT
To run the benchmark by yourself, please check the instruction.
Documents
Check out the latest document in official website.
Build from source
Preparation
- Go version >= 1.25
Compile
+ Binary:
- Binary:
$ make
- Binary files that support EdgeX:
$ make build_with_edgex
- Minimal binary file with core runtime only:
$ make build_core
+ Packages:
$ make pkg
- Packages:
$ make pkg
- Package files that support EdgeX:
$ make pkg_with_edgex
+ Docker images:
$ make docker`
> Docker images support EdgeX by default
Prebuilt binaries are provided in the release assets. If using os or arch which does not have prebuilt binaries, please use cross-compilation, refer to this doc.
During compilation, features can be selected through go build tags so that users can build a customized product with only the desired feature set to reduce binary size. This is critical when the target deployment environment has resource constraint. Please refer to features for more detail.
