Raspberry Pi Real-Time Object Detection and Tracking
* 1. Introduction
* 2. Dependency
+ 2.1. Packages requirement
+ 2.2. Hardware support
+ 2.3. Camera support
+ 2.4. Tested platforms
* 3. What's in this repository
+ 3.1. Camera Test
+ 3.2. Camera Streaming over HTTP
+ 3.3. Motion Detection
+ 3.4. Color-based Object Detection and Tracking
+ 3.5. Shape-based Object Detection and Tracking
+ 3.6. Feature-based Object Detection and Tracking (with ORB)
+ 3.7. Face Detection and Tracking
+ 3.8. Object Detection using YOLO
+ 3.9. Object Detection using TensorFlow Lite
* 4. How to Run
+ 4.1. Install the environment
+ 4.2. Run the scripts
+ 4.3. Change camera resolution
* 5. Benchmark
+ 5.1. Run the benchmark
+ 5.2. Results
* 6. Q&A
* License
Table of contents generated with markdown-toc
1. Introduction
Using a Raspberry Pi and a camera module for computer vision with OpenCV, YOLO, and TensorFlow Lite. The aim of this project is to provide a starting point for using RPi & CV in your own DIY / maker projects. Computer vision based on cameras is very powerful and will bring your project to the next level. This allows you to track complicated objects that would otherwise not be possible with other types of sensors (infrared, ultrasonic, LiDAR, etc).Note the code is based on Python and OpenCV meaning it is cross-platform. You can run this on other Linux-based platforms as well, e.g. x86/x64 PC, IPC, Jetson, Banana Pi, LattaPanda, BeagleBoard, etc.
2. Dependency
2.1. Packages requirement
This project is dependent on the following packages:- Python >= 3.6.9 (tested with Python 3.12 and 3.13)
- OpenCV-Python
- OpenCV-Contrib-Python
- NumPy
- SciPy
- Matplotlib
- Ultralytics (with CPU-only PyTorch on Raspberry Pi)
ai-edge-litert(optional, for the TensorFlow Lite example)
2.2. Hardware support
- Support Raspberry Pi 1 Model B, Raspberry Pi 2, Raspberry Pi Zero, and Raspberry Pi 3/4/5 (preferable)
- Nvidia Jetson
2.3. Camera support
- Any USB camera supported by Raspberry Pi
- The official RPi camera module is supported through
Picamera2.
Picamera2 automatically when it is installed (sudo apt install python3-picamera2 on Raspberry Pi OS), and fall back to a USB camera otherwise.
- On a Compute Module, the camera has to be declared in /boot/firmware/config.txt. For example, for a Camera Module v2 (IMX219) on the CAM1 connector, set camera_auto_detect=0 and add dtoverlay=imx219,cam1, then reboot. Use ov5647 for Camera Module v1, imx708 for v3, and imx477 for the HQ camera. Run rpicam-hello --list-cameras to check that the camera is detected.
- On Ubuntu, python3-picamera2 is not available from the Ubuntu archive, so use a USB camera.
- CSI cameras on Nvidia Jetson (e.g. Raspberry Pi Camera Module v2 / IMX219 on Jetson Nano) are supported through GStreamer.
cv2.VideoCapture(0). The scripts detect a Jetson CSI camera automatically and open it with an nvarguscamerasrc GStreamer pipeline instead (see src/utils/camera_utils.py). USB cameras on Jetson still use V4L2.
- This needs the OpenCV that ships with JetPack, which is built with GStreamer. Do not pip install opencv-python on Jetson: the pip wheels have no GStreamer support and will shadow JetPack's OpenCV.
- If the camera fails to open (e.g. Failed to create CaptureSession), restart the camera daemon: sudo systemctl restart nvargus-daemon.
2.4. Tested platforms
All the examples have been tested on the following setups:| Board | OS | Python | Camera |
|---|---|---|---|
| Raspberry Pi Compute Module 5 (RPi 5) | Ubuntu 24.04 LTS | 3.12 | Sony PlayStation Eye (USB) |
| Raspberry Pi Compute Module 4 (RPi 4) | Raspberry Pi OS (Debian 13 trixie) | 3.13 | Camera Module v2.1 (IMX219) on CAM1 |
| Raspberry Pi 3 Model B (RPi 3) | Raspberry Pi OS (Debian 13 trixie) | 3.13 | Camera Module v1 (OV5647) |
| Raspberry Pi Zero 2 W | Raspberry Pi OS Lite (Debian 13 trixie) | 3.13 | Camera Module v2 (IMX219) |
| Nvidia Jetson Nano 4GB | JetPack 4.6.1 (Ubuntu 18.04) | 3.8 (see 5.2) | — (processing FPS only) |
See 5. Benchmark for the frame rate of each example on these boards.
3. What's in this repository
Currently, the following applications are implemented:src/camera-test: Test if the camera is workingsrc/camera-stream: Stream the camera to a web browsersrc/motion-detection: Detect any motion in the framesrc/object-tracking-color: Object detection & tracking based on colorsrc/object-tracking-shape: Object detection & tracking based on shapesrc/object-tracking-feature: Object detection & tracking based on features using ORBsrc/face-detection: Face detection & trackingsrc/object-detection-yolo: Object detection using YOLO (RPi 3/4/5 only)src/object-detection-tflite: Object detection using TensorFlow Lite (RPi 3/4/5 only)
3.1. Camera Test
Test the RPi and OpenCV environment. You are expected to see a pop-up window that has video streams from your USB camera if everything is set up correctly. If the window does not appear, you need to check both of (1) your environment; (2) camera connection.3.2. Camera Streaming over HTTP
Stream the camera as MJPEG over HTTP, so you can watch it from any browser on the same network, without a display attached to the Pi. With the Raspberry Pi camera, frames are JPEG-encoded by the Pi's hardware encoder throughPicamera2, so CPU usage stays low. With a USB camera, frames are read and JPEG-encoded with OpenCV.
python3 src/camera-stream/cv_camera_stream.pyThen open http:// in a browser (the raw stream is at /stream.mjpg, which can also be opened in VLC). The script runs in the foreground by default. To keep it running in the background after you log out (like nohup), add --nohup:
python3 src/camera-stream/cv_camera_stream.py --nohupIt prints the process ID to stop it with (kill ), and writes its output to camera_stream.log in the current directory. The resolution and port are set by IMAGE_WIDTH, IMAGE_HEIGHT and PORT at the top of the script. If a USB camera does not support the requested resolution, it uses the closest one it has, and the script prints the resolution it actually streams at. Tested at about 30 FPS with a Camera Module v2 (IMX219) on a Compute Module 4 at 1280 x 720, and with a PlayStation Eye USB camera on a Compute Module 5 at 640 x 480.
Like the other examples, the script uses the Raspberry Pi camera when Picamera2 is installed and a camera is connected, and the USB camera otherwise. The Raspberry Pi camera does not need OpenCV. In the foreground, press Ctrl+C to stop it. The stream has no authentication, so anyone on your network can view it. Only one program can use the camera at a time, so stop the stream before running the other examples.
3.3. Motion Detection
Detect object movements in the camera frame and print a warning message when motion is sustained.Pipeline: the current and previous frames are converted to grayscale, Gaussian-blurred to suppress sensor noise, then compared with a per-pixel cv2.absdiff. The resulting diff is thresholded, dilated, and passed through cv2.findContours; only connected regions with area ≥ MIN_MOTION_AREA count as motion, and a debounce (MOTION_CONSECUTIVE_FRAMES) rejects one-frame transients. All thresholds are constants at the top of the script, so you can trade sensitivity for false-positive rate without touching the loop.
3.4. Color-based Object Detection and Tracking
Track an object based on its color in HSV and print its center position. You can choose your own color by clicking on the object of interest. Click multiple times on different points so a full color space is covered. You can hard code the parameters so you don't need to pick them again for the next run. The following demo shows how I track a Nintendo game controller in real-time:3.5. Shape-based Object Detection and Tracking
Detect and track geometric shapes in the frame:- Circles via
cv2.HoughCircles() - Triangles and rectangles via contour approximation (
cv2.findContours()+cv2.approxPolyDP())
MIN_CONTOUR_AREA pixels are filtered out to reduce noise; the epsilon used for polygon approximation and the Canny edge thresholds are configurable at the top of the script.
3.6. Feature-based Object Detection and Tracking (with ORB)
Detect and track an object using its feature. The algorithm I selected here is ORB (Oriented FAST and Rotated BRIEF) for its fast calculation speed to enable real-time detection. To use the example, please prepare an Arduino UNO board in hand (or replace thesimple.png).
3.7. Face Detection and Tracking
Detecting face using Harr Cascade detector.3.8. Object Detection using YOLO
Use YOLO (You Only Look Once) for object detection. Note this code is based on Ultralytics YOLO. The instruction can be found at their website: Quick Start Guide: Raspberry Pi with Ultralytics YOLO11. Double check if you need to use it in a commercialised project!3.9. Object Detection using Neural Network (TensorFlow Lite)
Use TensorFlow Lite to recognise objects.To use it, download the model and labels into the module directory:
cd src/object-detection-tflite
COCO labels
wget https://raw.githubusercontent.com/google-coral/test_data/master/coco_labels.txt
EfficientDet-Lite0 model (INT8 quantized, ~4MB, with built-in NMS post-processing)
wget -O efficientdet_lite0.tflite https://storage.googleapis.com/download.tensorflow.org/models/tflite/task_library/object_detection/rpi/lite-model_efficientdet_lite0_detection_metadata_1.tfliteThen install the TensorFlow Lite runtime and run:
pip install ai-edge-litert
python3 src/object-detection-tflite/object_detection_tflite.py4. How to Run
4.1. Install the environment
Run the installation script. It creates a virtual environment under ./venv and installs all Python dependencies from requirements.txt:
chmod +x install.sh
./install.shOn Raspberry Pi, the venv is created with --system-site-packages so that libcamera and picamera2 (installed in the system Python by Raspberry Pi OS) remain accessible from inside it. The script also installs the CPU-only build of PyTorch first, because the default PyPI build for ARM64 pulls in several GB of CUDA packages that a Raspberry Pi cannot use.
The face detection example uses the Haar cascade files from the libopencv-dev system package (installed by install.sh), since the OpenCV 5 pip wheels no longer ship them.
Activate the environment in your shell before running any of the scripts:
source venv/bin/activateThe TensorFlow Lite example (§3.9) has an extra optional dependency — install it into the same venv only if you plan to use it:
pip install ai-edge-litert4.2. Run the scripts
Run scripts in the/src folder by:
python3 src/$FOLDER_NAME$/$SCRIPT_NAME$.pyTo stop the code, press the ESC key on your keyboard.
4.3. Change camera resolution
Changing the resolution will significantly impact the FPS. By default it is set to be320 x 240, but you can change it to any value that your camera supports at the beginning of each source code (defined by IMAGE_WIDTH and IMAGE_HEIGHT). Typical resolutions are:
- 160 x 120
- 320 x 240
- 640 x 480 (480p)
- 1280 x 720 (720p)
- 1920 x 1080 (1080p: make sure your camera supports this high resolution.)
5. Benchmark
5.1. Run the benchmark
benchmark/benchmark_fps.py runs each example for a fixed time and reports its average frame rate as a Markdown table. It counts the frames each example prints, skips a warm-up period (model loading, camera start-up), and stops each example before starting the next one.
python3 benchmark/benchmark_fps.pyEach example is measured in two ways:
- Camera FPS: the example runs as normal on the live camera. This is the frame rate you see in practice, but it can never exceed the camera's frame rate (often 30 FPS), and several examples also wait 30 ms per frame in
cv2.waitKey(). - Processing FPS: a short clip is recorded from the camera first, then fed to each example as fast as it can take frames, with no windows and no
waitKey()delay. This shows how much processing headroom the board has, and the time per frame shows the cost of each algorithm. The examples' own code is not changed: the benchmark replaces the camera with the clip when it starts them.
Options:
--mode camera/--mode processing: measure only one of the two (default: both).--video clip.mp4: use the first frames of a video file for processing FPS instead of recording from the camera, so no camera is needed with--mode processing.--headless: do not open any windows for camera FPS, so it can be run over SSH without a display. Frame timing is kept the same as with a display. (Processing FPS never opens windows.)--only face yolo: run only some of the examples (motion-detection,color,shape,orb,face,tflite,yolo).--duration 20/--warmup 5: seconds measured per example, and seconds skipped after its first frame.--verbose: show the output of each example.
5.2. Results
Measured with the benchmark script (20 s per example after a 5 s warm-up, OpenCV 5.0). See 5.1. Run the benchmark to measure your own board.Processing FPS shows how fast each board runs the example's processing, without the camera's frame rate limit (time per frame in brackets). So that the boards can be compared directly, every board was fed the same 640 x 480 clip, recorded once with the PlayStation Eye (python3 benchmark/benchmark_fps.py --mode processing --video clip.avi):
| Example | RPi 5 (CM5) | RPi 4 (CM4) | RPi 3 | RPi Zero 2 W | Jetson Nano | |---|---|---|---|---|---| | Motion Detection | 2318 FPS (0.4 ms) | 645 FPS (1.6 ms) | 362 FPS (2.8 ms) | 321 FPS (3.1 ms) | 697 FPS (1.4 ms) | | Color-based Tracking | 1573 FPS (0.6 ms) | 440 FPS (2.3 ms) | 251 FPS (4.0 ms) | 222 FPS (4.5 ms) | 467 FPS (2.1 ms) | | Shape-based Tracking | 577 FPS (1.7 ms) | 202 FPS (4.9 ms) | 119 FPS (8.4 ms) | 110 FPS (9.1 ms) | 295 FPS (3.4 ms) | | Feature-based Tracking (ORB) | 53.6 FPS (18.7 ms) | 24.3 FPS (41.2 ms) | 13.0 FPS (76.9 ms) | 11.1 FPS (90.4 ms) | 23.9 FPS (41.8 ms) | | Face Detection | 164 FPS (6.1 ms) | 79.1 FPS (12.6 ms) | 42.5 FPS (23.5 ms) | 36.9 FPS (27.1 ms) | 75.7 FPS (13.2 ms) | | TensorFlow Lite (EfficientDet-Lite0) | 33.2 FPS (30.2 ms) | 12.3 FPS (81.6 ms) | 6.1 FPS (165 ms) | 5.1 FPS (194 ms) | 8.6 FPS (116 ms) | | YOLO11n (Ultralytics, 640x480) | 2.1 FPS (475 ms) | 1.3 FPS (765 ms) | 0.6 FPS (1578 ms) | out of memory | 7.1 FPS (140 ms, GPU) |
Notes on these results:
- RPi Zero 2 W and YOLO11n: YOLO11n needs about 430 MB of RAM to process one 640 x 480 frame (peak, measured on the RPi 3), but the Zero 2 W has only 512 MB in total (about 290 MB free with no desktop running), so it runs out of memory. Use the TensorFlow Lite example on the Zero 2 W instead.
- Jetson Nano: all the examples run on the CPU except YOLO11n, which runs on the GPU. JetPack 4.6 only has Python 3.6, which is too old for Ultralytics (3.8+) and
ai-edge-litert, and NVIDIA's CUDA builds of PyTorch for JetPack 4 are for Python 3.6 only. These results use a Python 3.8 virtual environment (python3.8, part of Ubuntu 18.04) with the CUDA 10.2 builds of PyTorch 1.11 and torchvision 0.12 that Ultralytics publish for JetPack 4 (the ones in their JetPack 4 Docker image),tflite-runtime2.13 andopencv-contrib-python5.0. As the pip OpenCV has no GStreamer, use JetPack's own OpenCV (Python 3.6) for a CSI camera (see 2.3. Camera support).
python3 benchmark/benchmark_fps.py --headless, RPi 5 and RPi 4 only):
| Example | RPi 5 (CM5) | RPi 4 (CM4) | |---|---|---| | Motion Detection | 30.0 FPS | 33.2 FPS | | Color-based Tracking | 29.1 FPS | 27.5 FPS | | Shape-based Tracking | 28.0 FPS | 26.4 FPS | | Feature-based Tracking (ORB) | 30.0 FPS | 22.6 FPS | | Face Detection | 26.6 FPS | 22.8 FPS | | TensorFlow Lite (EfficientDet-Lite0) | 30.0 FPS | 11.2 FPS | | YOLO11n (Ultralytics, 640x480) | 2.1 FPS | 0.8 FPS |
Camera FPS is limited by different things:
- The camera: the PlayStation Eye runs at up to 30 FPS, and the Camera Module v2 at about 33 FPS at 320 x 240. Motion detection reaches the camera's frame rate on both boards.
- The
cv2.waitKey()delay: color and shape tracking wait 33 ms per frame, which keeps them below 30 FPS. Face detection waits 30 ms per frame, which, with 12.7 ms of detection on the RPi 4, gives about 23 FPS. - The CPU: ORB and TensorFlow Lite on the RPi 4, and YOLO on both boards. For these, camera FPS is close to processing FPS.
6. Q&A
Q: Does this support Nvidia Jetson? A: Yes. I have tested with my Jetson Nano 4GB. Note that Jetson has its own repo now: jetson-object-detection.Q: Does this support the Raspberry Pi camera? A: This is implemented in [issue [#16]](https://github.com/automaticdai/rpi-object-detection/pull/16).
Q: Does this support Raspberry Pi 5? A: Yes. All examples have been tested on a Raspberry Pi Compute Module 5, which uses the same processor as the Raspberry Pi 5. See 2.4. Tested platforms.
Q: Can we run this project on Ubuntu?
A: Yes. All examples have been tested on Ubuntu 24.04 LTS on a Raspberry Pi Compute Module 5 with a USB camera. picamera2 is not available from the Ubuntu archive, so use a USB camera on Ubuntu.
Q: I am using virtual environment and get a message "no module called libcamera" issue
A: A simple solution would be (Reference):
python3 -m venv --system-site-packages env
Thanks VgerTest for [issue [#20]](https://github.com/automaticdai/rpi-object-detection/issues/20).