Real-time Esp32-cam Color Tracking
Detect blobs of color and track their position in real-time, directly on the Esp32-cam

Color Blob Detection refers to the task of finding large chunks of the image characterized by the same (or similar) color.
You may want to perform this task to track the position of a marker object (e.g. a ball) and move a pan-tilt servo motor to align the camera in that direction. Or maybe you want to know where in the image a reference object is located to decide if an actuator should trigger.
No matter the actuation part, you first need to detect:
- if a target blob is in the frame
- where it is located
- how large it is
To perform this task on your own, you would need to be an expert programmer. Extracting color components from the frame in real-time, creating a mask of matching pixels and finding the largest blob require specialized algorithms (a.k.a. connected component).
But not all of the Arduino enthusiasts out there are expert programmers, and this is where I can help you. I wrapped all the hard stuff into an Arduino library that you can use with minimal effort.
This library only works with the Esp32 camera
1. Install the library
Open the Arduino IDE Library Manager and search for "EloquentEsp32Cam", then install the latest version.
2. Pick a target color
To understand how to use the color blob detection properly, you must know that the Esp32-cam takes the images in the YCbCr colorspace, instead of the more famous RGB. In this colorspace, you have 3 components:
- Y (or Luma), which represents the intensity of the pixels. For any practical use, the Luma channel is a grayscale version of the image
- Cb encodes a color component
- Cr encodes another color component
Since it is not smart (read fast) to convert each YCbCr pixel to RGB to perform color detection, you want to set your target color directly in the YCbCr colorspace.
How to do this?
You upload the following sketch to your Esp32-cam, open the browser at its IP address and click on the object you want to track. An alert with the Y, Cb, Cr values will appear (see image below).

If your router supports mDNS (most do), you can also reach the webpage at http://esp32cam.local

// 21_Color_Blob_Detection_HTTP.ino
/**
* Run an HTTP server to debug the ColorBlobDetector
*/
#define MAX_RESOLUTION_VGA
#define MAX_RECURSION_DEPTH 13
#include "esp32cam.h"
#include "esp32cam/JpegDecoder.h"
#include "esp32cam/apps/ColorBlobDetector.h"
#include "esp32cam/http/ColorBlobDetectorHTTP.h"
using namespace Eloquent::Esp32cam;
Cam cam;
JpegDecoder decoder;
Applications::ColorBlobDetector detector(0, 0, 0); // will configure from the URL
Http::ColorBlobDetectorHTTP http(cam, decoder, detector);
void setup() {
Serial.begin(115200);
delay(3000);
Serial.println("Init");
cam.aithinker();
cam.highQuality();
cam.vga();
cam.highestSaturation();
cam.disableAutomaticWhiteBalance();
cam.disableAutomaticExposureControl();
cam.disableGainControl();
/**
* Set detector tollerance
* The higher, the more shade of colors it will pick
*/
detector.tollerate(16);
/**
* Skip blob localization (slow) if not enough
* pixels match color
*/
detector.setMinArea(30 * 30);
while (!cam.begin())
Serial.println(cam.getErrorMessage());
while (!cam.connect("SSID", "PASSWORD"))
Serial.println(cam.getErrorMessage());
while (!http.begin())
Serial.println(http.getErrorMessage());
cam.mDNS("esp32cam");
Serial.println(http.getWelcomeMessage());
}
void loop() {
http.handle();
}
The above code is taken from my eBook "Mastering the Esp32 Camera", which holds many more advanced applications for the Esp32 camera.
It creates a color detector with a default target color of 0, 0, 0. It doesn't really matter which color you set here now, because we're going to update it from the browser.
Then it configures the camera with a fluent interface, connects to WiFi and starts an HTTP server for browser debugging.
3. Track color in the browser
After you got the Y, Cb, Cr values of the color you want to track, you need to update the ColorBlobDetector
with them. You could update the sketch and re-upload, but you can save this boring operation by updating
the values directly from the browser.
How?
Append the values to the URL of the Esp32 cam. For the values in the example image above, the URL becomes:
http://esp32cam.local?y=209&cb=193&cr=136
After the page load, you can put your target object in front of the camera and see the real-time tracking, as shown in the video below.

Notes
Real-time color tracking is not an easy task and you can see a major limitation in the demo video: color detector cannot pick different shades of the target color.
In the example above, 209, 193, 136
represent light yellow with artificial light illumination.
Dark yellow is very different from it, so the tracker only detects a portion of the object.
To get the best out of the tracker, you have to ensure a proper illumination setup is in place.
How to get blob coordinates
Once you're done experimenting in the browser and found your best configuration, you want to
be able to access the <top, left, right, bottom>
coordinates of the detected blob.
The detector
object gives you access to its blob
property with the following API:
-
detector.blob.top
-
detector.blob.left
-
detector.blob.right
-
detector.blob.bottom
as shown in the sketch below.
// 20_Color_Blob_Detector.ino
/**
* Detect blobs of given color
*/
#define MAX_RESOLUTION_VGA
#define MAX_RECURSION_DEPTH 13
#include "esp32cam.h"
#include "esp32cam/JpegDecoder.h"
#include "esp32cam/apps/ColorBlobDetector.h"
using namespace Eloquent::Esp32cam;
Cam cam;
JpegDecoder decoder;
Applications::ColorBlobDetector detector(8*16, 7*16, 4*16); // yellow-ish
void setup() {
Serial.begin(115200);
delay(3000);
Serial.println("Init");
cam.aithinker();
cam.highQuality();
cam.vga();
cam.highestSaturation();
cam.disableAutomaticWhiteBalance();
cam.disableAutomaticExposureControl();
cam.disableGainControl();
/**
* Set detector tollerance
* The higher, the more shade of colors it will pick
*/
detector.tollerate(16);
/**
* Skip blob localization (slow) if not enough
* pixels match color
*/
detector.setMinArea(30 * 30);
while (!cam.begin())
Serial.println(cam.getErrorMessage());
}
void loop() {
if (!cam.capture()) {
Serial.println(cam.getErrorMessage());
return;
}
if (!decoder.decode(cam)) {
Serial.println(decoder.getErrorMessage());
return;
}
/**
* Detect blob in frame
*/
if (detector.detect(decoder)) {
Serial.print("Blob detected from top-left ");
Serial.print(detector.blob.top);
Serial.print(", ");
Serial.print(detector.blob.left);
Serial.print(" to bottom-right ");
Serial.print(detector.blob.bottom);
Serial.print(", ");
Serial.println(detector.blob.right);
Serial.print("Blob detection run in ");
Serial.print(detector.getExecutionTimeInMillis());
Serial.println("ms");
}
else {
Serial.println(detector.getErrorMessage());
}
// while debugging, these may turn out useful
Serial.print(detector.maskCount);
Serial.println(" pixels match target color");
Serial.println(detector.toString());
detector.printTo(Serial);
}
If you thought real-time color blob detection was an advanced task, you would be right. But from now on you have access to this powerful tecnique, too, thanks to the EloquentEsp32Cam library.
And if you liked this project, you'll find many more in my eBook "Mastering the Esp32 Camera", so don't waste time and go get your copy!

Having troubles? Ask a question
Related posts
esp32-cam
Esp32 Camera Object Detection
The beginner-friendly guide to run Edge Impulse FOMO Object Detection on the Esp32 Camera with little effort
esp32-cam
Motion detection without PIR
Implement real-time motion detection from Esp32-cam frames like an expert in less than 30 minutes!
esp32-cam
Mastering the Esp32 Camera
Want to use your Esp32-cam like an expert? Not a programming genius? I can help!
esp32-cam
Esp32 and Arduino Person Detection
Detect humans in images using a pre-trained neural network that works out of the box on Esp32 and Arduino boards
esp32-cam
Real-Time image recognition
30-minutes walkthrough to create a working Esp32-cam image recognition system that runs at 80 FPS in 1 Kb of RAM