Real-time Esp32-cam Color Tracking

Detect blobs of color and track their position in real-time, directly on the Esp32-cam

Color Blob Detection refers to the task of finding large chunks of the image characterized by the same (or similar) color.

You may want to perform this task to track the position of a marker object (e.g. a ball) and move a pan-tilt servo motor to align the camera in that direction. Or maybe you want to know where in the image a reference object is located to decide if an actuator should trigger.

No matter the actuation part, you first need to detect:

if a target blob is in the frame
where it is located
how large it is

To perform this task on your own, you would need to be an expert programmer. Extracting color components from the frame in real-time, creating a mask of matching pixels and finding the largest blob require specialized algorithms (a.k.a. connected component).

But not all of the Arduino enthusiasts out there are expert programmers, and this is where I can help you. I wrapped all the hard stuff into an Arduino library that you can use with minimal effort.

This library only works with the Esp32 camera

1. Install the library

Open the Arduino IDE Library Manager and search for "EloquentEsp32Cam", then install the latest version.

2. Pick a target color

To understand how to use the color blob detection properly, you must know that the Esp32-cam takes the images in the YCbCr colorspace, instead of the more famous RGB. In this colorspace, you have 3 components:

Y (or Luma), which represents the intensity of the pixels. For any practical use, the Luma channel is a grayscale version of the image
Cb encodes a color component
Cr encodes another color component

Since it is not smart (read fast) to convert each YCbCr pixel to RGB to perform color detection, you want to set your target color directly in the YCbCr colorspace.

How to do this?

You upload the following sketch to your Esp32-cam, open the browser at its IP address and click on the object you want to track. An alert with the Y, Cb, Cr values will appear (see image below).

If your router supports mDNS (most do), you can also reach the webpage at http://esp32cam.local

 // 21_Color_Blob_Detection_HTTP.ino

/**
 * Run an HTTP server to debug the ColorBlobDetector
 */
#define MAX_RESOLUTION_VGA
#define MAX_RECURSION_DEPTH 13

#include "esp32cam.h"
#include "esp32cam/JpegDecoder.h"
#include "esp32cam/apps/ColorBlobDetector.h"
#include "esp32cam/http/ColorBlobDetectorHTTP.h"

using namespace Eloquent::Esp32cam;

Cam cam;
JpegDecoder decoder;
Applications::ColorBlobDetector detector(0, 0, 0); // will configure from the URL
Http::ColorBlobDetectorHTTP http(cam, decoder, detector);



void setup() {
    Serial.begin(115200);
    delay(3000);
    Serial.println("Init");

    cam.aithinker();
    cam.highQuality();
    cam.vga();
    cam.highestSaturation();
    cam.disableAutomaticWhiteBalance();
    cam.disableAutomaticExposureControl();
    cam.disableGainControl();

    /**
     * Set detector tollerance
     * The higher, the more shade of colors it will pick
     */
    detector.tollerate(16);
    /**
     * Skip blob localization (slow) if not enough
     * pixels match color
     */
    detector.setMinArea(30 * 30);

    while (!cam.begin())
        Serial.println(cam.getErrorMessage());

    while (!cam.connect("SSID", "PASSWORD"))
        Serial.println(cam.getErrorMessage());

    while (!http.begin())
        Serial.println(http.getErrorMessage());

    cam.mDNS("esp32cam");
    Serial.println(http.getWelcomeMessage());
}


void loop() {
    http.handle();
}

The above code is taken from my eBook "Mastering the Esp32 Camera", which holds many more advanced applications for the Esp32 camera.

It creates a color detector with a default target color of 0, 0, 0. It doesn't really matter which color you set here now, because we're going to update it from the browser.

Then it configures the camera with a fluent interface, connects to WiFi and starts an HTTP server for browser debugging.

3. Track color in the browser

After you got the Y, Cb, Cr values of the color you want to track, you need to update the ColorBlobDetector with them. You could update the sketch and re-upload, but you can save this boring operation by updating the values directly from the browser.

How?

Append the values to the URL of the Esp32 cam. For the values in the example image above, the URL becomes:

http://esp32cam.local?y=209&cb=193&cr=136

After the page load, you can put your target object in front of the camera and see the real-time tracking, as shown in the video below.

Notes

Real-time color tracking is not an easy task and you can see a major limitation in the demo video: color detector cannot pick different shades of the target color.

In the example above, 209, 193, 136 represent light yellow with artificial light illumination. Dark yellow is very different from it, so the tracker only detects a portion of the object.

To get the best out of the tracker, you have to ensure a proper illumination setup is in place.

How to get blob coordinates

Once you're done experimenting in the browser and found your best configuration, you want to be able to access the <top, left, right, bottom> coordinates of the detected blob.

The detector object gives you access to its blob property with the following API:

detector.blob.top
detector.blob.left
detector.blob.right
detector.blob.bottom

as shown in the sketch below.

 // 20_Color_Blob_Detector.ino

/**
 * Detect blobs of given color
 */
#define MAX_RESOLUTION_VGA
#define MAX_RECURSION_DEPTH 13

#include "esp32cam.h"
#include "esp32cam/JpegDecoder.h"
#include "esp32cam/apps/ColorBlobDetector.h"

using namespace Eloquent::Esp32cam;

Cam cam;
JpegDecoder decoder;
Applications::ColorBlobDetector detector(8*16, 7*16, 4*16); // yellow-ish


void setup() {
    Serial.begin(115200);
    delay(3000);
    Serial.println("Init");

    cam.aithinker();
    cam.highQuality();
    cam.vga();
    cam.highestSaturation();
    cam.disableAutomaticWhiteBalance();
    cam.disableAutomaticExposureControl();
    cam.disableGainControl();

    /**
     * Set detector tollerance
     * The higher, the more shade of colors it will pick
     */
    detector.tollerate(16);
    /**
     * Skip blob localization (slow) if not enough
     * pixels match color
     */
    detector.setMinArea(30 * 30);

    while (!cam.begin())
        Serial.println(cam.getErrorMessage());
}


void loop() {
    if (!cam.capture()) {
        Serial.println(cam.getErrorMessage());
        return;
    }

    if (!decoder.decode(cam)) {
        Serial.println(decoder.getErrorMessage());
        return;
    }

    /**
     * Detect blob in frame
     */
    if (detector.detect(decoder)) {
        Serial.print("Blob detected from top-left ");
        Serial.print(detector.blob.top);
        Serial.print(", ");
        Serial.print(detector.blob.left);
        Serial.print(" to bottom-right ");
        Serial.print(detector.blob.bottom);
        Serial.print(", ");
        Serial.println(detector.blob.right);
        Serial.print("Blob detection run in ");
        Serial.print(detector.getExecutionTimeInMillis());
        Serial.println("ms");
    }
    else {
        Serial.println(detector.getErrorMessage());
    }

    // while debugging, these may turn out useful
    Serial.print(detector.maskCount);
    Serial.println(" pixels match target color");
    Serial.println(detector.toString());
    detector.printTo(Serial);


}

If you thought real-time color blob detection was an advanced task, you would be right. But from now on you have access to this powerful tecnique, too, thanks to the EloquentEsp32Cam library.

And if you liked this project, you'll find many more in my eBook "Mastering the Esp32 Camera", so don't waste time and go get your copy!