Low-code Esp32 and Arduino Person Detection

Person detection in Esp32 and Arduino cam video streams is a common task nowadays.

Only a few years ago this would have been simply impossible to do because of lack of both hardware (Arduino boards used to feature a mediocre 16 kb RAM) and software (neural networks support for embedded systems was simply inexisent).

As of today, lots of things changed and person detection has even become one the get started projects for TinyML.

Nonetheless, many tutorials on the subject are a bit convoluted and pretty hard to follow, let alone modify and integrate into your very own project.

It doesn't have to be that way.

Esp32 and Arduino person detection in realtime from a video stream can be implemented effectively with almost no-code on the user side. It is sufficient to install the EloquentTinyML Arduino library and call a single function.

I'm going to show you how you can add person detection on your very own Arduino project in 5 minutes!

Install the EloquentTinyML library

The whole magic of no-code person detection happens inside the EloquentTinyML Arduino library.

This is a wrapper around TensorFlow Lite for Microcontrollers and removes all the boilerplate code you will find in other tutorials on the internet. It makes using neural networks on Arduino a breeze.

So you have to install it.

Install the EloquentArduino library (optional)

To grab the frames from the Esp32 or Arduino camera, we'll leverage the EloquentArduino library.

This library implements lots of useful routines, including camera management.

If you're more familiar with direct camera configuration, feel free to skip this step.

Be sure to install EloquentArduino version 2.1.3.

Run person detection on Esp32 and Arduino

This part is easy, you don't have to write much code nor configure many options.

Esp32 camera

#include "EloquentTinyML.h"
#include "eloquent.h"
#include "eloquent_tinyml/tensorflow/person_detection.h"
// replace 'm5wide' with your own model
// possible values are 'aithinker', 'eye', 'm5stack', 'm5wide', 'wrover'
#include "eloquent/vision/camera/m5wide.h"

const uint16_t imageWidth = 160;
const uint16_t imageHeight = 120;

Eloquent::TinyML::TensorFlow::PersonDetection<imageWidth, imageHeight> personDetector;

void setup() {
    Serial.begin(115200);
    delay(3000);

    // configure camera
    camera.grayscale();
    camera.qqvga();

    while (!camera.begin())
        Serial.println("Cannot init camera");

    // configure a threshold for "robust" person detection
    // if no threshold is set, "person" would be detected everytime
    // person_score > not_person_score, even if just by 1
    // by trial and error, considering that scores range from 0 to 255,
    // a threshold of 190-200 dramatically reduces the number of false positives
    personDetector.setDetectionAbsoluteThreshold(190);
    personDetector.begin();

    // abort if an error occurred on the detector
    while (!personDetector.isOk()) {
        Serial.print("Detector init error: ");
        Serial.println(personDetector.getErrorMessage());
    }
}

void loop() {
    if (!camera.capture()) {
        Serial.println("Camera capture error");
        delay(1000);
        return;
    }

    bool isPersonInFrame = personDetector.detectPerson(camera.buffer);

    if (!personDetector.isOk()) {
        Serial.print("Person detector detection error: ");
        Serial.println(personDetector.getErrorMessage());
        delay(1000);
        return;
    }

    Serial.println(isPersonInFrame ? "Person detected" : "No person detected");
    Serial.print("\t > It took ");
    Serial.print(personDetector.getElapsedTime());
    Serial.println("ms to detect");
    Serial.print("\t > Person score: ");
    Serial.println(personDetector.getPersonScore());
    Serial.print("\t > Not person score: ");
    Serial.println(personDetector.getNotPersonScore());
}

It takes ~5 seconds to run the detection on a single frame

Arduino Portenta Vision Shield

#include "EloquentTinyML.h"
#include "eloquent.h"
#include "eloquent_tinyml/tensorflow/person_detection.h"
#include "eloquent/vision/camera/portenta.h"

const uint16_t imageWidth = 320;
const uint16_t imageHeight = 240;

Eloquent::TinyML::TensorFlow::PersonDetection<imageWidth, imageHeight> personDetector;

void setup() {
    Serial.begin(115200);
    delay(3000);

    // configure camera
    camera.grayscale();

    while (!camera.begin())
        Serial.println("Cannot init camera");

    // configure a threshold for "robust" person detection
    // if no threshold is set, "person" would be detected everytime
    // person_score > not_person_score, even if just by 1
    // by trial and error, considering that scores range from 0 to 255,
    // a threshold of 190-200 dramatically reduces the number of false positives
    personDetector.setDetectionAbsoluteThreshold(190);
    personDetector.begin();

    // abort if an error occurred on the detector
    while (!personDetector.isOk()) {
        Serial.print("Detector init error: ");
        Serial.println(personDetector.getErrorMessage());
    }
}

void loop() {
    if (!camera.capture()) {
        Serial.println("Camera capture error");
        delay(1000);
        return;
    }

    bool isPersonInFrame = personDetector.detectPerson(camera.buffer.getBuffer());

    if (!personDetector.isOk()) {
        Serial.print("Person detector detection error: ");
        Serial.println(personDetector.getErrorMessage());
        delay(1000);
        return;
    }

    Serial.println(isPersonInFrame ? "Person detected" : "No person detected");
    Serial.print("\t > It took ");
    Serial.print(personDetector.getElapsedTime());
    Serial.println("ms to detect");
    Serial.print("\t > Person score: ");
    Serial.println(personDetector.getPersonScore());
    Serial.print("\t > Not person score: ");
    Serial.println(personDetector.getNotPersonScore());
}

It takes ~0.6 seconds to run the detection on a single frame

Customize

Customizing the sketch is easy. You can add your own logic to be perfomed when a person is detected.

A single if statement will do the job.

A few use cases you can implement:

• a smart door bell that activates when it recognizes a person in front of it
• a smart relay that turns the lights on when you're back at home
• count the number of people passing in front of your camera

Feel free to let me know what you implemented with the form below.

If you're stuck, don't hesitate to ask for help.

Want to become an Expert with Vision TinyML?

I'm writing an eBook that will make you as expert on the Esp32-cam as I am (you agree I am, right?). It contains A LOT of contents, such as:

• Color blob detection
• Passing people counter
• Line crossing counter
• Person detection
• Fade detection
• Object recognition
• TensorFlow
• Edge Impulse

By buying the pre-sale here, you will have

• 50% discount on retail price (19,99 $ when done)
• early access to chapters drafts and code
• partecipate in polls and feature requests to shape the contents of the book while I'm writing
• join the Discord community with all the other people who bought this pre-sale

Click the image below to read more.