Color classification with TinyML

This project is a clone of one of the first tutorial for TensorFlow on Arduino, which uses the Arduino Nano 33 BLE Sense built-in colorimeter to classify different fruits based on their color.

It is a pretty simple project, but we'll replicate it to get familiar with the Eloquent TinyML workflow for classification tasks for static datatypes.

Features definition

The Nano 33 BLE Sense colorimeter (or any other RGB color sensor like the TCS3200) can detect the RGB values of the object it is pointed at.

We will use these 3 values as input features for our Machine Learning model.

Even though many classifier will work fine with the raw feature values (in the range 0 to 255), we will introduce a pre-processing pipeline to show how you can modify the data to make it a better fit for later classification.

Data collection

Our dataset will be made of 3-4 fruits (or any other kind of object, really), each from a different color.

We'll take 20 samples of each object, from different angles and distances to make sure we capture all the variability of each color.

To create a robust model, it would be ideal if you could change the room illumination while capturing the samples, so that our model is insensitive to light variations too.

To collect the samples, load the following sketch to your Arduino Nano 33 BLE Sense and run the Python code below.

If you don't have an Arduino Nano 33 BLE Sense and want to use your own color sensor (e.g. TCS3200), update the get RGB values line accordingly.

// file Fruit_identification_data_collection.ino


#include <Arduino_APDS9960.h>


void setup() {
  Serial.begin(9600);

  // check if sensor is ok
  while (!APDS.begin()) {
    Serial.println("Error initializing APDS9960 sensor.");
  }
}

void loop() {
  int r, g, b;
  
  // await for sensor data
  while (!APDS.colorAvailable())
    delay(5);

  // get RGB values
  APDS.readColor(r, g, b);

  // print values to Serial
  Serial.print("RGB: ");
  Serial.print(r);
  Serial.print(",");
  Serial.print(g);
  Serial.print(",");
  Serial.println(b);

  delay(300);
}

The Python code below makes use of the SerialCollector class. It is a utility class that makes it really easy to collect data from your development board without having to manually move data around.

Pre-processing pipeline

It is often the case that your dataset needs to be pre-processed in some way to achieve good classification results.

Many classifiers, for example, require that all the features lie in a given (or the same) range. If you were to classify a dataset of people where the height ranges from 0 to 2 m and the weight from 0 to 100 kg, your model would complain about such a difference.

More complex datasets will also have more complex pre-processing requirements and you will have to pipe multiple pre-processing steps to transform the raw values into meaningful features.

This is where the pipeline concept comes in.

You can think of a pipeline as a list of steps that operate in waterfall: every step receives as input the output of the previous one.

They can perform any sort of manipulation to the data and no one is (or should be) aware of the others, nor has any dependency on them.

For the sake of this project, we'll use an minimalistic pipeline made of a single step: a MinMaxScaler.

A MinMaxScaler maps each feature from its original range (from 0 to 255 in this case) to a new range (usually from 0 to 1).

Machine Learning model

For such a simple dataset, any Machine Learning model will work fine. In this example, we'll use a Decision Tree, since it's fast and needs little resources.

Port to Arduino

We need to port two piece of code to Arduino:

1. the pre-processing pipeline
2. the classifier

Both of them implement the to_arduino_file() function to perform this task.

// file Fruit_identification_classification.ino

#include <Arduino_APDS9960.h>
#include "Pipeline.h"
#include "Classifier.h"


void setup() {
  Serial.begin(9600);

  // check if sensor is ok
  while (!APDS.begin()) {
    Serial.println("Error initializing APDS9960 sensor.");
  }
}

void loop() {
  int r, g, b;
  
  // await for sensor data
  while (!APDS.colorAvailable())
    delay(5);

  // get RGB values
  APDS.readColor(r, g, b);

  // print values to Serial
  Serial.print("RGB: ");
  Serial.print(r);
  Serial.print(",");
  Serial.print(g);
  Serial.print(",");
  Serial.println(b);
    
  // perform feature extraction
  float features[] = {r, g, b};
    
  if (!pipeline.transform(features))
      return;

  // perform classification on pipeline result
  Serial.print("Predicted color: ");
  Serial.println(tree.predictLabel(pipeline.X));

  delay(300);
}

Well done!

If you were able to follow the project from start to finish, you got a good overview of what a TinyML project looks like:

1. define your features
2. collect data
3. train a model to classify that data
4. deploy the model back to your board

These steps used to be very difficult to implement for those who are not expert C++ programmers, but now it's much easier thanks to the Eloquent ecosystem of libraries.