Retrain the machine learning model

The model that we are using for speech recognition was trained on a dataset of one-second spoken commands called the Speech Commands Dataset. The dataset includes examples of the following ten different words:

yes, no, up, down, left, right, on, off, stop, go

While the model in this sample was originally trained to recognize “yes” and “no”, the TensorFlow Lite for Microcontrollers source contains scripts that make it easy to retrain the model to classify any other combination of these words.

We are going to use another pre-trained model to recognize “on” and “off”, instead. If you are interested in the full flow including the training of the model refer to the Supplementary information: model training section of this guide.

To build our new ML application we will now follow these steps:

  1. Download a pretrained model that has been trained and frozen using TensorFlow.
  2. Look at how the TensorFlow model gets converted to the TensorFlow Lite format.
  3. Convert the TensorFlow Lite model into a C source file.
  4. Modify the code and deploy to the device.

Note: Building TensorFlow and training the model will each take a couple of hours on an average computer. We will not perform this at this stage. For a full guide on how to do this, refer to the Supplementary information: model training section in this guide.

Convert the model

Starting from the trained model to obtain a converted model that can run on the controller itself, we need to run a conversion script: the TensorFlow Lite converter. This tool uses clever tricks to make our model as small and efficient as possible, and to convert it to a TensorFlow Lite FlatBuffer. To reduce the size of the model, we used a technique called quantization. All weights and activations in the model get converted from 32-bit floating point format to an 8-bit and fixed-point format, as you can see in the following command:

Convert the model to the TensorFlow Lite format code image

This conversion will not only reduce the size of the network, but will avoid floating point computations that are more computationally expensive.

To save time, we will skip this step and instead download the tiny_conv.tflite file from the guide page.

The final step in the process is to convert this model into a C file that we can drop into our Mbed project.

To do this conversion, we will use a tool called xxd. Issue the following command:

xxd -i tiny_conv.tflite > tiny_conv_micro_features_model_data.cc

Next, we need to update tiny_conv_micro_features_model_data.cc so that it is compatible with our code. First, open the file. The top two lines should look similar to the following code, although the exact variable name and hex values may be different:

unsigned char tiny_conv_tflite[] = {
  0x18, 0x00, 0x00, 0x00, 0x54, 0x46, 0x4c, 0x33, 0x00, 0x00, 0x0e, 0x00,

You need to add the include from the following snippet and change the variable declaration without changing the hex values:

#include "tensorflow/lite/experimental/micro/examples/micro_speech/micro_features/tiny_conv_micro_features_model_data.h"
const unsigned char g_tiny_conv_micro_features_model_data[] = {
  0x18, 0x00, 0x00, 0x00, 0x54, 0x46, 0x4c, 0x33, 0x00, 0x00, 0x0e, 0x00,

Next, go to the very bottom of the file and find the unsigned int variable.

unsigned int tiny_conv_tflite_len = 18216;

Change the declaration to the following code, but do not change the number assigned to it, even if your number is different from the one in this guide.

const int g_tiny_conv_micro_features_model_data_len = 18216;

Finally, save the file, then copy the tiny_conv_micro_features_model_data.cc file into the tensorflow/tensorflow/lite/experimental/micro/tools/make/gen/mbed_cortex-m4/prj/micro_speech/mbed/tensorflow/lite/experimental/micro/examples/micro_speech/micro_features directory.

Modify the device code

If you build and run your code now, your device should respond to the words “stop” and “go”. However, the code was written to assume that the words are “yes” and “no”. Let’s update the references and the user interface so that the appropriate words are printed.

First, go to the following directory: 

tensorflow/lite/experimental/micro/examples/micro_speech/ 

and open the file:

micro_features/micro_model_settings.cc

You will see the following category labels:

const char* kCategoryLabels[kCategoryCount] = {
    "silence",
    "unknown",
    "yes",
    "no",
};

The code uses this array to map the output of the model to the correct value. Because we specified our wanted_words as “on, off”in the training script, we should update this array to reflect these words in the same order. Edit the code so it appears as follows:

const char* kCategoryLabels[kCategoryCount] = {
    "silence",
    "unknown",
    "on",
    "off",
};

Next, we will update the code in command_responder.cc to reflect these new labels, modifying the if statements and the DisplayStringAt call:

void RespondToCommand(tflite::ErrorReporter* error_reporter,
                      int32_t current_time, const char* found_command,
                      uint8_t score, bool is_new_command) {
  if (is_new_command) {
    error_reporter->Report("Heard %s (%d) @%dms", found_command, score,
                           current_time);
    if(strcmp(found_command, "on") == 0) {
      lcd.Clear(0xFF0F9D58);
      lcd.DisplayStringAt(0, LINE(5), (uint8_t *)"Heard on", CENTER_MODE);
    } else if(strcmp(found_command, "off") == 0) {
      lcd.Clear(0xFFDB4437);
      lcd.DisplayStringAt(0, LINE(5), (uint8_t *)"Heard off", CENTER_MODE);
    } else if(strcmp(found_command, "unknown") == 0) {
      lcd.Clear(0xFFF4B400);
      lcd.DisplayStringAt(0, LINE(5), (uint8_t *)"Heard unknown", CENTER_MODE);
    } else {
      lcd.Clear(0xFF4285F4);
      lcd.DisplayStringAt(0, LINE(5), (uint8_t *)"Heard silence", CENTER_MODE);
    }
  }
}

Now that we have updated the code, go back to the mbed directory:

cd <path_to_tensorflow>/tensorflow/lite/experimental/micro/tools/make/gen/mbed_cortex-m4/prj/micro_speech/mbed

and run the following command to rebuild the project:

mbed compile -m DISCO_F746NG -t GCC_ARM

Finally, copy the binary to the USB storage of the device, using the same method that you used earlier. You should now be able to say “stop” and “go” to update the display.

Previous Next