Demo 47: Deep learning - Computer vision with ESP32 and tensorflow.js

1. Introduction

- Deep learning is a hot topic and esp32 is a hot IoT MCU. Recently many applications related to computer vision are deployed on ESP32 (face detection, face recognition, ...). In this post I will show you a new approach to deploy Deep learning - Computer vision applications on ESP32 such as object classification (SqueezeNet), object detection and recognition (YOLOv3). After reading this post I am sure you can deploy hot network such as YOLOv3 on ESP32.

- My approach is using TensorFlow.js is a library for developing and training ML models in JavaScript, and deploying in browser.

- In this post, I will create a simple Deep learning - Computer vision application that is object classification using SqueezeNet. The esp32 will act as a webserver and when the client connect to it, a slideshow of objects will start and the objects will be classified using SqueezeNet.
You can do similar steps for YOLOv3, but instead of reading pictures from sdcard, you will use esp32-camera module and pass each camera frame to YOLOv3 model created by tensorflow.js.

 Figure: Deep learning - Computer vision with ESP32 and tensorflow.js

2. Hardware
You need a micro sdcard module as in Demo 7: How to use Arduino ESP32 to store data to microsdcard (Software SPI and Hardware SPI)
In this demo, I used Hardware SPI so please connect pins as below:
MICROSD CS    -      ESP32 IO5
MICROSD SCK   -     ESP32 IO18
MICROSD MOSI  -    ESP32 IO23
MICROSD MISO   -   ESP32 IO19
MICROSD Vcc   -      ESP32 3.3V
MICROSD GND   -    ESP32 GND

3. Software

- In order to make this demo, you have to review some demos:
Demo 12: How to turn the Arduino ESP32 into a Web Server
Demo 7: How to use Arduino ESP32 to store data to microsdcard (Software SPI and Hardware SPI)
- Knowledge of Jquery and Javascript.
- Material for deep learning part make by me: https://github.com/nhatuan84/tensorflowjs-squeezenet (or you can use the outputs that I generated)
- Knowledge of Deep learning. If you don't know, just follow me. I had another blog about Machine Leaning. It is here.
- I had to modify the webserver library in Demo 12: How to turn the Arduino ESP32 into a Web Server so that It can be used for this demo.
- Here are the steps:
  + Download all the resources here and unzip it.
  + Reinstall the ESP32WebServer.zip (in resources) for Arduino (you may uninstall old ESP32WebServer library).
  + Copy files: group1-shard1of2.bin, group1-shard2of2.bin, model.json, index.html, 1.jpg, 2.jpg, 3.jpg (in resources) to sdcard.
  + Create an Arduino project with code:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105

#include <WiFiClient.h> #include <ESP32WebServer.h> #include <WiFi.h> #include <ESPmDNS.h> #include "FS.h" #include <SD.h> #include <SPI.h> const char* ssid = "ssid"; const char* password = "pass"; ESP32WebServer server(80); File root; void handleRoot() { root = SD.open("/index.html"); if (root) { /* respond the content of file to client by calling streamFile()*/ size_t sent = server.streamFile(root, "text/html"); /* close the file */ root.close(); } else { Serial.println("error opening index"); } } bool loadFromSDCARD(String path){ path.toLowerCase(); Serial.println(path); String dataType = "text/plain"; if(path.endsWith("/")) path += "/index.html"; if(path.endsWith(".src")) path = path.substring(0, path.lastIndexOf(".")); else if(path.endsWith(".jpg")) dataType = "image/jpeg"; else if(path.endsWith(".txt")) dataType = "text/plain"; else if(path.endsWith(".zip")) dataType = "application/zip"; if(path == "/favicon.ico") return false; root = SD.open((String("/") + path).c_str()); if (!root){ Serial.println("failed to open file"); return false; } if (server.streamFile(root, dataType) != root.size()) { Serial.println("Sent less data than expected!"); } root.close(); return true; } void handleNotFound(){ if(loadFromSDCARD(server.uri())) return; String message = "SDCARD Not Detected\n\n"; message += "URI: "; message += server.uri(); message += "\nMethod: "; message += (server.method() == HTTP_GET)?"GET":"POST"; message += "\nArguments: "; message += server.args(); message += "\n"; for (uint8_t i=0; i<server.args(); i++){ message += " NAME:"+server.argName(i) + "\n VALUE:" + server.arg(i) + "\n"; } server.send(404, "text/plain", message); Serial.println(message); } void setup(void){ Serial.begin(115200); WiFi.begin(ssid, password); Serial.println(""); // Wait for connection while (WiFi.status() != WL_CONNECTED) { delay(500); Serial.print("."); } Serial.println(""); Serial.print("Connected to "); Serial.println(ssid); Serial.print("IP address: "); Serial.println(WiFi.localIP()); //use IP or iotsharing.local to access webserver if (MDNS.begin("iotsharing")) { Serial.println("MDNS responder started"); } if (!SD.begin()) { Serial.println("initialization failed!"); return; } Serial.println("initialization done."); //handle uri server.on("/", handleRoot); server.onNotFound(handleNotFound); server.begin(); Serial.println("HTTP server started"); } void loop(void){ server.handleClient(); }

  + Open web browser and type the IP address from Terminal, you will see:

Figure: esp32-tensorflowjs-squeezenet prediction