Demo 47: Deep learning - Computer vision with ESP32 and tensorflow.js - IoT Sharing

Hot

Monday, July 29, 2019

Demo 47: Deep learning - Computer vision with ESP32 and tensorflow.js

1. Introduction
- Deep learning is a hot topic and esp32 is a hot IoT MCU. Recently many applications related to computer vision are deployed on ESP32 (face detection, face recognition, ...). In this post I will show you a new approach to deploy Deep learning - Computer vision applications on ESP32 such as object classification (SqueezeNet), object detection and recognition (YOLOv3). After reading this post I am sure you can deploy hot network such as YOLOv3 on ESP32.
- My approach is using TensorFlow.js is a library for developing and training ML models in JavaScript, and deploying in browser.
- In this post, I will create a simple Deep learning - Computer vision application that is object classification using SqueezeNet. The esp32 will act as a webserver and when the client connect to it, a slideshow of objects will start and the objects will be classified using SqueezeNet.
You can do similar steps for YOLOv3, but instead of reading pictures from sdcard, you will use esp32-camera module and pass each camera frame to YOLOv3 model created by tensorflow.js.
 Figure: Deep learning - Computer vision with ESP32 and tensorflow.js
2. Hardware
You need a micro sdcard module as in Demo 7: How to use Arduino ESP32 to store data to microsdcard (Software SPI and Hardware SPI)
In this demo, I used Hardware SPI so please connect pins as below:
MICROSD CS    -      ESP32 IO5
MICROSD SCK   -     ESP32 IO18
MICROSD MOSI  -    ESP32 IO23
MICROSD MISO   -   ESP32 IO19
MICROSD Vcc   -      ESP32 3.3V
MICROSD GND   -    ESP32 GND
3. Software
- In order to make this demo, you have to review some demos:
Demo 12: How to turn the Arduino ESP32 into a Web Server
Demo 7: How to use Arduino ESP32 to store data to microsdcard (Software SPI and Hardware SPI)
- Knowledge of Jquery and Javascript.
- Material for deep learning part make by me: https://github.com/nhatuan84/tensorflowjs-squeezenet (or you can use the outputs that I generated)
- Knowledge of Deep learning. If you don't know, just follow me. I had another blog about Machine Leaning. It is here.
- I had to modify the webserver library in Demo 12: How to turn the Arduino ESP32 into a Web Server so that It can be used for this demo.
- Here are the steps:
  + Download all the resources here and unzip it.
  + Reinstall the ESP32WebServer.zip (in resources) for Arduino (you may uninstall old ESP32WebServer library).
  + Copy files: group1-shard1of2.bin, group1-shard2of2.bin, model.json, index.html, 1.jpg, 2.jpg, 3.jpg (in resources) to sdcard.
  + Create an Arduino project with code:
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
#include <WiFiClient.h>
#include <ESP32WebServer.h>
#include <WiFi.h>
#include <ESPmDNS.h>
#include "FS.h"
#include <SD.h>
#include <SPI.h>

const char* ssid = "ssid";
const char* password = "pass";

ESP32WebServer server(80);
File root;

void handleRoot() {
  root = SD.open("/index.html");
  if (root) {  
    /* respond the content of file to client by calling streamFile()*/
    size_t sent = server.streamFile(root, "text/html");
    /* close the file */
    root.close();
  } else {
    Serial.println("error opening index");
  }
}

bool loadFromSDCARD(String path){
  path.toLowerCase();
  Serial.println(path);
  String dataType = "text/plain";
  if(path.endsWith("/")) path += "/index.html";
  if(path.endsWith(".src")) path = path.substring(0, path.lastIndexOf("."));
  else if(path.endsWith(".jpg")) dataType = "image/jpeg";
  else if(path.endsWith(".txt")) dataType = "text/plain";
  else if(path.endsWith(".zip")) dataType = "application/zip";  
  if(path == "/favicon.ico")
    return false;
  
  root = SD.open((String("/") + path).c_str());
  if (!root){
    Serial.println("failed to open file");
    return false;
  }

  if (server.streamFile(root, dataType) != root.size()) {
    Serial.println("Sent less data than expected!");
  }

  root.close();
  return true;
}

void handleNotFound(){
  if(loadFromSDCARD(server.uri())) return;
  String message = "SDCARD Not Detected\n\n";
  message += "URI: ";
  message += server.uri();
  message += "\nMethod: ";
  message += (server.method() == HTTP_GET)?"GET":"POST";
  message += "\nArguments: ";
  message += server.args();
  message += "\n";
  for (uint8_t i=0; i<server.args(); i++){
    message += " NAME:"+server.argName(i) + "\n VALUE:" + server.arg(i) + "\n";
  }
  server.send(404, "text/plain", message);
  Serial.println(message);
}

void setup(void){
  Serial.begin(115200);
  WiFi.begin(ssid, password);
  Serial.println("");

  // Wait for connection
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.print("Connected to ");
  Serial.println(ssid);
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());
  
  //use IP or iotsharing.local to access webserver
  if (MDNS.begin("iotsharing")) {
    Serial.println("MDNS responder started");
  }
  if (!SD.begin()) {
    Serial.println("initialization failed!");
    return;
  }
  Serial.println("initialization done.");
  //handle uri  
  server.on("/", handleRoot);
  server.onNotFound(handleNotFound);

  server.begin();
  Serial.println("HTTP server started");
}

void loop(void){
  server.handleClient();
}
  + Open web browser and type the IP address from Terminal, you will see:

Figure: esp32-tensorflowjs-squeezenet prediction

19 comments:

markson said...

What sorts of apparatuses will be conveyed to help end client requirements for reports and examination?
data science course in pune

seoexpert said...

Nice Post...I have learn some new information.thanks for sharing.
ExcelR data analytics course in Pune | business analytics course | data scientist course in Pune

seoexpert said...

Such a very useful article. I have learn some new information.thanks for sharing.
data scientist course in mumbai

ravali said...

I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.
Data science course in mumbai

Manikanta said...

Such a very useful article. Very interesting to read this article. I have learn some new information.thanks for sharing. ExcelR

ExcelR Pune said...

Very nice blog here and thanks for post it.. Keep blogging...
ExcelR data science training

Priyanka said...

Attend The Analytics Course in Bangalore From ExcelR. Practical Analytics Course in Bangalore Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Analytics Course in Bangalore.
ExcelR Analytics Course in Bangalore

ravali said...

Great post I must say and thanks for the information. Education is definitely a sticky subject. However, it is still among the leading topics of our time. I appreciate your post and look forward to more.
ExcelR data science course in mumbai

Sankar said...

Great Article
Deep Learning Final Year Projects for CSE

JavaScript Training in Chennai

Project Centers in Chennai

JavaScript Training in Chennai

Excelrsolutions said...

This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck... data science course

ek said...

I like viewing web sites which comprehend the price of delivering the excellent useful resource free of charge. I truly adored reading your posting. Thank you!
Please check this ExcelR Courses in Business Analytics

Unknown said...

The information provided on the site is informative. Looking forward more such blogs. Thanks for sharing .
Artificial Inteligence course in Mysuru
AI Course in Mysuru

Unknown said...

The information provided on the site is informative. Looking forward more such blogs. Thanks for sharing .
Artificial Inteligence course in Mysuru
AI Course in Mysuru

Priyanka said...

Attend The Data Science Training Bangalore From ExcelR. Practical Data Science Training Bangalore Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Science Training Bangalore.
Data Science Training Bangalore

abid said...

Always so interesting to visit your site.What a great info, thank you for sharing. this will help me so much in my learning
360digitmg data science course in guwahati

Cho co said...

Data science trainings are provided on online platforms and coaching classes as well. With effective training, students can get well versed in algorithms like random forest, decision trees, naive bayes etc 360DigiTMG data science course in hyderabad

ek said...

wow, great, I was wondering how to cure acne naturally. and found your site by google, learned a lot, now I am a bit clear. I’ve bookmarked your site. keep us updated.
<a href="https://www.excelr.com/business-analytics-training-in-pune/”> Business Analytics Courses</a>
I have express a few of the articles on your website now, and I really like your style of blogging. I added it to my favorite’s blog site list and will be checking back soon…

shane lee said...

When a human can parallel process information, we call it memory. While talking about something, we remember something else. We say "by the way, I forgot to tell you" and then we continue on a different subject. artificial intelligence training in hyderabad

Data Science Bangalore said...

Very interesting blog. Many blogs I see these days do not really provide anything that attracts others, but believe me the way you interact is literally awesome.You can also check my articles as well.

Data Science In Banglore With Placements
Data Science Course In Bangalore
Data Science Training In Bangalore
Best Data Science Courses In Bangalore
Data Science Institute In Bangalore

Thank you..

Thường mất vài phút để quảng cáo xuất hiện trên trang nhưng thỉnh thoảng, việc này có thể mất đến 1 giờ. Hãy xem hướng dẫn triển khai mã của chúng tôi để biết thêm chi tiết. Ðã xong