Demo 47: Deep learning - Computer vision with ESP32 and tensorflow.js

1. Introduction
- Deep learning is a hot topic and esp32 is a hot IoT MCU. Recently many applications related to computer vision are deployed on ESP32 (face detection, face recognition, ...). In this post I will show you a new approach to deploy Deep learning - Computer vision applications on ESP32 such as object classification (SqueezeNet), object detection and recognition (YOLOv3). After reading this post I am sure you can deploy hot network such as YOLOv3 on ESP32.
- My approach is using TensorFlow.js is a library for developing and training ML models in JavaScript, and deploying in browser.
- In this post, I will create a simple Deep learning - Computer vision application that is object classification using SqueezeNet. The esp32 will act as a webserver and when the client connect to it, a slideshow of objects will start and the objects will be classified using SqueezeNet.
You can do similar steps for YOLOv3, but instead of reading pictures from sdcard, you will use esp32-camera module and pass each camera frame to YOLOv3 model created by tensorflow.js.
 Figure: Deep learning - Computer vision with ESP32 and tensorflow.js
2. Hardware
You need a micro sdcard module as in Demo 7: How to use Arduino ESP32 to store data to microsdcard (Software SPI and Hardware SPI)
In this demo, I used Hardware SPI so please connect pins as below:
MICROSD CS    -      ESP32 IO5
MICROSD SCK   -     ESP32 IO18
MICROSD MOSI  -    ESP32 IO23
MICROSD MISO   -   ESP32 IO19
MICROSD Vcc   -      ESP32 3.3V
MICROSD GND   -    ESP32 GND
3. Software
- In order to make this demo, you have to review some demos:
Demo 12: How to turn the Arduino ESP32 into a Web Server
Demo 7: How to use Arduino ESP32 to store data to microsdcard (Software SPI and Hardware SPI)
- Knowledge of Jquery and Javascript.
- Material for deep learning part make by me: https://github.com/nhatuan84/tensorflowjs-squeezenet (or you can use the outputs that I generated)
- Knowledge of Deep learning. If you don't know, just follow me. I had another blog about Machine Leaning. It is here.
- I had to modify the webserver library in Demo 12: How to turn the Arduino ESP32 into a Web Server so that It can be used for this demo.
- Here are the steps:
  + Download all the resources here and unzip it.
  + Reinstall the ESP32WebServer.zip (in resources) for Arduino (you may uninstall old ESP32WebServer library).
  + Copy files: group1-shard1of2.bin, group1-shard2of2.bin, model.json, index.html, 1.jpg, 2.jpg, 3.jpg (in resources) to sdcard.
  + Create an Arduino project with code:
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
#include <WiFiClient.h>
#include <ESP32WebServer.h>
#include <WiFi.h>
#include <ESPmDNS.h>
#include "FS.h"
#include <SD.h>
#include <SPI.h>

const char* ssid = "ssid";
const char* password = "pass";

ESP32WebServer server(80);
File root;

void handleRoot() {
  root = SD.open("/index.html");
  if (root) {  
    /* respond the content of file to client by calling streamFile()*/
    size_t sent = server.streamFile(root, "text/html");
    /* close the file */
    root.close();
  } else {
    Serial.println("error opening index");
  }
}

bool loadFromSDCARD(String path){
  path.toLowerCase();
  Serial.println(path);
  String dataType = "text/plain";
  if(path.endsWith("/")) path += "/index.html";
  if(path.endsWith(".src")) path = path.substring(0, path.lastIndexOf("."));
  else if(path.endsWith(".jpg")) dataType = "image/jpeg";
  else if(path.endsWith(".txt")) dataType = "text/plain";
  else if(path.endsWith(".zip")) dataType = "application/zip";  
  if(path == "/favicon.ico")
    return false;
  
  root = SD.open((String("/") + path).c_str());
  if (!root){
    Serial.println("failed to open file");
    return false;
  }

  if (server.streamFile(root, dataType) != root.size()) {
    Serial.println("Sent less data than expected!");
  }

  root.close();
  return true;
}

void handleNotFound(){
  if(loadFromSDCARD(server.uri())) return;
  String message = "SDCARD Not Detected\n\n";
  message += "URI: ";
  message += server.uri();
  message += "\nMethod: ";
  message += (server.method() == HTTP_GET)?"GET":"POST";
  message += "\nArguments: ";
  message += server.args();
  message += "\n";
  for (uint8_t i=0; i<server.args(); i++){
    message += " NAME:"+server.argName(i) + "\n VALUE:" + server.arg(i) + "\n";
  }
  server.send(404, "text/plain", message);
  Serial.println(message);
}

void setup(void){
  Serial.begin(115200);
  WiFi.begin(ssid, password);
  Serial.println("");

  // Wait for connection
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.print("Connected to ");
  Serial.println(ssid);
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());
  
  //use IP or iotsharing.local to access webserver
  if (MDNS.begin("iotsharing")) {
    Serial.println("MDNS responder started");
  }
  if (!SD.begin()) {
    Serial.println("initialization failed!");
    return;
  }
  Serial.println("initialization done.");
  //handle uri  
  server.on("/", handleRoot);
  server.onNotFound(handleNotFound);

  server.begin();
  Serial.println("HTTP server started");
}

void loop(void){
  server.handleClient();
}
  + Open web browser and type the IP address from Terminal, you will see:

Figure: esp32-tensorflowjs-squeezenet prediction

Post a Comment

45 Comments

markson said…
What sorts of apparatuses will be conveyed to help end client requirements for reports and examination?
data science course in pune
ravali said…
I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.
Data science course in mumbai
Manikanta said…
Such a very useful article. Very interesting to read this article. I have learn some new information.thanks for sharing. ExcelR
ravali said…
Great post I must say and thanks for the information. Education is definitely a sticky subject. However, it is still among the leading topics of our time. I appreciate your post and look forward to more.
ExcelR data science course in mumbai
ek said…
I like viewing web sites which comprehend the price of delivering the excellent useful resource free of charge. I truly adored reading your posting. Thank you!
Please check this ExcelR Courses in Business Analytics
Unknown said…
The information provided on the site is informative. Looking forward more such blogs. Thanks for sharing .
Artificial Inteligence course in Mysuru
AI Course in Mysuru
Unknown said…
The information provided on the site is informative. Looking forward more such blogs. Thanks for sharing .
Artificial Inteligence course in Mysuru
AI Course in Mysuru
abid said…
Always so interesting to visit your site.What a great info, thank you for sharing. this will help me so much in my learning
360digitmg data science course in guwahati
Cho co said…
Data science trainings are provided on online platforms and coaching classes as well. With effective training, students can get well versed in algorithms like random forest, decision trees, naive bayes etc 360DigiTMG data science course in hyderabad
shane lee said…
When a human can parallel process information, we call it memory. While talking about something, we remember something else. We say "by the way, I forgot to tell you" and then we continue on a different subject. artificial intelligence training in hyderabad
saketh321 said…
As always your articles do inspire me. Every single detail you have posted was great. ExcelR Courses In Business Analytics
I feel appreciative that I read this. It is useful and extremely educational and I truly took in a ton from it.
data scientist training
EXCELR said…
"Thanks for the Information.Interesting stuff to read.Great Article.
I enjoyed reading your post, very nice share.data science training"
Excelr Tuhin said…
I've read this post and if I could I desire to suggest you some interesting things or suggestions. Perhaps you could write next articles referring to this article. I want to read more things about it!
data scientist courses
Unknown said…
hello sir,
thanks for giving that type of information. I am really happy to visit your blog.Leading Solar company in Andhra Pradesh

i am glad to discover this page : i have to thank you for the time i spent on this especially great reading !! i really liked each part and also bookmarked you for new information on your site.
Data Scientist Course
technology said…
Without investing on any new software or hardware, it can provide you customized applications, to meet the requirements of it and provide you success in achieving your business goals. salesforce training in bangalore
Unknown said…
Cool stuff you have and you keep overhaul every one of us

Digital Marketing Course
Great post I would like to thank you for the efforts you have made in writing this interesting and knowledgeable article.
data scientist training in malaysia
Ramesh Sampangi said…
Awesome blog. Informative and knowledgeable content. Keep sharing more stuff like this. Thank you for sharing this blog with us.
AI Patasala Courses in Data Science
This is really very nice post you shared, i like the post, thanks for sharing..
cyber security course
360DigiTMG said…
I was just browsing through the internet looking for some information and came across your blog. I am impressed by the information that you have on this blog. It shows how well you understand this subject. Bookmarked this page, will come back for more.
data scientist course in hyderabad
Thanks for posting this info. I just want to let you know that I just check out your site and I find it very interesting and informative. I can't wait to read lots of your posts.
cyber security training malaysia
ksowjanya said…
Data Science is a booming field with ample job opportunities. Start your preparation today with 360DigiTMG and become a Data Scientist in the right way.
business analytics course in pondicherry
Develop technical skills and become an expert in analyzing large sets of data by enrolling for the Best Data Science course in Bangalore. Gain in-depth knowledge in Data Visualization, Statistics, and Predictive Analytics along with the two famous programming languages and Python. Learn to derive valuable insights from data using skills of Data Mining, Statistics, Machine Learning, Network Analysis, etc, and apply the skills you will learn in your final Capstone project to get recognized by potential employers.


Data Science Course in Bangalore
Data Science handles structured and unstructured and data that is generated at an unprecedented rate every day. Anyone with a strong statistical background and an analytical mindset enjoys the challenges of big data that involves building data models and software platforms along with creating attractive visualizations and machine learning algorithms. Sign up for the Data Science courses in Bangalore with Placements and get access to resume building and mock interviews that will help you get placed with top brands in this field.


Data Science in Bangalore

nagabhushan said…
Attempting to express profound gratitude won't just be satisfactory, for the fabulous clearness in your creation. I will in a brief instant get your rss channel to stay instructed with respect to any updates. data analytics course in pune
Embark on a journey to achieve your professional goals by enrolling in the Data Scientist course in Bangalore. Learn the skills of collecting, extracting, analyzing, preparing, visualizing, and presenting results to make valuable decisions. Master the concepts of data science through hands-on projects and case studies to learn the latest trends and skills in this field.

Data Scientist Course in Delhi
Anjali said…
It has been used in detecting diseases, and in this way, physicians can make crucial decisions about it.data science course in chandigarh
data analytics course in patna said…
Explore what the best Data Science training institute is offering you. Learn advanced technologies from the best industry experts.
<a href="https://360digitmg.com/india/data-analytics-certification-training-course-in-patna>data analytics course in patna</a>
data science training in borivali said…
You have to learn all the required skills to transform raw data into a form that will improve the organization.

data science training in borivali
Deep learning is a subset of machine learning that uses algorithms to learn from data in a way that mimics the way the human brain works.
deekshitha said…
Data Science Course in Nasik Best Training Institute Fees, Duration, Certification Online Classes Available There are no limitations to learning courses and one can indeed get multiple certificates provided he she completes the courses and clears the exam. PassedB.Sc. Degree from a honored University as defined by UGC, with at least 45 marks( 40 marks in case of candidates belonging to reserved order) and passed 10 2 examination with Mathematics as a subject. Curated by Hadoop experts, this Big Data Analytics course covers everything you need to gain proficiency in this field.data science course institute in nagpur
Arun Sharma said…
Glad to visit your blog. Thanks for great post that you share to us...

kusumagowda said…
I found the section on model evaluation and validation in this article to be informative and comprehensive.data analyst course in chennai
Arun Sharma said…
Its really wonderful and watchable. I like to share it with all my friends and hope they will definitely like it.
chitra reddy said…
I found the tips and best practices shared in this article to be actionable and beneficial for improving data analytics workflows data analytics course in chennai.
Anonymous said…
This is very nice post.I’m happy to see some great article on your site. Ziyyara Edutech spoken English classes in Saudi Arabia provide the platform for you to achieve linguistic excellence.
For Book a demo now English language courses in Saudi Arabia