Demo 47: Deep learning - Computer vision with ESP32 and tensorflow.js

1. Introduction
- Deep learning is a hot topic and esp32 is a hot IoT MCU. Recently many applications related to computer vision are deployed on ESP32 (face detection, face recognition, ...). In this post I will show you a new approach to deploy Deep learning - Computer vision applications on ESP32 such as object classification (SqueezeNet), object detection and recognition (YOLOv3). After reading this post I am sure you can deploy hot network such as YOLOv3 on ESP32.
- My approach is using TensorFlow.js is a library for developing and training ML models in JavaScript, and deploying in browser.
- In this post, I will create a simple Deep learning - Computer vision application that is object classification using SqueezeNet. The esp32 will act as a webserver and when the client connect to it, a slideshow of objects will start and the objects will be classified using SqueezeNet.
You can do similar steps for YOLOv3, but instead of reading pictures from sdcard, you will use esp32-camera module and pass each camera frame to YOLOv3 model created by tensorflow.js.
 Figure: Deep learning - Computer vision with ESP32 and tensorflow.js
2. Hardware
You need a micro sdcard module as in Demo 7: How to use Arduino ESP32 to store data to microsdcard (Software SPI and Hardware SPI)
In this demo, I used Hardware SPI so please connect pins as below:
MICROSD CS    -      ESP32 IO5
MICROSD SCK   -     ESP32 IO18
MICROSD MOSI  -    ESP32 IO23
MICROSD MISO   -   ESP32 IO19
MICROSD Vcc   -      ESP32 3.3V
MICROSD GND   -    ESP32 GND
3. Software
- In order to make this demo, you have to review some demos:
Demo 12: How to turn the Arduino ESP32 into a Web Server
Demo 7: How to use Arduino ESP32 to store data to microsdcard (Software SPI and Hardware SPI)
- Knowledge of Jquery and Javascript.
- Material for deep learning part make by me: https://github.com/nhatuan84/tensorflowjs-squeezenet (or you can use the outputs that I generated)
- Knowledge of Deep learning. If you don't know, just follow me. I had another blog about Machine Leaning. It is here.
- I had to modify the webserver library in Demo 12: How to turn the Arduino ESP32 into a Web Server so that It can be used for this demo.
- Here are the steps:
  + Download all the resources here and unzip it.
  + Reinstall the ESP32WebServer.zip (in resources) for Arduino (you may uninstall old ESP32WebServer library).
  + Copy files: group1-shard1of2.bin, group1-shard2of2.bin, model.json, index.html, 1.jpg, 2.jpg, 3.jpg (in resources) to sdcard.
  + Create an Arduino project with code:
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
#include <WiFiClient.h>
#include <ESP32WebServer.h>
#include <WiFi.h>
#include <ESPmDNS.h>
#include "FS.h"
#include <SD.h>
#include <SPI.h>

const char* ssid = "ssid";
const char* password = "pass";

ESP32WebServer server(80);
File root;

void handleRoot() {
  root = SD.open("/index.html");
  if (root) {  
    /* respond the content of file to client by calling streamFile()*/
    size_t sent = server.streamFile(root, "text/html");
    /* close the file */
    root.close();
  } else {
    Serial.println("error opening index");
  }
}

bool loadFromSDCARD(String path){
  path.toLowerCase();
  Serial.println(path);
  String dataType = "text/plain";
  if(path.endsWith("/")) path += "/index.html";
  if(path.endsWith(".src")) path = path.substring(0, path.lastIndexOf("."));
  else if(path.endsWith(".jpg")) dataType = "image/jpeg";
  else if(path.endsWith(".txt")) dataType = "text/plain";
  else if(path.endsWith(".zip")) dataType = "application/zip";  
  if(path == "/favicon.ico")
    return false;
  
  root = SD.open((String("/") + path).c_str());
  if (!root){
    Serial.println("failed to open file");
    return false;
  }

  if (server.streamFile(root, dataType) != root.size()) {
    Serial.println("Sent less data than expected!");
  }

  root.close();
  return true;
}

void handleNotFound(){
  if(loadFromSDCARD(server.uri())) return;
  String message = "SDCARD Not Detected\n\n";
  message += "URI: ";
  message += server.uri();
  message += "\nMethod: ";
  message += (server.method() == HTTP_GET)?"GET":"POST";
  message += "\nArguments: ";
  message += server.args();
  message += "\n";
  for (uint8_t i=0; i<server.args(); i++){
    message += " NAME:"+server.argName(i) + "\n VALUE:" + server.arg(i) + "\n";
  }
  server.send(404, "text/plain", message);
  Serial.println(message);
}

void setup(void){
  Serial.begin(115200);
  WiFi.begin(ssid, password);
  Serial.println("");

  // Wait for connection
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.print("Connected to ");
  Serial.println(ssid);
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());
  
  //use IP or iotsharing.local to access webserver
  if (MDNS.begin("iotsharing")) {
    Serial.println("MDNS responder started");
  }
  if (!SD.begin()) {
    Serial.println("initialization failed!");
    return;
  }
  Serial.println("initialization done.");
  //handle uri  
  server.on("/", handleRoot);
  server.onNotFound(handleNotFound);

  server.begin();
  Serial.println("HTTP server started");
}

void loop(void){
  server.handleClient();
}
  + Open web browser and type the IP address from Terminal, you will see:

Figure: esp32-tensorflowjs-squeezenet prediction

Post a Comment

63 Comments

markson said…
What sorts of apparatuses will be conveyed to help end client requirements for reports and examination?
data science course in pune
seoexpert said…
Such a very useful article. I have learn some new information.thanks for sharing.
data scientist course in mumbai
ravali said…
I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.
Data science course in mumbai
Manikanta said…
Such a very useful article. Very interesting to read this article. I have learn some new information.thanks for sharing. ExcelR
Priyanka said…
Attend The Analytics Course in Bangalore From ExcelR. Practical Analytics Course in Bangalore Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Analytics Course in Bangalore.
ExcelR Analytics Course in Bangalore
ravali said…
Great post I must say and thanks for the information. Education is definitely a sticky subject. However, it is still among the leading topics of our time. I appreciate your post and look forward to more.
ExcelR data science course in mumbai
Excelrsolutions said…
This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck... data science course
ek said…
I like viewing web sites which comprehend the price of delivering the excellent useful resource free of charge. I truly adored reading your posting. Thank you!
Please check this ExcelR Courses in Business Analytics
Unknown said…
The information provided on the site is informative. Looking forward more such blogs. Thanks for sharing .
Artificial Inteligence course in Mysuru
AI Course in Mysuru
Unknown said…
The information provided on the site is informative. Looking forward more such blogs. Thanks for sharing .
Artificial Inteligence course in Mysuru
AI Course in Mysuru
Priyanka said…
Attend The Data Science Training Bangalore From ExcelR. Practical Data Science Training Bangalore Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Science Training Bangalore.
Data Science Training Bangalore
abid said…
Always so interesting to visit your site.What a great info, thank you for sharing. this will help me so much in my learning
360digitmg data science course in guwahati
Cho co said…
Data science trainings are provided on online platforms and coaching classes as well. With effective training, students can get well versed in algorithms like random forest, decision trees, naive bayes etc 360DigiTMG data science course in hyderabad
shane lee said…
When a human can parallel process information, we call it memory. While talking about something, we remember something else. We say "by the way, I forgot to tell you" and then we continue on a different subject. artificial intelligence training in hyderabad
Ziarulunirea.ro said…
Very interesting blog. Many blogs I see these days do not really provide anything that attracts others, but believe me the way you interact is literally awesome.You can also check my articles as well.

Data Science In Banglore With Placements
Data Science Course In Bangalore
Data Science Training In Bangalore
Best Data Science Courses In Bangalore
Data Science Institute In Bangalore

Thank you..
This is my first visit to your blog! We are a team of volunteers and new
initiatives in the same niche. Blog gave us useful information to work. You
have done an amazing job!

artificial intelligence training in Bangalore
lionelmessi said…
The Way Of Explaination Is Very Good And Nice Informative Topic You Have Choosen..
AWS Course Training in Hyderabad
saketh321 said…
As always your articles do inspire me. Every single detail you have posted was great. ExcelR Courses In Business Analytics
Really interesting post. I was looking for this kind of information and enjoyed reading this one. Keep posting. Thanks for sharing.
Data Science Course Training in Hyderabad
Priyanka said…
Attend The Course in Data Analytics From ExcelR. Practical Course in Data Analytics Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Course in Data Analytics.
Course in Data Analytics
I feel appreciative that I read this. It is useful and extremely educational and I truly took in a ton from it.
data scientist training
EXCELR said…
"Thanks for the Information.Interesting stuff to read.Great Article.
I enjoyed reading your post, very nice share.data science training"
Excelr Tuhin said…
I've read this post and if I could I desire to suggest you some interesting things or suggestions. Perhaps you could write next articles referring to this article. I want to read more things about it!
data scientist courses
Unknown said…
hello sir,
thanks for giving that type of information. I am really happy to visit your blog.Leading Solar company in Andhra Pradesh

I want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging enedevors.
Data Analytics Courses in Bangalore
i am glad to discover this page : i have to thank you for the time i spent on this especially great reading !! i really liked each part and also bookmarked you for new information on your site.
Data Scientist Course
technology said…
Without investing on any new software or hardware, it can provide you customized applications, to meet the requirements of it and provide you success in achieving your business goals. salesforce training in bangalore
Extremely overall quite fascinating post. I was searching for this sort of data and delighted in perusing this one. Continue posting. A debt of gratitude is in order for sharing. python course in delhi
data science said…
I was just examining through the web looking for certain information and ran over your blog.It shows how well you understand this subject. Bookmarked this page, will return for extra. data science course in vadodara
Unknown said…
Cool stuff you have and you keep overhaul every one of us

Digital Marketing Course
Great post I would like to thank you for the efforts you have made in writing this interesting and knowledgeable article.
data scientist training in malaysia
Ramesh Sampangi said…
Awesome blog. Informative and knowledgeable content. Keep sharing more stuff like this. Thank you for sharing this blog with us.
AI Patasala Courses in Data Science
This is really very nice post you shared, i like the post, thanks for sharing..
cyber security course
Anonymous said…
Thank you for providing this blog really appreciate the efforts you have taken into curating this article if you want you can check out data science course in bangalore they have a lot to offer with regards to data science in terms of training and live projects.
Anonymous said…
I want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging enedevors
business analytics course in varanasi
360DigiTMG said…
I was just browsing through the internet looking for some information and came across your blog. I am impressed by the information that you have on this blog. It shows how well you understand this subject. Bookmarked this page, will come back for more.
data scientist course in hyderabad
It's like you've got the point right, but forgot to include your readers. Maybe you should think about it from different angles.

Data Scientist Course in Bangalore
Thanks for posting this info. I just want to let you know that I just check out your site and I find it very interesting and informative. I can't wait to read lots of your posts.
cyber security training malaysia
ksowjanya said…
Data Science is a booming field with ample job opportunities. Start your preparation today with 360DigiTMG and become a Data Scientist in the right way.
business analytics course in pondicherry
Develop technical skills and become an expert in analyzing large sets of data by enrolling for the Best Data Science course in Bangalore. Gain in-depth knowledge in Data Visualization, Statistics, and Predictive Analytics along with the two famous programming languages and Python. Learn to derive valuable insights from data using skills of Data Mining, Statistics, Machine Learning, Network Analysis, etc, and apply the skills you will learn in your final Capstone project to get recognized by potential employers.


Data Science Course in Bangalore
Data Science handles structured and unstructured and data that is generated at an unprecedented rate every day. Anyone with a strong statistical background and an analytical mindset enjoys the challenges of big data that involves building data models and software platforms along with creating attractive visualizations and machine learning algorithms. Sign up for the Data Science courses in Bangalore with Placements and get access to resume building and mock interviews that will help you get placed with top brands in this field.


Data Science in Bangalore

nagabhushan said…
Attempting to express profound gratitude won't just be satisfactory, for the fabulous clearness in your creation. I will in a brief instant get your rss channel to stay instructed with respect to any updates. data analytics course in pune
Get a comprehensive overview of Data Science and learn all the essential skills including collecting, modeling, and interpreting data. Register with Data Science institute Bangalore and build a strong foundation for a career where you will be involved in uncovering valuable information for your organization. Learn Python, Machine Learning, Big Data, Deep Learning, and Analytics to take center stage in Data Science.

Data Science Course in Bangalore with Placement
Embark on a journey to achieve your professional goals by enrolling in the Data Scientist course in Bangalore. Learn the skills of collecting, extracting, analyzing, preparing, visualizing, and presenting results to make valuable decisions. Master the concepts of data science through hands-on projects and case studies to learn the latest trends and skills in this field.

Data Scientist Course in Delhi
Anjali said…
It has been used in detecting diseases, and in this way, physicians can make crucial decisions about it.data science course in chandigarh
deekshitha said…
If you have a statistics background, then R will be an excellent choice for you, and you can learn it quickly and accurately because R is based on statistical concepts.data science training in varanasi
data analytics course in patna said…
Explore what the best Data Science training institute is offering you. Learn advanced technologies from the best industry experts.
<a href="https://360digitmg.com/india/data-analytics-certification-training-course-in-patna>data analytics course in patna</a>
data science training in borivali said…
You have to learn all the required skills to transform raw data into a form that will improve the organization.

data science training in borivali
Deep learning is a subset of machine learning that uses algorithms to learn from data in a way that mimics the way the human brain works.
deekshitha said…
Data Science Course in Nasik Best Training Institute Fees, Duration, Certification Online Classes Available There are no limitations to learning courses and one can indeed get multiple certificates provided he she completes the courses and clears the exam. PassedB.Sc. Degree from a honored University as defined by UGC, with at least 45 marks( 40 marks in case of candidates belonging to reserved order) and passed 10 2 examination with Mathematics as a subject. Curated by Hadoop experts, this Big Data Analytics course covers everything you need to gain proficiency in this field.data science course institute in nagpur
Arun Sharma said…
I have been through the whole content of this blog which is very informative and knowledgeable stuff, So i would like to visit again.
Rise Food Beach Noida Extension
techooly said…
This is a great blog posting and very useful. I really appreciate the research you put into it..
Techooly
TannuGowda said…
The real-world case studies included in this article make it relatable and demonstrate the practical application of best data analytics courses in chennai.