Demo 47: Deep learning - Computer vision with ESP32 and tensorflow.js

1. Introduction
- Deep learning is a hot topic and esp32 is a hot IoT MCU. Recently many applications related to computer vision are deployed on ESP32 (face detection, face recognition, ...). In this post I will show you a new approach to deploy Deep learning - Computer vision applications on ESP32 such as object classification (SqueezeNet), object detection and recognition (YOLOv3). After reading this post I am sure you can deploy hot network such as YOLOv3 on ESP32.
- My approach is using TensorFlow.js is a library for developing and training ML models in JavaScript, and deploying in browser.
- In this post, I will create a simple Deep learning - Computer vision application that is object classification using SqueezeNet. The esp32 will act as a webserver and when the client connect to it, a slideshow of objects will start and the objects will be classified using SqueezeNet.
You can do similar steps for YOLOv3, but instead of reading pictures from sdcard, you will use esp32-camera module and pass each camera frame to YOLOv3 model created by tensorflow.js.
 Figure: Deep learning - Computer vision with ESP32 and tensorflow.js
2. Hardware
You need a micro sdcard module as in Demo 7: How to use Arduino ESP32 to store data to microsdcard (Software SPI and Hardware SPI)
In this demo, I used Hardware SPI so please connect pins as below:
MICROSD CS    -      ESP32 IO5
MICROSD SCK   -     ESP32 IO18
MICROSD MOSI  -    ESP32 IO23
MICROSD MISO   -   ESP32 IO19
MICROSD Vcc   -      ESP32 3.3V
MICROSD GND   -    ESP32 GND
3. Software
- In order to make this demo, you have to review some demos:
Demo 12: How to turn the Arduino ESP32 into a Web Server
Demo 7: How to use Arduino ESP32 to store data to microsdcard (Software SPI and Hardware SPI)
- Knowledge of Jquery and Javascript.
- Material for deep learning part make by me: https://github.com/nhatuan84/tensorflowjs-squeezenet (or you can use the outputs that I generated)
- Knowledge of Deep learning. If you don't know, just follow me. I had another blog about Machine Leaning. It is here.
- I had to modify the webserver library in Demo 12: How to turn the Arduino ESP32 into a Web Server so that It can be used for this demo.
- Here are the steps:
  + Download all the resources here and unzip it.
  + Reinstall the ESP32WebServer.zip (in resources) for Arduino (you may uninstall old ESP32WebServer library).
  + Copy files: group1-shard1of2.bin, group1-shard2of2.bin, model.json, index.html, 1.jpg, 2.jpg, 3.jpg (in resources) to sdcard.
  + Create an Arduino project with code:
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
#include <WiFiClient.h>
#include <ESP32WebServer.h>
#include <WiFi.h>
#include <ESPmDNS.h>
#include "FS.h"
#include <SD.h>
#include <SPI.h>

const char* ssid = "ssid";
const char* password = "pass";

ESP32WebServer server(80);
File root;

void handleRoot() {
  root = SD.open("/index.html");
  if (root) {  
    /* respond the content of file to client by calling streamFile()*/
    size_t sent = server.streamFile(root, "text/html");
    /* close the file */
    root.close();
  } else {
    Serial.println("error opening index");
  }
}

bool loadFromSDCARD(String path){
  path.toLowerCase();
  Serial.println(path);
  String dataType = "text/plain";
  if(path.endsWith("/")) path += "/index.html";
  if(path.endsWith(".src")) path = path.substring(0, path.lastIndexOf("."));
  else if(path.endsWith(".jpg")) dataType = "image/jpeg";
  else if(path.endsWith(".txt")) dataType = "text/plain";
  else if(path.endsWith(".zip")) dataType = "application/zip";  
  if(path == "/favicon.ico")
    return false;
  
  root = SD.open((String("/") + path).c_str());
  if (!root){
    Serial.println("failed to open file");
    return false;
  }

  if (server.streamFile(root, dataType) != root.size()) {
    Serial.println("Sent less data than expected!");
  }

  root.close();
  return true;
}

void handleNotFound(){
  if(loadFromSDCARD(server.uri())) return;
  String message = "SDCARD Not Detected\n\n";
  message += "URI: ";
  message += server.uri();
  message += "\nMethod: ";
  message += (server.method() == HTTP_GET)?"GET":"POST";
  message += "\nArguments: ";
  message += server.args();
  message += "\n";
  for (uint8_t i=0; i<server.args(); i++){
    message += " NAME:"+server.argName(i) + "\n VALUE:" + server.arg(i) + "\n";
  }
  server.send(404, "text/plain", message);
  Serial.println(message);
}

void setup(void){
  Serial.begin(115200);
  WiFi.begin(ssid, password);
  Serial.println("");

  // Wait for connection
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println("");
  Serial.print("Connected to ");
  Serial.println(ssid);
  Serial.print("IP address: ");
  Serial.println(WiFi.localIP());
  
  //use IP or iotsharing.local to access webserver
  if (MDNS.begin("iotsharing")) {
    Serial.println("MDNS responder started");
  }
  if (!SD.begin()) {
    Serial.println("initialization failed!");
    return;
  }
  Serial.println("initialization done.");
  //handle uri  
  server.on("/", handleRoot);
  server.onNotFound(handleNotFound);

  server.begin();
  Serial.println("HTTP server started");
}

void loop(void){
  server.handleClient();
}
  + Open web browser and type the IP address from Terminal, you will see:

Figure: esp32-tensorflowjs-squeezenet prediction

Post a Comment

71 Comments

markson said…
What sorts of apparatuses will be conveyed to help end client requirements for reports and examination?
data science course in pune
seoexpert said…
Such a very useful article. I have learn some new information.thanks for sharing.
data scientist course in mumbai
ravali said…
I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.
Data science course in mumbai
Manikanta said…
Such a very useful article. Very interesting to read this article. I have learn some new information.thanks for sharing. ExcelR
ExcelR Pune said…
Very nice blog here and thanks for post it.. Keep blogging...
ExcelR data science training
Priyanka said…
Attend The Analytics Course in Bangalore From ExcelR. Practical Analytics Course in Bangalore Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Analytics Course in Bangalore.
ExcelR Analytics Course in Bangalore
ravali said…
Great post I must say and thanks for the information. Education is definitely a sticky subject. However, it is still among the leading topics of our time. I appreciate your post and look forward to more.
ExcelR data science course in mumbai
Excelrsolutions said…
This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck... data science course
ek said…
I like viewing web sites which comprehend the price of delivering the excellent useful resource free of charge. I truly adored reading your posting. Thank you!
Please check this ExcelR Courses in Business Analytics
Unknown said…
The information provided on the site is informative. Looking forward more such blogs. Thanks for sharing .
Artificial Inteligence course in Mysuru
AI Course in Mysuru
Unknown said…
The information provided on the site is informative. Looking forward more such blogs. Thanks for sharing .
Artificial Inteligence course in Mysuru
AI Course in Mysuru
Priyanka said…
Attend The Data Science Training Bangalore From ExcelR. Practical Data Science Training Bangalore Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Data Science Training Bangalore.
Data Science Training Bangalore
abid said…
Always so interesting to visit your site.What a great info, thank you for sharing. this will help me so much in my learning
360digitmg data science course in guwahati
Cho co said…
Data science trainings are provided on online platforms and coaching classes as well. With effective training, students can get well versed in algorithms like random forest, decision trees, naive bayes etc 360DigiTMG data science course in hyderabad
ek said…
wow, great, I was wondering how to cure acne naturally. and found your site by google, learned a lot, now I am a bit clear. I’ve bookmarked your site. keep us updated.
<a href="https://www.excelr.com/business-analytics-training-in-pune/”> Business Analytics Courses</a>
I have express a few of the articles on your website now, and I really like your style of blogging. I added it to my favorite’s blog site list and will be checking back soon…
shane lee said…
When a human can parallel process information, we call it memory. While talking about something, we remember something else. We say "by the way, I forgot to tell you" and then we continue on a different subject. artificial intelligence training in hyderabad
Ziarulunirea.ro said…
Very interesting blog. Many blogs I see these days do not really provide anything that attracts others, but believe me the way you interact is literally awesome.You can also check my articles as well.

Data Science In Banglore With Placements
Data Science Course In Bangalore
Data Science Training In Bangalore
Best Data Science Courses In Bangalore
Data Science Institute In Bangalore

Thank you..
This is my first visit to your blog! We are a team of volunteers and new
initiatives in the same niche. Blog gave us useful information to work. You
have done an amazing job!

artificial intelligence training in Bangalore
lionelmessi said…
The Way Of Explaination Is Very Good And Nice Informative Topic You Have Choosen..
AWS Course Training in Hyderabad
saketh321 said…
As always your articles do inspire me. Every single detail you have posted was great. ExcelR Courses In Business Analytics
ek said…
I wanted to leave a little comment to support you and wish you a good continuation. Wishing you the best of luck for all your blogging efforts.
Data Analytics course Pune
I like viewing web sites which comprehend the price of delivering the excellent useful resource free of charge. I truly adored reading your posting. Thank you!
Really interesting post. I was looking for this kind of information and enjoyed reading this one. Keep posting. Thanks for sharing.
Data Science Course Training in Hyderabad
Priyanka said…
Attend The Course in Data Analytics From ExcelR. Practical Course in Data Analytics Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Course in Data Analytics.
Course in Data Analytics
I feel appreciative that I read this. It is useful and extremely educational and I truly took in a ton from it.
data scientist training
EXCELR said…
"Thanks for the Information.Interesting stuff to read.Great Article.
I enjoyed reading your post, very nice share.data science training"
Excelr Tuhin said…
I've read this post and if I could I desire to suggest you some interesting things or suggestions. Perhaps you could write next articles referring to this article. I want to read more things about it!
data scientist courses
Unknown said…
hello sir,
thanks for giving that type of information. I am really happy to visit your blog.Leading Solar company in Andhra Pradesh

I want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging enedevors.
Data Analytics Courses in Bangalore
i am glad to discover this page : i have to thank you for the time i spent on this especially great reading !! i really liked each part and also bookmarked you for new information on your site.
Data Scientist Course
technology said…
Without investing on any new software or hardware, it can provide you customized applications, to meet the requirements of it and provide you success in achieving your business goals. salesforce training in bangalore
Extremely overall quite fascinating post. I was searching for this sort of data and delighted in perusing this one. Continue posting. A debt of gratitude is in order for sharing. python course in delhi
data science said…
I was just examining through the web looking for certain information and ran over your blog.It shows how well you understand this subject. Bookmarked this page, will return for extra. data science course in vadodara
Unknown said…
Cool stuff you have and you keep overhaul every one of us

Digital Marketing Course
Techwriter said…
I recently came across your article and have been reading along. I want to express my admiration of your writing skill and ability to make readers read from the beginning to the end.
Data Analytics Courses In Pune
Great post I would like to thank you for the efforts you have made in writing this interesting and knowledgeable article.
data scientist training in malaysia
Ramesh Sampangi said…
Awesome blog. Informative and knowledgeable content. Keep sharing more stuff like this. Thank you for sharing this blog with us.
AI Patasala Courses in Data Science
This is really very nice post you shared, i like the post, thanks for sharing..
cyber security course
Sherman Souto said…
Thank you for providing this blog really appreciate the efforts you have taken into curating this article if you want you can check out data science course in bangalore they have a lot to offer with regards to data science in terms of training and live projects.
Anonymous said…
I want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging enedevors
business analytics course in varanasi
We are really grateful for your blog post. You will find a lot of approaches after visiting your post. Great work
data scientist training in malaysia
360DigiTMG said…
I was just browsing through the internet looking for some information and came across your blog. I am impressed by the information that you have on this blog. It shows how well you understand this subject. Bookmarked this page, will come back for more.
data scientist course in hyderabad
Very informative message! There is so much information here that can help any business start a successful social media campaign!


Data Science Course in Gorakhpur
It's like you've got the point right, but forgot to include your readers. Maybe you should think about it from different angles.

Data Scientist Course in Bangalore
Anonymous said…
I want to leave a little comment to support and wish you the best of luck.we wish you the best of luck in all your blogging enedevors
data science course in varanasi
I was just browsing through the internet looking for some information and came across your blog. I am impressed by the information that you have on this blog. It shows how well you understand this subject. Bookmarked this page, will come back for more.business analytics course in jabalpur
Thanks for posting this info. I just want to let you know that I just check out your site and I find it very interesting and informative. I can't wait to read lots of your posts.
cyber security training malaysia
ksowjanya said…
Data Science is a booming field with ample job opportunities. Start your preparation today with 360DigiTMG and become a Data Scientist in the right way.
business analytics course in pondicherry
Develop technical skills and become an expert in analyzing large sets of data by enrolling for the Best Data Science course in Bangalore. Gain in-depth knowledge in Data Visualization, Statistics, and Predictive Analytics along with the two famous programming languages and Python. Learn to derive valuable insights from data using skills of Data Mining, Statistics, Machine Learning, Network Analysis, etc, and apply the skills you will learn in your final Capstone project to get recognized by potential employers.


Data Science Course in Bangalore
Data Science handles structured and unstructured and data that is generated at an unprecedented rate every day. Anyone with a strong statistical background and an analytical mindset enjoys the challenges of big data that involves building data models and software platforms along with creating attractive visualizations and machine learning algorithms. Sign up for the Data Science courses in Bangalore with Placements and get access to resume building and mock interviews that will help you get placed with top brands in this field.


Data Science in Bangalore

Innomatics said…
Thanks for sharing such informative news.
Become Data science expert with Innomatics where you get a great experience and better knowledge.
Data science course in hyderabad
Data science traning in hyderabad
nagabhushan said…
Attempting to express profound gratitude won't just be satisfactory, for the fabulous clearness in your creation. I will in a brief instant get your rss channel to stay instructed with respect to any updates. data analytics course in pune
Get a comprehensive overview of Data Science and learn all the essential skills including collecting, modeling, and interpreting data. Register with Data Science institute Bangalore and build a strong foundation for a career where you will be involved in uncovering valuable information for your organization. Learn Python, Machine Learning, Big Data, Deep Learning, and Analytics to take center stage in Data Science.

Data Science Course in Bangalore with Placement
Get a comprehensive overview of Data Science and learn all the essential skills including collecting, modeling, and interpreting data. Register with Data Science institute Bangalore and build a strong foundation for a career where you will be involved in uncovering valuable information for your organization. Learn Python, Machine Learning, Big Data, Deep Learning, and Analytics to take center stage in Data Science.

Best Data Science Training institute in Bangalore
Rajesh said…
Advance your technical skills required to crack huge datasets to bring out new possibilities from data. Join the Data Science institutes in Bangalore and get access to top industry trainers, LMS, live projects, assignments, and mock interviews to skyrocket your career in the ever- evolving field of Data Science.
Data Science Training in Bangalore
360DigiTMG offers the best Data Analytics courses in the market with placement assistance. Enroll today and fast forward your career.


Data Science Course in Delhi
Embark on a journey to achieve your professional goals by enrolling in the Data Scientist course in Bangalore. Learn the skills of collecting, extracting, analyzing, preparing, visualizing, and presenting results to make valuable decisions. Master the concepts of data science through hands-on projects and case studies to learn the latest trends and skills in this field.

Data Scientist Course in Delhi
Anjali said…
It has been used in detecting diseases, and in this way, physicians can make crucial decisions about it.data science course in chandigarh
deekshitha said…
If you have a statistics background, then R will be an excellent choice for you, and you can learn it quickly and accurately because R is based on statistical concepts.data science training in varanasi
data analytics course in patna said…
Explore what the best Data Science training institute is offering you. Learn advanced technologies from the best industry experts.
<a href="https://360digitmg.com/india/data-analytics-certification-training-course-in-patna>data analytics course in patna</a>
PMP Course in Malaysia said…
wow, great, I was wondering how to cure acne naturally. and found your site by google, learned a lot, now i’m a bit clear. I’ve bookmark your site and also add rss. keep us updated. PMP Course in Malaysia
data science training in borivali said…
You have to learn all the required skills to transform raw data into a form that will improve the organization.

data science training in borivali
Deep learning is a subset of machine learning that uses algorithms to learn from data in a way that mimics the way the human brain works.