Saturday, November 25, 2017

Demo 36: Firmware update OTA via ESP Http Web Server

1. Introduction
In Demo 34: firmware update OTA for ESP32 using HTTP and sdcard and Demo 35: firmware update OTA for ESP32 directly using HTTP, I showed ways to update firmware OTA. In this demo, I will show you another way. That is updating firmware OTA for ESP via ESP Http Web server. With this demo, ESP will act as a web server and user will access the web server and upload the firmware file to ESP via web browser.
Figure: Web interface of the demo
User choose Browse button, navigate to firmware file and press Update button. The updating progress will be shown.
2. Hardware
Using Demo 1 to connect ESP to LED.
3. Software
- First, we will create a simple LED blink application, export the binary file for updating. In order to export the .bin file from Arduino IDE Menu, we choose Sketch -> Export compiled Binary. After finishing we choose Sketch -> Show Sketch Folder. You will see the .bin file there, rename it as led.bin. Below is the LED blinky code.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
int ledPin = 14; 

void setup(){
  pinMode(ledPin, OUTPUT);
}

void loop(){
  digitalWrite(ledPin, HIGH);
  delay(1000);
  digitalWrite(ledPin, LOW);
  delay(1000);
} 
- Second, we will re-use the Web Server library in Demo 12: How to turn the Arduino ESP32 into a Web Server. Beside that, i also used the jquery library to create uploading Http POST request. MDNS (Demo 9: How to use mDNS to resolve host names to Arduino ESP32 IP addresses) was used to resolve host name for our web server instead of using IP address directly. The code will be explained below.
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
#include <WiFi.h>
#include <WiFiClient.h>
#include <ESP32WebServer.h>
#include <ESPmDNS.h>
#include <Update.h>
#include "esp_wps.h"
const char* host = "esp32webupdate";
const char* ssid = "dd-wrt";
const char* password = "0000000000";

ESP32WebServer server(80);
const char* serverIndex = "<script src='https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js'></script>"
"<form method='POST' action='#' enctype='multipart/form-data' id='upload_form'>"
    "<input type='file' name='update'>"
    "<input type='submit' value='Update'>"
"</form>"
"<div id='prg'>progress: 0%</div>"
"<script>"
"$('form').submit(function(e){"
    "e.preventDefault();"
      "var form = $('#upload_form')[0];"
      "var data = new FormData(form);"
      " $.ajax({"
            "url: '/update',"
            "type: 'POST',"               
            "data: data,"
            "contentType: false,"                  
            "processData:false,"  
            "xhr: function() {"
                "var xhr = new window.XMLHttpRequest();"
                "xhr.upload.addEventListener('progress', function(evt) {"
                    "if (evt.lengthComputable) {"
                        "var per = evt.loaded / evt.total;"
                        "$('#prg').html('progress: ' + Math.round(per*100) + '%');"
                    "}"
               "}, false);"
               "return xhr;"
            "},"                                
            "success:function(d, s) {"    
                "console.log('success!')"
           "},"
            "error: function (a, b, c) {"
            "}"
          "});"
"});"
"</script>";

void setup(void){
    Serial.begin(115200);

    // Connect to WiFi network
    WiFi.begin(ssid, password);
    Serial.println("");

    // Wait for connection
    while (WiFi.status() != WL_CONNECTED) {
        delay(500);
        Serial.print(".");
    }
    Serial.println("");
    Serial.print("Connected to ");
    Serial.println(ssid);
    Serial.print("IP address: ");
    Serial.println(WiFi.localIP());

    /*use mdns for host name resolution*/
    if (!MDNS.begin(host)) {
        Serial.println("Error setting up MDNS responder!");
        while(1) {
            delay(1000);
        }
    }
    Serial.println("mDNS responder started");
    /*return index page which is stored in serverIndex */
    server.on("/", HTTP_GET, [](){
      server.sendHeader("Connection", "close");
      server.send(200, "text/html", serverIndex);
    });
    /*handling uploading firmware file */
    server.on("/update", HTTP_POST, [](){
      server.sendHeader("Connection", "close");
      server.send(200, "text/plain", (Update.hasError())?"FAIL":"OK");
      esp_wifi_wps_disable(); ESP.restart();
    },[](){
      HTTPUpload& upload = server.upload();
      if(upload.status == UPLOAD_FILE_START){
        Serial.printf("Update: %s\n", upload.filename.c_str());
        if(!Update.begin(UPDATE_SIZE_UNKNOWN)){//start with max available size
          Update.printError(Serial);
        }
      } else if(upload.status == UPLOAD_FILE_WRITE){
        /* flashing firmware to ESP*/
        if(Update.write(upload.buf, upload.currentSize) != upload.currentSize){
          Update.printError(Serial);
        }
      } else if(upload.status == UPLOAD_FILE_END){
        if(Update.end(true)){ //true to set the size to the current progress
          Serial.printf("Update Success: %u\nRebooting...\n", upload.totalSize);
        } else {
          Update.printError(Serial);
        }
      }
    });
    server.begin();
}

void loop(void){
  server.handleClient();
  delay(1);
}
In the code, the variable serverIndex holds the index page which is return to the web browser firstly. We will use "$.ajax" to create asynchronous uploading request. This request will be handled by "/update" action at web server. We also use "var xhr = new window.XMLHttpRequest()" to handle the progress of uploading.
The code "MDNS.begin(host)" will use MDNS to resolve "http://esp32webupdate.local" to our web server IP address.
The code "server.on("/", HTTP_GET, []()" will handle the first HTTP GET request from web browser and return the http status code 200 and the web page content in serverIndex variable.
The code "server.on("/update", HTTP_POST, []()" will handle the uploading firmware file process via HTTP POST. We handle the order of process via "upload.status" and use Update for flashing firmware. After finishing, we call "ESP.restart();" to restart ESP to get effect.
4. Result
Open web browser and Browse to the file "led.bin" and press Update button.

30 comments:

Reiner Rosin said...

Great Work! Is there any way to obtain the total size of the file within the callback function? I like to build a progress bar

Regards Reiner

iotsharing dotcom said...

Hi friend

yes, it is in HTTPUpload& upload. using upload.totalSize

Regards

Reiner Rosin said...

Thanks for your reply! I am using upload.totalSize already, but it gives me only the amount of data already transferred. To calculate the percentage I need to know the final size while transferring

Regards Reiner

iotsharing dotcom said...

Hi friend

because in uploading, server do not use content length to know whether the file is uploaded completely so it does not care about that info.
So you can create a text field with the size of the file in the html form. when you press the upload button this text field will also be submitted to server.

:)
Regards

Reiner Rosin said...

Using a text field is a funny idea ;-). I found a way now reading the size from the DOM tree and passing it as an additional parameter.

I found one strange behavior when uploading OTA: your ESP32 code uses about 60 seconds for uploading 600 KB, whereas a similar ESP8266 code transfers much faster (about 5 seconds for 300 KB). Do you have any idea what the bottleneck is? I disabled the code moving the received data to the flash already as well as the JavaScript library with no change. Even increasing the buffer size from 2048 bytes to a larger number did only improve the speed around 10% only.

Regards Reiner

iotsharing dotcom said...

:))). it is a general idea maybe in future you want to submit more data to server such as: md5, SW version, ...

This code I ported from ESP8266 so it is not optimized. I will check it soon.

Regards

Javier Ema said...

Hello,

First of all, good work!. I have the OTA webserver woking on my ESP8266 with no problems at all. I am trying to do the same with the ESP32 with your code. Everything seems fine, I can access the upload page but once uploaded it doesn't seem to change the sketch. Am I missing something? I don't understand, the progress bar says it's 100% uploaded but after restart it goes into the same sketch. Please help.

iotsharing dotcom said...

Hi,

Tell me steps that you did?

You should have 2 sketch: 1 LED blinky (for updating) and the sketch in the post.

Regards,
Tuan

Reiner Rosin said...
This comment has been removed by the author.
Reiner Rosin said...

@Javier Ema: Do you force the reboot manually or do you wait until the device reboots automatically? If you reboot manually, the sketch upload might not be finished. In my environment the web browser reaches 100% after 45 seconds, but data is still received for nearly 20 seconds more.

Regards Reiner

Javier Ema said...

I leave it to restart on its own but nothing changes, it stays with the same sketch. I will try again now

Javier Ema said...

It connects again to the wifi as if nothing happens. I am using a NodeMCU ESP-32S

Reiner Rosin said...

For further investigation you might add additional debug code here:

server.on("/update", HTTP_POST, [](){
**** HERE ADD OUTPUT OF Update.hasError() TO SERIAL ***
server.sendHeader("Connection", "close");
server.send(200, "text/plain", (Update.hasError())?"FAIL":"OK");
esp_wifi_wps_disable(); ESP.restart();

I can consider two reasons for your problem:
[1] flash size. Try to switch to board "ESP32 DEV Module", this let's you change flash size
[2] outdated environment - maybe the ESP32 package you use is not up to date or some other libraries interfere with your installation

I am using this code (with large modifications, but the basic structure is still the same) with many ESP32 boards and it's working (except the slow speed) very well.

Regards Reiner

Javier Ema said...

Hello,

I have removed the ESP package I had and installed again. I have tried on a Mac and windows and changed the flash to 2M (8M) with no success. I have also added the line you suggested but nothing came out in the serial monitor, just the same as usual. The device reboots but always comes back to the original sketch. I have also tried with a different ESP32 to see if that was the problem with no success. I really don't know what else to do. I guess the browser has nothing to do.

Javier Ema said...

do you have any other code I can try? thanks anyway

iotsharing dotcom said...

Hi

I used this module:
https://vi.aliexpress.com/item/ESP32-Development-Board-WiFi-Bluetooth-Ultra-Low-Power-Consumption-Dual-Cores-ESP-32-ESP-32S/32799057508.html

and configurations:
http://www.iotsharing.com/2017/05/blinky-hello-world-on-arduino-esp32.html

Hope it help

iotsharing dotcom said...

Hi

1 user faced the same problem with you. And he said that he re-downloaded the library and it works.

Regards

Reiner Rosin said...

Suggestion for further troubleshooting: try to find out, whether the new firmware gets uploaded or not.

First, disable ESP.restart() in the server.on("/update".... routine. You should see the debug output now of "Update.hasError()"

Second, trace the incoming of 2K packages in the "} else if (upload.status == UPLOAD_FILE_WRITE) {" section: add Serial.println(upload.totalSize); to the code

Regards Reiner

iotsharing dotcom said...

Thanks Reiner.

Or you can write the firmwware to sdcard first.

Reards

Kenny Thum said...

Hello Tuan,

Great work on the tutorial! Thumbs up to your effort.
I followed your code and tried it out, it work well.
Meanwhile, after I had successfully upload the led.bin via web browser, somehow I could not trigger back the http://esp32webupdate.local/ link anymore. It seems to execute the second sketch instead of the first one(http OTA)...Is this applicable to one-time-upload only? How do I go back to the first sketch to upload a new sketch to overwrite the existing second sketch?

Thanks once again.

Regards,
Kenny

Reiner Rosin said...

@Kenny Thum

Sure the first sketch gets overwritten when you upload the second sketch. One solution for this is to copy the OTA code into a library and include it in every sketch you use. So you can always overwrite any sketch with a new one.

Regards Reiner

iotsharing dotcom said...

Hi Kenny, Reiner

Please following Reiner 's comment.

I am writting new blog: http://www.fossreview.com/ so I am busy at this moment.

Thanks Reiner for your support :)

Regards,

Kenny Thum said...

@Reiner, thank you for your helping! I'm not very good at creating a new library from a sketch. Meanwhile, I just make a function to contain all the necessary line, and call the function out. It works good!

Btw, I ported Preference library into the sketch, which save the latest firmware name I had uploaded via HTTP. I can call out the saved preference and display the firmware name on the serial port, but I have no knowledge on javascript ::sad:: Could you help me to put a line on the javascript to show out the saved preference variable? In other words, how could we pass variable/message/string and show it on the HTTP? For example, can we display Serial.println message to the HTTP?

Thank you so much!

After this, the next step is trying to upload .bin file from different WAN network...which enable me upload my sketch to the board located far away. Looking forward to any advise from you.

Reiner Rosin said...

Some comments:

[1] By moving the code into a function you already made a library somehow. It's a small step to seperate .h and .cpp files and put in into the library foler for easy import into any sketch

[2] You don't need to know javascript for including a variable in the requested html page. You can concatenate the fixed parts of the page with the variable parts. You can even use String operations (which are a little bit critical because of memory fragmentation, but the ESP32 is much more save than the ESP8266 here). I recommend to reduce the serverIndex-page for your tests. The progress indication of the javascript code is a nice gadget, but not really necessary

[3] OTA updates over the internet are comfortable, but dangerous as well. Some helpful ideas:
a) protect your ESP from DoS by changing port 80 to something unusual
b) not use / or /upload.html or /update.html as index page, but something like ymca-omg-wtf.html
c) protect your ESP from beeing overwritten by hacker kiddies. Add a secret parameter in an input field and verify it in the OTA code. You can even use variant codes with a challenge/response mechanism
d) for be 100% tap-proof you can switch to https transfer

One more recommendation: you can improve your skills with learning by doing and sharing your thoughts with other beginners. Have a look at an internet forum (there are many of them) or some facebook groups.

Regards Reiner

Kenny Thum said...

@Reiner,

Wow!! Great work and thank you for your advises! Indeed, learning from the internet forum is one of the best way, and yet time consuming. Sometimes its either the information is not complete or the info is obsoleted.

Paitoon Saelim said...
This comment has been removed by the author.
Darren Gibbard said...

Thanks for sharing this code, it's exactly what I needed :) @Kenny - for updates over the web, consider implementing Basic Auth, and restricting access via network firewalls where possible.

There seems to be some auth bits in the ESP32WebServer.h code already but I haven't looked into it much:
bool authenticate(const char * username, const char * password);

Failing that, for basic auth, you should be able to read in an 'Authorization' header, and look for 'Basic xxxxxxxx' where 'xxxxxxxxx' is a base64 encoded hash of "username:password" - you can generate this on a Linux shell with something like "echo -n username:password" | base64". That paired with HTTPS would be pretty secure.

jan kowalski said...

Hello,
I have a problem with OTA over http, in my case it works only on internet exprorer 11, can it be caused that my ESP32 is configured as an Access point ?, and there is no access to the internet and there is a problem with accessing the ajax library ?.

iotsharing dotcom said...

yes man,

it used jquery that is loadded from internet.
If you want ur case to run. please check the post that i used sdcard to store resources such as images, ... (search sdcard in search box)

Regards

jan kowalski said...

Hi, thanks for your reply.
I have one more question, because in my device I can't connect SDCARD, but I have a SIM808 modem connected to ESP32 via serial port and I would like to update firmware directly from the serial port of the modem, is it possible implement firmware update in my hardware by using yours libraries?

Best Regards