Monday, January 22, 2018

Demo 40: Create a Facebook Messenger chat bot for monitoring and controlling home devices using Raspberry/Orange Pi and ESP32/8266

1. Introduction
In this demo, I will show you how to create a Facebook Messenger chat bot for monitoring and controlling home devices using Raspberry/Orange Pi and ESP32/8266. This is not a full solution so you need to improve it by yourself.
Figure: Facebook Messenger chat bot for monitoring and controlling home devices
There are 2 steps to make this demo:
- How to setup a Facebook chat bot
- How to setup local system (Raspberry/Orange Pi, ESP32/8266)
This is the model of the demo:
Figure: The model of the demo
2. Setup
2.1 Create Facebook page
You can refer to this guideline. And choose type of Page is Business, Brand or Place. In my demo, I created a page named IoT Sharing.
2.2 Setup Facebook chat bot
In order to develop or use Facebook services for Software development you need to register a Facebook developer account here.
After created a FB developer account choose My Apps -> Add a new App -> Fill Display name
In Select a Product -> choose messenger -> Set Up
Choose Webhooks -> Setup Webhooks
Note: We do not do this step now. This step will be done after running the Raspbbery/Orange Pi software. Because after filled Callback URL, Verify Token and press Verify and Save, FB will send a GET request with "hub.challenge=value of Verify Token" (in this case Verify Token is "iotsharing.com" string) to Pi. Pi need to check the value of hub.challenge must match the value that we filled in Verify Token field (in this case Verify Token is "iotsharing.com" string). And then send this hub.challenge back to FB to finish the verification.
1
2
3
4
5
6
@app.route('/', methods=['GET'])
def handle_verification():
 if(request.args['hub.challenge'] == 'iotsharing.com'):
  return request.args['hub.challenge']
 else:
  return 'not match'
Fill information like below and choose Verify and Save.
choose the Page that you created in previous step and Subscribe/Unsubscribe.
Finally, extract the ACCESS_TOKEN of the FB app so that our application can authenticate to use FB services.
It is done for Facebook setup step.
2.2 Setup local system (Raspberry/Orange Pi, ESP32/8266)
In this demo, Pi will keep some roles:
- Communicate with Facebook server to receive and respond message.
- Parse Facebook message and then using MQTT protocol to publish the commands to ESP32/8266 clients and subscribe responses from ESP32/8266 clients.
- Refer to this post to setup MQTT for Pi.
- We use Python Paho MQTT for local communication. Refer to this post to install it.
- Besides we need to install some packages (Flask, ngrok) on Raspberry/Orange Pi for our demo. 
+ Flask is a Python web framework. In our demo, It is a web server to handle Facebook https request.
+ ngrok secure introspect-able tunnels to local host web hook development tool and debugging tool. This tool helps Facebook server can see our Flask local web server.
From Pi Terminal running the commands:
+ Install Flask Python server: "sudo pip install Flask"
+ Download ngrok: "wget https://bin.equinox.io/a/26q6mq7ddJR/ngrok-2.2.9-linux-arm.zip"
+ Unzip zip file for executable app: "unzip ngrok-2.2.9-linux-arm.zip"
3. Hardware
- 1 ESP32 or ESP8266
- 1 Raspberry Pi or Orange Pi
- 1 LED connect to ESP
- 1 temperature sensor connect to ESP. 
In this demo I do not use temperature sensor. I used random(min, max) function to generate temperature.
4. Software
The source code of this demo can be found on github.
Define messages:
- Pi send MQTT request topics to ESP:
Temperature topic: "floor1/room1/temp1" to measure temperature
Led topic:               "floor1/room1/led1" to set "0" (off) or "1" (on)
- ESP send MQTT response topics to Pi:
Temperature response topic: "floor1/room1/temp1/res" with temperature value
Led response topic:               "floor1/room1/led1/res" to inform that the request was executed
- The template for Facebook message:
"set floor1/room1/led1 on" -> turn on LED and get response
"set floor1/room1/led1 off" -> turn off LED and get response
"get floor1/room1/temp1" -> request temperature measurement and get response
ESP code I reused the post (esp32chatbot.ino)
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
#include <WiFi.h>
#include <PubSubClient.h>

const char* ssid = "dd-wrt";
const char* password = "0000000000";
//ip address of Raspberry/orange pi
const char* mqtt_server = "192.168.1.106";
char msg[20];

/* create an instance of PubSubClient client */
WiFiClient espClient;
PubSubClient client(espClient);

/*LED GPIO pin*/
const char led = 4;
/* topics */
#define TEMP_TOPIC    "floor1/room1/temp1"
#define LED_TOPIC     "floor1/room1/led1" /* on, off */

void receivedCallback(char* topic, byte* payload, unsigned int length) {
    Serial.print("topic: ");
    Serial.println(topic);
      if(strcmp(topic, LED_TOPIC) == 0){
        Serial.print("payload: ");
        for (int i = 0; i < length; i++) {
          Serial.print((char)payload[i]);
        }
        Serial.println();
        /* we got '1' -> on */
        if ((char)payload[0] == '1') {
          digitalWrite(led, HIGH); 
          snprintf (msg, 20, "%s", "on");
        /* publish the response */
        client.publish(LED_TOPIC "/res", msg);
        } else {
          /* we got '0' -> on */
          digitalWrite(led, LOW);
          snprintf (msg, 20, "%s", "off");
          client.publish(LED_TOPIC "/res", msg);
        }
    }else {
        snprintf (msg, 20, "%d", random(0, 40));
        client.publish(TEMP_TOPIC "/res", msg);
    }
}

void mqttconnect() {
  /* Loop until reconnected */
  while (!client.connected()) {
    Serial.print("MQTT connecting ...");
    /* client ID */
    String clientId = "ESP32Client";
    /* connect now */
    if (client.connect(clientId.c_str())) {
      Serial.println("connected");
      /* subscribe topic */
      client.subscribe(LED_TOPIC);
      client.subscribe(TEMP_TOPIC);
    } else {
      Serial.print("failed, status code =");
      Serial.print(client.state());
      Serial.println("try again in 5 seconds");
      /* Wait 5 seconds before retrying */
      delay(5000);
    }
  }
}

void setup() {
  Serial.begin(115200);
  // We start by connecting to a WiFi network
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  /* set led as output to control led on-off */
  pinMode(led, OUTPUT);

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());

  /* configure the MQTT server with IPaddress and port */
  client.setServer(mqtt_server, 1883);
  /* this receivedCallback function will be invoked 
  when client received subscribed topic */
  client.setCallback(receivedCallback);
}
void loop() {
  /* if client was disconnected then try to reconnect again */
  if (!client.connected()) {
    mqttconnect();
  }
  /* this function will listen for incomming 
  subscribed topic-process-invoke receivedCallback */
  client.loop();
}
Pi code (chatbot.py):
We use variable history to hold the id of Facebook messenger sender while waiting for the response from ESP. This sender id is used in reply() function to send response back to Facebook messenger sender.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
from flask import Flask, request
import requests
import paho.mqtt.publish as publish
import paho.mqtt.client as mqtt
import thread

ACCESS_TOKEN = "your_fb_access token_here";

#hold sender id
history = dict()

set_cmd = 'floor1/room1/led1'
get_cmd = 'floor1/room1/temp1'

#respond to FB messenger
def reply(user_id, msg):
    data = {
        "recipient": {"id": user_id},
        "message": {"text": msg}
    }
    resp = requests.post("https://graph.facebook.com/v2.6/me/messages?access_token=" + ACCESS_TOKEN, json=data)
    print(resp.content)

#MQTT handler
def on_connect(mqttc, obj, flags, rc):
    print("rc: "+str(rc))
def on_message(mqttc, obj, msg):
    print(msg.topic+": "+str(msg.payload))
    if msg.topic in history:
        user_id = history[msg.topic]
  #reply FB messenger
        reply(user_id, msg.payload)
        history.pop(msg.topic, None)

mqttc = mqtt.Client()
mqttc.on_connect = on_connect
mqttc.on_message = on_message
mqttc.connect("localhost", 1883, 60)
mqttc.subscribe("floor1/#", 0)

#MQTT subscribe thread 
def mqtt_thread( threadName, delay):
   mqttc.loop_forever()
try:
   thread.start_new_thread( mqtt_thread, ("mqtt-thread", 0, ) )
except:
   print "Error: unable to start thread"

#Flask web server instance
app = Flask(__name__)

#handle GET request from Facebook
@app.route('/', methods=['GET'])
def handle_verification():
 if(request.args['hub.challenge'] == 'iotsharing.com'):
  return request.args['hub.challenge']
 else:
  return 'not matched'

#handle POST request from Facebook
@app.route('/', methods=['POST'])
def handle_incoming_messages():
    data = request.json
    print(data)
    sender = data['entry'][0]['messaging'][0]['sender']['id']
    message = data['entry'][0]['messaging'][0]['message']['text']
    print(message)
    if(message.startswith('set')):
        arr = message.split()
        l = len(arr)
        if(l == 3 and set_cmd == arr[1]):
            cmd = arr[1]
            val = '1' if arr[2]=='on' else '0'
            publish.single(cmd, val, hostname="localhost")
   #record command with sender id
            history[cmd+'/res'] = sender
            return 'ok'
    elif(message.startswith('get')):
        arr = message.split()
        l = len(arr)
        if(l == 2 and get_cmd == arr[1]):
            cmd = arr[1]
            publish.single(cmd, '', hostname="localhost")
   #record command with sender id
            history[cmd+'/res'] = sender
            return 'ok'
    reply(sender, 'invalid query')
    return "ok"

if __name__ == '__main__':
app.run(debug=True)
5. Steps to deploy
- MQTT broker run on Pi
- Open 2 Terminal on Pi:
   + Run "python chatbot.py" with Flask in it.
   + Run ngrok (5000 is the port Flask listen on): "./ngrok http 5000"

Copy the url in red box to Callback URL field that mention in step 2.2 Setup Facebook chat bot
Note: in case we shutdown ngrok and run it again the new url will be generated. We have to re-register this url to Facebook.
Choose Webhooks -> Edit Subscription
And then Unsubcribe and Subscribe the page again:
Now go to the Page that we created, choose About and Send Message
And type one of commands:
"set floor1/room1/led1 on"
"set floor1/room1/led1 off"
"get floor1/room1/temp1"

6. Result



Tuesday, January 16, 2018

Demo 39: ESP32/8266 multipart upload a file and download a file via HTTP

1. Introduction
In previous demos, I showed you how to use MQTT/MQTTS and how to update firmware OTA (TCP/UDP and HTTP). In this demo, I will show you another interesting topic. That is how to download and multi-part upload a file via HTTP.
Figure: multipart upload and download a file via http
Based on this demo, you can build an application that can download the new firmware or the configuration file to memory card that is attached to client board for using later. Or you can upload the log file that records occurred events in run time to a server.
This is the requirement of this demo: a ESP32 will download a file from internet and after finish downloading, upload it to local apache server. The downloaded file will be stored in sdcard.
In order to make it easy, I created a library here in github. You just install and use it.
Note: this library can be apply for ESP8266.
2. Software
The library is simple to use. It has 2 APIs: upload() and download() with parameters. The parameters are:
- url of uploading and downloading file.
- progress displaying callback function.
- response processing callback function.
- reading and writing data to storage (sdcard, SPIFFS, ...) callback functions.
You should change them accordingly to your application.
Note: The url parser of this library is quite simple and it will be updated later.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
#include "UDHttp.h"
#include "mySD.h"


const char* ssid     = "dd-wrt";
const char* password = "0000000000";

File root;
//these callbacks will be invoked to read and write data to sdcard
//and process response
//and showing progress 
int responsef(uint8_t *buffer, int len){
  Serial.printf("%s\n", buffer);
  return 0;
}
//read data callback
int rdataf(uint8_t *buffer, int len){
  //read file to upload
  if (root.available()) {
    return root.read(buffer, len);
  }
  return 0;
}
//write data callback
int wdataf(uint8_t *buffer, int len){
  //write downloaded data to file
  return root.write(buffer, len);
}

void progressf(int percent){
  Serial.printf("%d\n", percent);
}

void setup() {
  // put your setup code here, to run once:
  Serial.begin(115200);

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
      delay(500);
      Serial.print(".");
  }

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
    
  Serial.print("Initializing SD card...");
  if (!SD.begin(32, 14, 12, 27)) {
    Serial.println("initialization failed!");
    return;
  }
  Serial.println("initialization done.");
  SD.remove("test.pdf");
  {
    UDHttp udh;
    //open file on sdcard to write
    root = SD.open("test.pdf", FILE_WRITE);
    if (!root) {
       Serial.println("can not open file!");
       return;
    }
    //download the file from url
    udh.download("http://www.smart-words.org/linking-words/linking-words.pdf", wdataf, progressf);
    root.close();
    Serial.printf("done downloading\n");
  }
  {
    UDHttp udh;
    //open file on sdcard to read
    root = SD.open("test.pdf");
    if (!root) {
       Serial.println("can not open file!");
       return;
    }
    //upload downloaded file to local server
    udh.upload("http://192.168.1.107:80/upload/upload.php", "test.pdf", root.size(), rdataf, progressf, responsef);
    root.close();
    Serial.printf("done uploading\n");
  }
}

void loop() {
  // put your main code here, to run repeatedly:

}
And the php script was used for this demo:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
<?php
$target_path = "uploads/";

$target_path = $target_path . basename( $_FILES['data']['name']); 

if(move_uploaded_file($_FILES['data']['tmp_name'], $target_path)) {
    echo "The file ".  basename( $_FILES['data']['name']). 
    " has been uploaded";
} else{
    echo "There was an error uploading the file, please try again!";
}
?>
Note: The variable $_FILES['data']['name'] has fixed 'data' field name (similar to html form ""). The downloaded file will be stored in sub-folder 'uploads' which is relative to folder where the php script located. Reference this post to install local apache server. 
3. Hardware
For harware connection please refer here.

4. Result
Figure: start downloading with progress displaying

Thursday, December 7, 2017

Demo 38: How to decode error/exception "CPU halted" of ESP on Arduino

1. Introduction
When developing software for ESP8266 or ESP32, you often face error(s) that caused the ESP halt.  At that moment, looking the Serial Monitor window, you will see the error/exception like below:
Figure: error that caused ESP CPU halt
It is not easy to know what was happening with our software. Actually, the string that is highlighted has meaning; it is the called stack of software when the CPU halted. There is a tool that supports you to decode the ESP error/exception above. It called EspExceptionDecoder. It is tool that is attached with Arduino IDE.
In order to install it, please follow these steps:
    - Install Arduino IDE with ESP8266/ESP32 core.
    - Download the tool here.
    - Create tools directory under Arduino directory if it is not exist.
    - Unpack the downloaded tool into tools directory (the path will look like /Arduino/tools/EspExceptionDecoder/tool/EspExceptionDecoder.jar).
    - Restart Arduino IDE.



Figure: Under Tools menu, new entry ESP Exception Decoder
2. Demo
I will make a simple program demo for this tool.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
void setup() {
  Serial.begin(115200);
  char *buf = NULL;
  buf[2] = 4;
  Serial.printf("val %d\n", buf[2]);
}

/* the forever loop() function is invoked by Arduino ESP32 loopTask */
void loop() {
}
Here the bug is that buf is not initialized buf buf[2] is assigned to 4.
Loading the software to ESP and look the Serial Monitor:
Figure: error in software
After that choosing Tools->ESP Exception Decoder, a window will occur.
 Figure: Exception Decoder window

Copy and paste the string "Backtrace: 0x400d05be:0x3ffc77c0 0x400d9b1a:0x3ffc77e0 " to it and you will see the output:
 Figure: the result after decoding stacktrace
So you see the CPU halted executing the code at line 4: buf[2] = 4; as we knew.
This is just a simple application. In real world, you may face a more difficult situation than this, but at least you could know where your application is halted.

Sunday, December 3, 2017

Demo 37: Display distance measured by ultrasonic sensor using module 7-segment-LED-N-Digits

1. Introduction
Today I will show you how to use module 7-segment-LED-N-Digits to display distance which is measured by ultrasonic sensor.
2. Hardware and Software 
2.1 Ultrasonic sensor
I used HC - SR04.

Figure: Ultrasonic sensor HC - SR04 (Source: Internet) 
The sensor has 2 heads: one is emit the ultrasonic and one receives it when the ultrasonic is reflect by the obstacle. The range of this sensor is 2cm -  400cm non-contact.
Figure: the operation mechanism of ultrasonic sensor
The picture below is timing diagram of HC - SR04.
 
Figure: timing diagram of HC - SR04
The basic principle of work:
(1) Using IO trigger for at least 10us high level signal.
(2) The Module automatically sends eight 40 kHz and detect whether there is a ultrasonic signal back.
(3) IF the ultrasonic signal back, through high level, time of high output IO duration is the time from sending ultrasonic to returning.
Test distance = (High level time x velocity of sound (299 cm/us) / 2
2.2 7-segment-LED-8-digit module
I used
Figure: 7-segment-LED-8-digit module
This module used IC 74HC595. It is a 8 bits shift register. With this IC we can save more IO digital instead of using a lot of IO pins to trigger each segment of LED.
Figure: schematic application of IC 74HC595
In order to bring data to this LED module, first we shift data that will be displayed on LED then we shift the value that indicate which LED in order that will display the data. i created the library here. The library is easy to use. You create an instance of library  
EspLed7SegNDigit ledm(SCLK, RCLK, DIO, 8);
where SCLK, RCLK, DIO is pins that connect between ESP and LED module. 8 is number of LEDs in module.
We can use ledm.setCharAt(8, 'd'); to set the value (here is character 'd') that will be displayed at specific LED (here is LED number 8).
In order to human can see the LED clearly, the LED scan time will be default 400ms (25 frames/second). You can set it using setRefreshTime(ms). The callback function updateDisplayCb() will be invoked after every refreshing time to update new value for LED displaying. You should call "ledm.clearDisplay()" to clear LED module before updating new display. You can use ledm.displayNum(num, 3); to display the float number with 3 digits behind the dot character. The function "ledm.loop();" will run continuously to update the display on LED.
Pins connection:
LED module
+SCLK with ESP32 GPIO14+RCLK with ESP32 GPIO27
+DIO with ESP32 GPIO12
Ultrasonic sensor
+TRIG with ESP32 GPIO25
+ECHO with ESP32 GPIO33
2.3 Full software
Our application has 2 FreeRTOS tasks: 1 for ultrasonic measurement and 1 for LED updating.
This version do not use FreeRTOS
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
#include "EspLed7SegNDigit.h"

//LED pins
int SCLK = 14; //pulse
int RCLK = 27; //latch
int DIO = 12;  //data

//utrasonic pins
int TRIG_PIN = 25;
int ECHO_PIN = 33;

unsigned long startMeasure = 0;
unsigned long endMeasure = 0;
unsigned long measureTime = 0;
unsigned long distance = 0;

EspLed7SegNDigit ledm(SCLK, RCLK, DIO, 8);

void updateDisplayCb(void){
  ledm.clearDisplay();
  ledm.setCharAt(8, 'd');
  ledm.setCharAt(7, 's');
  ledm.setCharAt(6, 't');
  ledm.displayNum(distance, 3);
}

void initUltra(){
  pinMode(TRIG_PIN, OUTPUT);
  pinMode(ECHO_PIN, INPUT);
}

void ultraTask(){

  //trigger sensor with pulse LOW-HIGH-LOW
  digitalWrite(TRIG_PIN, LOW);
  //wait 2 us
  delayMicroseconds(2);
  digitalWrite(TRIG_PIN, HIGH);
  //wait 10 us
  delayMicroseconds(10);
  digitalWrite(TRIG_PIN, LOW);
  
  //at the beginning ECHO pin will be pull LOW until finishing transmitting ultrasonic signal
  while (digitalRead(ECHO_PIN) == 0){
    startMeasure = micros();
  }
  //ECHO pin will be pulled HIGH until get response
  while (digitalRead(ECHO_PIN) == 1){
    endMeasure = micros();
  }
  
  //response time will be calculated by
  measureTime = endMeasure - startMeasure;

  //convert to cm
  distance = (measureTime)/29/2;
  Serial.printf("distance = %d\n", distance);
}
void setup() {
  // put your setup code here, to run once:
  Serial.begin(115200);
  initUltra();
  //this callback will be invoked for updating new ultrasonic value
  ledm.setUpdateCb(updateDisplayCb);

  ledm.setCharAt(8, 'd');
  ledm.setCharAt(7, 's');
  ledm.setCharAt(6, 't');
  //in loop it take time to do ultraTask so we decrease refresh time
  ledm.setRefreshTime(100);
}
long tick = 0;
void loop() {
  // put your main code here, to run repeatedly:
  ledm.loop();
  long now = millis();
  if (now - tick > 1000) {
    tick = now;
    ultraTask();
  }
}
This version uses FreeRTOS

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
#include "EspLed7SegNDigit.h"

//utrasonic pins
int TRIG_PIN = 25;
int ECHO_PIN = 33;

//LED pins
int SCLK = 14; //pulse
int RCLK = 27; //latch
int DIO = 12;  //data

unsigned long measureTime = 0;
unsigned long distance = 0;
EspLed7SegNDigit ledm(SCLK, RCLK, DIO, 8);

void initUltra(){
  pinMode(TRIG_PIN, OUTPUT);
  pinMode(ECHO_PIN, INPUT);
}

void updateDisplayCb(void){
  ledm.clearDisplay();
  ledm.setCharAt(8, 'd');
  ledm.setCharAt(7, 's');
  ledm.setCharAt(6, 't');
  ledm.displayNum(distance, 3);
}

void setup() {
  Serial.begin(112500);
  initUltra();
  //this callback will be invoked for updating new ultrasonic value
  ledm.setUpdateCb(updateDisplayCb);
  ledm.setCharAt(8, 'd');
  ledm.setCharAt(7, 's');
  ledm.setCharAt(6, 't');
  ledm.setRefreshTime(100);
  /* we create a new task here */
  xTaskCreate(
      ultraTask,                /* Task function. */
      "ultrasonic Task",        /* name of task. */
      10000,                    /* Stack size of task */
      NULL,                     /* parameter of the task */
      3,                        /* priority of the task */
      NULL);                    /* Task handle to keep track of created task */
}

/* the forever loop() function is invoked by Arduino ESP32 loopTask */
void loop() {
  ledm.loop();
  delay(1);
}
/* this function will be invoked when ultraTask was created */
void ultraTask( void * parameter )
{
  for(;;){

    //trigger sensor with pulse LOW-HIGH-LOW
    digitalWrite(TRIG_PIN, LOW);
    //wait 2 us
    delayMicroseconds(2);
    digitalWrite(TRIG_PIN, HIGH);
    //wait 10 us
    delayMicroseconds(10);
    digitalWrite(TRIG_PIN, LOW);
    measureTime = pulseIn(ECHO_PIN, HIGH);//read measurement time for HIGH level from Echo
    //convert to cm
    distance = (measureTime)/29/2;
    Serial.printf("distance = %d\n", distance);
    delay(1000);
  }
  vTaskDelete( NULL );
}
3. Result

Saturday, November 25, 2017

Demo 36: Firmware update OTA via ESP Http Web Server

1. Introduction
In Demo 34: firmware update OTA for ESP32 using HTTP and sdcard and Demo 35: firmware update OTA for ESP32 directly using HTTP, I showed ways to update firmware OTA. In this demo, I will show you another way. That is updating firmware OTA for ESP via ESP Http Web server. With this demo, ESP will act as a web server and user will access the web server and upload the firmware file to ESP via web browser.
Figure: Web interface of the demo
User choose Browse button, navigate to firmware file and press Update button. The updating progress will be shown.
We will create a simple LED blink application, export the binary file for updating. In order to export the .bin file from Arduino IDE Menu, we choose Sketch -> Export compiled Binary. After finishing we choose Sketch -> Show Sketch Folder. You will se the .bin file there, rename it as led.bin
2. Hardware
Using Demo 1 to connect ESP to LED.
3. Software
We will re-use the Web Server library in Demo 12: How to turn the Arduino ESP32 into a Web Server. Beside that, i also used the jquery library to create uploading Http POST request. MDNS (Demo 9: How to use mDNS to resolve host names to Arduino ESP32 IP addresses) was used to resolve host name for our web server instead of using IP address directly. The code will be explained below.
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
#include <WiFi.h>
#include <WiFiClient.h>
#include <ESP32WebServer.h>
#include <ESPmDNS.h>
#include <Update.h>

const char* host = "esp32webupdate";
const char* ssid = "dd-wrt";
const char* password = "0000000000";

ESP32WebServer server(80);
const char* serverIndex = "<script src='https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js'></script>"
"<form method='POST' action='#' enctype='multipart/form-data' id='upload_form'>"
    "<input type='file' name='update'>"
    "<input type='submit' value='Update'>"
"</form>"
"<div id='prg'>progress: 0%</div>"
"<script>"
"$('form').submit(function(e){"
    "e.preventDefault();"
      "var form = $('#upload_form')[0];"
      "var data = new FormData(form);"
      " $.ajax({"
            "url: '/update',"
            "type: 'POST',"               
            "data: data,"
            "contentType: false,"                  
            "processData:false,"  
            "xhr: function() {"
                "var xhr = new window.XMLHttpRequest();"
                "xhr.upload.addEventListener('progress', function(evt) {"
                    "if (evt.lengthComputable) {"
                        "var per = evt.loaded / evt.total;"
                        "$('#prg').html('progress: ' + Math.round(per*100) + '%');"
                    "}"
               "}, false);"
               "return xhr;"
            "},"                                
            "success:function(d, s) {"    
                "console.log('success!')"
           "},"
            "error: function (a, b, c) {"
            "}"
          "});"
"});"
"</script>";

void setup(void){
    Serial.begin(115200);

    // Connect to WiFi network
    WiFi.begin(ssid, password);
    Serial.println("");

    // Wait for connection
    while (WiFi.status() != WL_CONNECTED) {
        delay(500);
        Serial.print(".");
    }
    Serial.println("");
    Serial.print("Connected to ");
    Serial.println(ssid);
    Serial.print("IP address: ");
    Serial.println(WiFi.localIP());

    /*use mdns for host name resolution*/
    if (!MDNS.begin(host)) {
        Serial.println("Error setting up MDNS responder!");
        while(1) {
            delay(1000);
        }
    }
    Serial.println("mDNS responder started");
    /*return index page which is stored in serverIndex */
    server.on("/", HTTP_GET, [](){
      server.sendHeader("Connection", "close");
      server.send(200, "text/html", serverIndex);
    });
    /*handling uploading firmware file */
    server.on("/update", HTTP_POST, [](){
      server.sendHeader("Connection", "close");
      server.send(200, "text/plain", (Update.hasError())?"FAIL":"OK");
      ESP.restart();
    },[](){
      HTTPUpload& upload = server.upload();
      if(upload.status == UPLOAD_FILE_START){
        Serial.printf("Update: %s\n", upload.filename.c_str());
        if(!Update.begin(UPDATE_SIZE_UNKNOWN)){//start with max available size
          Update.printError(Serial);
        }
      } else if(upload.status == UPLOAD_FILE_WRITE){
        /* flashing firmware to ESP*/
        if(Update.write(upload.buf, upload.currentSize) != upload.currentSize){
          Update.printError(Serial);
        }
      } else if(upload.status == UPLOAD_FILE_END){
        if(Update.end(true)){ //true to set the size to the current progress
          Serial.printf("Update Success: %u\nRebooting...\n", upload.totalSize);
        } else {
          Update.printError(Serial);
        }
      }
    });
    server.begin();
}

void loop(void){
  server.handleClient();
  delay(1);
}
In the code, the variable serverIndex holds the index page which is return to the web browser firstly. We will use "$.ajax" to create asynchronous uploading request. This request will be handled by "/update" action at web server. We also use "var xhr = new window.XMLHttpRequest()" to handle the progress of uploading.
The code "MDNS.begin(host)" will use MDNS to resolve "http://esp32webupdate.local" to our web server IP address.
The code "server.on("/", HTTP_GET, []()" will handle the first HTTP GET request from web browser and return the http status code 200 and the web page content in serverIndex variable.
The code "server.on("/update", HTTP_POST, []()" will handle the uploading firmware file process via HTTP POST. We handle the order of process via "upload.status" and use Update for flashing firmware. After finishing, we call "ESP.restart();" to restart ESP to get effect.
4. Result

Wednesday, November 22, 2017

Demo 35: firmware update OTA for ESP32 directly using HTTP

1. Introduction
This tutorial is similar to Demo 34 but without using sdcard. It means firmware will be flashed to ESP directly from HTTP downloading process. Here is the model of this demo:
Figure: model of demo
2. Hardware
Refer Demo 7 to connect ESP32 pins with sdcard module. And Demo 1 to connect ESP to LED.
3. Software
I made the library in github. Just download, install it and run example esp32httpota2. Change the Wifi ssid, password and IP of MQTT server according to yours. The code will be explained below.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
#include <WiFi.h>
#include <HttpFOTA.h>
#include <PubSubClient.h>

typedef enum {
  Runnning_e = 0x01,
  Fota_e  
}SysState;

const char* ssid = "dd-wrt";
const char* password = "0000000000";
const char* mqtt_server = "192.168.1.107";

char url[100];
char md5[50];
WiFiClient espClient;
PubSubClient client(espClient);
SysState state = Runnning_e;

/* topics */
#define OTA_TOPIC    "smarthome/room1/ota"

void progress(DlState state, int percent){
  Serial.printf("state = %d - percent = %d\n", state, percent);
}

void receivedCallback(char* topic, byte* payload, unsigned int length) {
  
  if(strncmp(OTA_TOPIC, topic, strlen(OTA_TOPIC)) == 0){
    memset(url, 0, 100);
    memset(md5, 0, 50);
    char *tmp = strstr((char *)payload, "url:");
    char *tmp1 = strstr((char *)payload, ",");
    memcpy(url, tmp+strlen("url:"), tmp1-(tmp+strlen("url:")));
    
    char *tmp2 = strstr((char *)payload, "md5:");
    memcpy(md5, tmp2+strlen("md5:"), length-(tmp2+strlen("md5:")-(char *)&payload[0]));

    Serial.printf("started fota url: %s\n", url);
    Serial.printf("started fota md5: %s\n", md5);
    state = Fota_e;
  }
}
void mqttconnect() {
  /* Loop until reconnected */
  while (!client.connected()) {
    Serial.print("MQTT connecting ...");
    /* client ID */
    String clientId = "ESP32Client";
    /* connect now */
    if (client.connect(clientId.c_str())) {
      Serial.println("connected");
      /* subscribe topic */
      client.subscribe(OTA_TOPIC);
    } else {
      Serial.print("failed, status code =");
      Serial.print(client.state());
      Serial.println("try again in 5 seconds");
      /* Wait 5 seconds before retrying */
      delay(5000);
    }
  }
}

void error(char *message){
  printf("%s\n", message);
}

void startDl(void){

}
void endDl(void){

}

void setup() {
  // put your setup code here, to run once:
  // put your setup code here, to run once:
  Serial.begin(115200);
  Serial.print("Connecting to "); 
  Serial.print(ssid);
  WiFi.begin(ssid, password);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  } 
  Serial.println("");
  
  Serial.print("WiFi connected, IP address: "); 
  Serial.println(WiFi.localIP());

  /* configure the MQTT server with IPaddress and port */
  client.setServer(mqtt_server, 1883);
  /* this receivedCallback function will be invoked 
  when client received subscribed topic */
  client.setCallback(receivedCallback);
}
 
void loop() {
  switch(state)
  {
    case Runnning_e:
      /* if client was disconnected then try to reconnect again */
      if (!client.connected()) {
        mqttconnect();
      }
      /* this function will listen for incomming 
      subscribed topic-process-invoke receivedCallback */
      client.loop();
    break;
    case Fota_e:
      DlInfo info;
      info.url = url;
      info.md5 = md5;
      info.startDownloadCallback =  startDl;
      info.endDownloadCallback =    endDl;
      info.progressCallback  = progress;
      info.errorCallback     = error;
      httpFOTA.start(info);
    
      client.publish(OTA_TOPIC, "ok");
    break;
    default:
    break;
  }
}
In order to start fota updating process, using the Python code below and type "1" to publish FOTA command to ESP. The ESP will receive  the command, parse it and starting FOTA process. Change the url and md5 value according to yours.
The library is simple. It only has 1 interface that is start with input is DlInfo structure. When calling this function, it will block the loop() function. We must initialize the structure before using it. The structure includes:
- url: is the http url of the firmware file.
- md5: is the md5 checksum of the firmware file.
- startDownloadCallback: is the function that will be invoked before starting downloading. I left it empty in this demo.
- endDownloadCallback: is the function that will be invoked after downloading was finished. I left it empty in this demo.
- progressCallback: is the function that will be invoked to show the progress of downloading and flashing process.
- errorCallback: is the function that will be invoked to show the error of downloading and flashing process.
The functions: saveData, readData, progress, error, startDl, endDl, startFl, endFl will be invoked by the library in the update process. They are assigned to the members of DlInfo structure.
The function receiveCallback will be invoked whenever ESP received the MQTT command. It will check if the topic is OTA_TOPIC then parsing the payload to get url and md5 value of firmware file. After that, it change the state of system to Fota_e to start updating process.
After finishing the whole process, the code "client.publish(OTA_TOPIC, "ok");" will publish message "ok" back to Python application.
When the system is in Running_e, it just listen the MQTT command.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
from random import randint
import thread
import sys
try:
    import paho.mqtt.client as mqtt
except ImportError:
    import os
    import inspect
    cmd_subfolder = os.path.realpath(os.path.abspath(os.path.join(os.path.split(inspect.getfile( inspect.currentframe() ))[0],"../src")))
    if cmd_subfolder not in sys.path:
        sys.path.insert(0, cmd_subfolder)
    import paho.mqtt.client as mqtt


server = "192.168.1.107";
topic = "smarthome/room1/ota"


def on_connect(mqttc, obj, flags, rc):
    print("rc: "+str(rc))

def on_message(mqttc, obj, msg):
    print(msg.topic+" "+str(msg.qos)+" "+str(msg.payload))

def on_publish(mqttc, obj, mid):
    print("mid: "+str(mid))

def on_subscribe(mqttc, obj, mid, granted_qos):
    print("Subscribed: "+str(mid)+" "+str(granted_qos))

def on_log(mqttc, obj, level, string):
    print(string)

mqttc = mqtt.Client()
mqttc.on_message = on_message
mqttc.on_connect = on_connect
mqttc.on_publish = on_publish
mqttc.on_subscribe = on_subscribe
mqttc.connect(server, 1883, 60)
mqttc.subscribe(topic, 0)


def fotaControl( threadName, delay):
    while True:
        val = raw_input('Enter 1 to update firmware OTA ')
        if(val == "1"):
            mqttc.publish(topic, "url:http://192.168.1.107/phphello/led.bin,md5:6bd07139c21f572370242905c4465056")

try:
    thread.start_new_thread( fotaControl, ("Fota Control", 0, ) )
except:
    print "Error: unable to start thread"

mqttc.loop_forever()
4. Result