Saturday, May 26, 2018

Demo 42: How to build an IoT Dashboard using Node-Red dashboard and ESP

1. Introduction
In this post, we will learn how to build an IoT Dashboard using Node-Redand node-red-dashboard.
These set-up can be deployed on Raspberry Pi, Orange Pi, ... easily.
We will make a demo for this post, a simple smart home demo: a floor has 1 bulb and 1 temperature sensor. They will be control by a ESP32. This ESP32 send and receive data using MQTT protocol. A server with MQTT broker, Node-Red, NodeJS and a dashboard with a chart to monitor temperature and a switch to control the bulb.
Figure: Demo model
Note: This post will re-use the posts:
How to turn the Orange Pi/Raspberry Pi into an IoT node : To install Mosquito and use host name instead of remembering the IP address
Demo 8: How to use TCP/IP with Arduino ESP32 : part 1.2 - Introduction to Node-Red (installation and usage)
Demo 14: How to use MQTT and Arduino ESP32 to build a simple Smart home system : build a smart home using Mosquito MQTT, ESP32 MQTT.
2. node-red-dashboard
Figure: node-red-dashboard
2.1 Concepts
With this dashboard:
+ Layout will be considered as a grid. It is in Dashboard Tab.
+ A group element has a width - by default 6 'units'.
+ A unit is 48px wide by default with a 6px gap.
+ A widget in the group also has a default 'auto'. It means it will fill the width of the group contained it, or you can set it to a fixed number of units.
2.2 Features
+ Layout: Tab, Link to other web pages.
+ Theme: Light, Dark or Custom Theme.
+ Widgets include Button, Chart, Form, Gauge, Notification, Switch ...
Note: the node-red-dashboard will be deployed at: http://localhost:1880/ui (change localhost accordingly)
2.3 Installation
Note: I instaled all things on my localhost
I assume that you did install NodeJS
Then install Mosquito broker following How to turn the Orange Pi/Raspberry Pi into an IoT node
Figure: ensuare MQTT broker is running 
Then you need to install Node-Red following Demo 8: How to use TCP/IP with Arduino ESP32 - part 1.
Then install node-red-dashboard. It requires Node-RED version 0.14 or more recent. Open Terminal and typing commands:
cd ~/.node-red
npm i node-red-dashboard
Then run Node-Red from Terminal using command:
node-red
You will see:
Figure: Node-Red started
Now open your web browser and go to the link http://127.0.0.1:1880/
If you use MDNS as in How to turn the Oavahi-daemonrange Pi/Raspberry Pi into an IoT node.
You will see:
Figure: left side is widgets area, right side is layout area, middile is working space of node-red-dashboard
3. Hardware
2 modules ESP32, an Raspberry Pi or Orange Pi or PC. I will use a Led for testing and generate temperature randomly.
4. Software:
We define MQTT topics and values:
    + For Led: "floor1/led" with value false-off, true-on
    + For temerature: "floor1/temp" with value is an integer or string
4.1 Node-Red side
The steps to create our model on Node-Red:
Step 1: Create 2 tabs (when deploying real website, it becomes 1 menu - 2 categories) standing for 2 floors.
Step 2: Create Tab 1 - Floor 1 has 2 groups: Group 1 contains 1 switch, Group 2 contains 1 chart.
Step 3: Create Tab 2 - Floor 2 has 2 groups: Group 1 contains 1 switch, Group 2 contains 1 Gauge.
Just follow red boxes and steps in order number.
 Figure: Node-Red GUI

Figure: Create Tab 1 - Floor 1
Figure: create Floor 1-Group 1-Switch
Figure: create MQTT publish when switch is pressed
Figure: create MQTT subscribe temperature sensor floor1/temp"
Figure: create a chart

Figure: deploy our model
Open Web Browser and go to the url: http://localhost:1880/ui
Our first draft:
Figure: first draft
Do similar things for Floor 2.
Change the theme to dark:
Figure: Dark theme
Our final model:
Figure: final model
Or you can import Node-Red model from script below:
[{"id":"bc6c4c2e.077c5","type":"mqtt in","z":"d4b07266.f397","name":"","topic":"floor2/temp","qos":"2","broker":"fc2a1412.370a68","x":284.5,"y":478,"wires":[["b9eef4aa.a11648"]]},{"id":"fc2a1412.370a68","type":"mqtt-broker","z":"","broker":"localhost","port":"1883","clientid":"","usetls":false,"compatmode":true,"keepalive":"60","cleansession":true,"willTopic":"","willQos":"0","willPayload":"","birthTopic":"","birthQos":"0","birthPayload":""}] 

4.2 ESP32 side
ESP32 Arduino code:
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
#include <WiFi.h>
#include <PubSubClient.h>

/* change it with your ssid-password */
const char* ssid = "dd-wrt";
const char* password = "0000000000";
/* this is the IP of PC/raspberry where you installed MQTT Server 
on Wins use "ipconfig" 
on Linux use "ifconfig" to get its IP address */
const char* mqtt_server = "192.168.1.107";
float temperature = 0;

/* create an instance of PubSubClient client */
WiFiClient espClient;
PubSubClient client(espClient);

/*LED GPIO pin*/
const char led = 4;

/* topics */
#define TEMP_TOPIC    "floor1/temp"
#define LED_TOPIC     "floor1/led" /* true=on, false=off */

long lastMsg = 0;
char msg[20];

void receivedCallback(char* topic, byte* payload, unsigned int length) {
  Serial.print("Message received: ");
  Serial.println(topic);

  Serial.print("payload: ");
  for (int i = 0; i < length; i++) {
    Serial.print((char)payload[i]);
  }
  Serial.println();
  /* we got '1' -> on */
  if ((char)payload[0] == 't') {
    digitalWrite(led, HIGH); 
  } else {
    /* we got '0' -> on */
    digitalWrite(led, LOW);
  }

}

void mqttconnect() {
  /* Loop until reconnected */
  while (!client.connected()) {
    Serial.print("MQTT connecting ...");
    /* client ID */
    String clientId = "ESP32Client";
    /* connect now */
    if (client.connect(clientId.c_str())) {
      Serial.println("connected");
      /* subscribe topic with default QoS 0*/
      client.subscribe(LED_TOPIC);
    } else {
      Serial.print("failed, status code =");
      Serial.print(client.state());
      Serial.println("try again in 5 seconds");
      /* Wait 5 seconds before retrying */
      delay(5000);
    }
  }
}

void setup() {
  Serial.begin(115200);
  // We start by connecting to a WiFi network
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  /* set led as output to control led on-off */
  pinMode(led, OUTPUT);

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());

  /* configure the MQTT server with IPaddress and port */
  client.setServer(mqtt_server, 1883);
  /* this receivedCallback function will be invoked 
  when client received subscribed topic */
  client.setCallback(receivedCallback);
}
void loop() {
  /* if client was disconnected then try to reconnect again */
  if (!client.connected()) {
    mqttconnect();
  }
  /* this function will listen for incomming 
  subscribed topic-process-invoke receivedCallback */
  client.loop();
  /* we measure temperature every 3 secs
  we count until 3 secs reached to avoid blocking program if using delay()*/
  long now = millis();
  if (now - lastMsg > 3000) {
    lastMsg = now;
    temperature = random(0, 40);
    if (!isnan(temperature)) {
      snprintf (msg, 20, "%lf", temperature);
      /* publish the message */
      client.publish(TEMP_TOPIC, msg);
    }
  }
}
5. Result

Wednesday, March 21, 2018

Demo 41: ESP32 connects with nRF24L01 2.4 GHz wireless chip

1. Introduction
Figure: Module nRF24L01
The nRF24L01 2.4 GHz wireless chip from Nordic Semiconductor. It has:
- SPI interface, hardware link layer, multiple pipelines, ...
- The chip is very cheap.
Comparing to Wifi modules such as ESP8266, ESP32:
- It has less power consumption.
- It is cheaper (but need a MCU to control).
- It has lower data rate.
If your Wifi infrastructure is good, you can consider using Wifi modules for your application.
If power consumption is not your issue, you can consider using Wifi modules for your application.
If we use nrf24l01 and need to send data to cloud via Internet, consider using ESP or Raspberry as a Gateway.
Some main characteristics:
1.1 NRF24L01 Hardware Interface
- 8 pins Vcc, GND, IRQ, CE, SPI pins (CSN (chip select not), SCK, MISO, and MOSI).
- IO pins are 5V-tolerant and operating range of 1.9 to 3.6V for Vcc.
- CSN, SCK, MISO, and MOSI for data transmission and reception.
- The CSN pin is active-low, and is normally kept high. When this pin goes low, the 24L01 begins listening on its SPI port for data and processes it accordingly.
- CE is used to control data transmission and reception when in TX and RX modes.
- IRQ is the interrupt pin, and is active-low. There are three internal interrupts that can cause this pin to go low when they are active. Each of these bits can be masked out such that when the bit’s respective interrupt becomes active, the status of the IRQ pin is not changed.
1.2 Interfacing the nRF24L01 via SPI
- The SPI interface allows you to read/write registers, transmit data, receive data from and to the 24L01. For start step we use SPI at 2 Mbps. The high SPI data rates are used if you needed huge on-air data rates.
1.3 SPI Instruction Set Summary
- In order to send data to or receive data from the SPI port on the nRF24L01:
  + The CSN pin on the 24L01 must be high then bring the CSN pin low to receive SPI data. Note: this pin will stay low throughout the entire transaction.
  + You will transmit the command byte of the instruction you wish to send. If you are receiving data bytes for this instruction, you must then send one byte to the 24L01 for every one byte that you wish to get out of the 24L01. If you are just sending the 24L01 data, you simply send your data bytes and generally don’t worry about what it sends back to you.
  + Once you have transmitted and/or read all of the bytes that you need, you bring CSN back high.
  + When you send any command byte, the 24L01 always returns to you the STATUS register.
1.4 FIFO Info
- There are FIFOs for both TX and RX modes.
- Both of the FIFOs  will hold the three newest packets that have been put into them. If you receive three packets in RX mode and you don’t read the payloads, the first (oldest) packet received is pushed out by the newest packet received. The same goes for the TX payload – if you load three packets and don’t transmit them (by executing the aforementioned CE toggle), then the fourth packet will push out the first packet you loaded.
1.5 Data Packet Format
- The transceiver protocol has 2 modes: Shockburst and Enhanced Shockburst
- The message format for both Shockburst and Enhanced Shockburst modes are different.
- In both modes, the preamble is sent first (1 byte), is used to allow the receiver to know that what it is hearing is the beginning of a packet and not just on-air noise.
- The next bytes sent are the address bytes. This is set by the user, and is between three and five bytes long.
- The next bytes sent is different between the two modes. In Enhanced Shockburst only, a flag word of nine bits is sent to indicate the message status concerning re-transmissions. Only two bits are currently used (a count of resent packets), and the other seven are reserved for future use.
- The last half of the packet that is sent is the same in both modes. The first of the fields to be sent in both modes is the payload data. The length of the payload is also set by the user (1 to 32 bytes long). The final part of the packet to be sent is the CRC, which is user-settable (0, 1, or 2 byte(s)).
Let 's make 2 demos:
- Demo 1: ESP32 Nano will send "hello" message to Arduino.
- Demo 2: Arduino Nano will send "hello" message to ESP32.
If you have 2 ESP32 you can replace Arduino Nano with ESP32.
2. Hardware
You need 2 Arduino/ESP8266/ESP32/Raspberry boards for testing: 1 board will send/receive message and 1 board will receive/send message which was sent by first board.
I used Arduino for testing, just do wiring like below:
Arduino_D8      x   NRF24_CSN
Arduino_D7      x   NRF24_CE
Arduino_D13    x   NRF24_SCK
Arduino_D11    x   NRF24_MOSI
Arduino_D12    x   NRF24_MISO

Arduino_3.3      x   NRF24_Vcc
Arduino_GND  x   NRF24_GND
For ESP32, I modified the library RF24 so that the software can use SPI software instead of using SPI hardware. Do wiring like below:
NRF24_CE      x   ESP32_IO12
NRF24_CSN    x   ESP32_IO14
NRF24_SCK    x   ESP32_IO26
NRF24_MISO  x   ESP32_IO25
NRF24_MOSI  x   ESP32_IO27
NRF24_Vcc      x   ESP32_3.3V
NRF24_GND    x   ESP32_GND  
3. Software
For ESP32, I modified the library RF24 so that the software can use SPI software insted of using SPI hardware. The new API has new form:
RF24(uint16_t _cepin, uint16_t _cspin, uint16_t sck, uint16_t miso, uint16_t mosi)
and to create an instance: RF24 radio(12, 14, 26, 25, 27);
For Arduino, we use SPI hardware so just use form:  
RF24(uint16_t _cepin, uint16_t _cspin)
and the instance is RF24 radio(7,8);
- Demo 1: ESP32 Nano will send "hello" message to Arduino.
ESP32 transmitter code
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
#include  <SPI.h>
#include "nRF24L01.h"
#include "RF24.h"
char msg[6] = "hello";
RF24 radio(12, 14, 26, 25, 27);
const uint64_t pipe = 0xE8E8F0F0E1LL;

void setup(void) {
  Serial.begin(115200);
  radio.begin();
  radio.setChannel(2);
  radio.setPayloadSize(7);
  radio.setDataRate(RF24_250KBPS);
  radio.openWritingPipe(pipe);
}
void loop(void) {
  Serial.println("send ...");
  radio.write(msg, 6);
  delay(3000);
}
Arduino receiver code
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#include <SPI.h>
#include "nRF24L01.h"
#include "RF24.h"
char msg[6];
RF24 radio(7,8);
const uint64_t pipe = 0xE8E8F0F0E1LL;
void setup(void){
 Serial.begin(115200);
 radio.begin();
 radio.setChannel(2);
 radio.setPayloadSize(7);
 radio.setDataRate(RF24_250KBPS);
 radio.openReadingPipe(1,pipe);
 radio.startListening();
}

void loop(void){
 if (radio.available()){  
     radio.read(msg, 6);      
     Serial.println(msg);
     delay(10);
 }
 else{
  //Serial.println("No radio available");
 }
}
- Demo 2: Arduino Nano will send "hello" message to ESP32.
ESP32 receiver code
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#include <SPI.h>
#include "nRF24L01.h"
#include "RF24.h"
char msg[6];
RF24 radio(12, 14, 26, 25, 27);
const uint64_t pipe = 0xE8E8F0F0E1LL;
void setup(void){
 Serial.begin(115200);
 radio.begin();
 radio.setChannel(2);
 radio.setPayloadSize(7);
 radio.setDataRate(RF24_250KBPS);
 radio.openReadingPipe(1,pipe);
 radio.startListening();
}

void loop(void){
 if (radio.available()){  
     radio.read(msg, 6);      
     Serial.println(msg);
     delay(10);
 }
 else{
  //Serial.println("No radio available");
 }
}
Arduino transmitter code

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
#include  <SPI.h>
#include "nRF24L01.h"
#include "RF24.h"
char msg[6] = "hello";
RF24 radio(7,8);
const uint64_t pipe = 0xE8E8F0F0E1LL;

void setup(void) {
  Serial.begin(115200);
  radio.begin();
  radio.setChannel(2);
  radio.setPayloadSize(7);
  radio.setDataRate(RF24_250KBPS);
  radio.openWritingPipe(pipe);
}
void loop(void) {
  Serial.println("send ...");
  radio.write(msg, 6);
  delay(3000);
}
4. Result
Figure: ESP32 received "hello" message from Arduino

Monday, January 22, 2018

Demo 40: Create a Facebook Messenger chat bot for monitoring and controlling home devices using Raspberry/Orange Pi and ESP32/8266

1. Introduction
In this demo, I will show you how to create a Facebook Messenger chat bot for monitoring and controlling home devices using Raspberry/Orange Pi and ESP32/8266. This is not a full solution so you need to improve it by yourself.
Figure: Facebook Messenger chat bot for monitoring and controlling home devices
There are 2 steps to make this demo:
- How to setup a Facebook chat bot
- How to setup local system (Raspberry/Orange Pi, ESP32/8266)
This is the model of the demo:
Figure: The model of the demo
2. Setup
2.1 Create Facebook page
You can refer to this guideline. And choose type of Page is Business, Brand or Place. In my demo, I created a page named IoT Sharing.
2.2 Setup Facebook chat bot
In order to develop or use Facebook services for Software development you need to register a Facebook developer account here.
After created a FB developer account choose My Apps -> Add a new App -> Fill Display name
In Select a Product -> choose messenger -> Set Up
Choose Webhooks -> Setup Webhooks
Note: We do not do this step now. This step will be done after running the Raspbbery/Orange Pi software. Because after filled Callback URL, Verify Token and press Verify and Save, FB will send a GET request with "hub.challenge=value of Verify Token" (in this case Verify Token is "iotsharing.com" string) to Pi. Pi need to check the value of hub.challenge must match the value that we filled in Verify Token field (in this case Verify Token is "iotsharing.com" string). And then send this hub.challenge back to FB to finish the verification.
1
2
3
4
5
6
@app.route('/', methods=['GET'])
def handle_verification():
 if(request.args['hub.challenge'] == 'iotsharing.com'):
  return request.args['hub.challenge']
 else:
  return 'not match'
Fill information like below and choose Verify and Save.
choose the Page that you created in previous step and Subscribe/Unsubscribe.
Finally, extract the ACCESS_TOKEN of the FB app so that our application can authenticate to use FB services.
It is done for Facebook setup step.
2.2 Setup local system (Raspberry/Orange Pi, ESP32/8266)
In this demo, Pi will keep some roles:
- Communicate with Facebook server to receive and respond message.
- Parse Facebook message and then using MQTT protocol to publish the commands to ESP32/8266 clients and subscribe responses from ESP32/8266 clients.
- Refer to this post to setup MQTT for Pi.
- We use Python Paho MQTT for local communication. Refer to this post to install it.
- Besides we need to install some packages (Flask, ngrok) on Raspberry/Orange Pi for our demo. 
+ Flask is a Python web framework. In our demo, It is a web server to handle Facebook https request.
+ ngrok secure introspect-able tunnels to local host web hook development tool and debugging tool. This tool helps Facebook server can see our Flask local web server.
From Pi Terminal running the commands:
+ Install Flask Python server: "sudo pip install Flask"
+ Download ngrok: "wget https://bin.equinox.io/a/26q6mq7ddJR/ngrok-2.2.9-linux-arm.zip"
+ Unzip zip file for executable app: "unzip ngrok-2.2.9-linux-arm.zip"
3. Hardware
- 1 ESP32 or ESP8266
- 1 Raspberry Pi or Orange Pi
- 1 LED connect to ESP
- 1 temperature sensor connect to ESP. 
In this demo I do not use temperature sensor. I used random(min, max) function to generate temperature.
4. Software
The source code of this demo can be found on github.
Define messages:
- Pi send MQTT request topics to ESP:
Temperature topic: "floor1/room1/temp1" to measure temperature
Led topic:               "floor1/room1/led1" to set "0" (off) or "1" (on)
- ESP send MQTT response topics to Pi:
Temperature response topic: "floor1/room1/temp1/res" with temperature value
Led response topic:               "floor1/room1/led1/res" to inform that the request was executed
- The template for Facebook message:
"set floor1/room1/led1 on" -> turn on LED and get response
"set floor1/room1/led1 off" -> turn off LED and get response
"get floor1/room1/temp1" -> request temperature measurement and get response
ESP code I reused the post (esp32chatbot.ino)
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
#include <WiFi.h>
#include <PubSubClient.h>

const char* ssid = "dd-wrt";
const char* password = "0000000000";
//ip address of Raspberry/orange pi
const char* mqtt_server = "192.168.1.106";
char msg[20];

/* create an instance of PubSubClient client */
WiFiClient espClient;
PubSubClient client(espClient);

/*LED GPIO pin*/
const char led = 4;
/* topics */
#define TEMP_TOPIC    "floor1/room1/temp1"
#define LED_TOPIC     "floor1/room1/led1" /* on, off */

void receivedCallback(char* topic, byte* payload, unsigned int length) {
    Serial.print("topic: ");
    Serial.println(topic);
      if(strcmp(topic, LED_TOPIC) == 0){
        Serial.print("payload: ");
        for (int i = 0; i < length; i++) {
          Serial.print((char)payload[i]);
        }
        Serial.println();
        /* we got '1' -> on */
        if ((char)payload[0] == '1') {
          digitalWrite(led, HIGH); 
          snprintf (msg, 20, "%s", "on");
        /* publish the response */
        client.publish(LED_TOPIC "/res", msg);
        } else {
          /* we got '0' -> on */
          digitalWrite(led, LOW);
          snprintf (msg, 20, "%s", "off");
          client.publish(LED_TOPIC "/res", msg);
        }
    }else {
        snprintf (msg, 20, "%d", random(0, 40));
        client.publish(TEMP_TOPIC "/res", msg);
    }
}

void mqttconnect() {
  /* Loop until reconnected */
  while (!client.connected()) {
    Serial.print("MQTT connecting ...");
    /* client ID */
    String clientId = "ESP32Client";
    /* connect now */
    if (client.connect(clientId.c_str())) {
      Serial.println("connected");
      /* subscribe topic */
      client.subscribe(LED_TOPIC);
      client.subscribe(TEMP_TOPIC);
    } else {
      Serial.print("failed, status code =");
      Serial.print(client.state());
      Serial.println("try again in 5 seconds");
      /* Wait 5 seconds before retrying */
      delay(5000);
    }
  }
}

void setup() {
  Serial.begin(115200);
  // We start by connecting to a WiFi network
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  /* set led as output to control led on-off */
  pinMode(led, OUTPUT);

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());

  /* configure the MQTT server with IPaddress and port */
  client.setServer(mqtt_server, 1883);
  /* this receivedCallback function will be invoked 
  when client received subscribed topic */
  client.setCallback(receivedCallback);
}
void loop() {
  /* if client was disconnected then try to reconnect again */
  if (!client.connected()) {
    mqttconnect();
  }
  /* this function will listen for incomming 
  subscribed topic-process-invoke receivedCallback */
  client.loop();
}
Pi code (chatbot.py):
We use variable history to hold the id of Facebook messenger sender while waiting for the response from ESP. This sender id is used in reply() function to send response back to Facebook messenger sender.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
from flask import Flask, request
import requests
import paho.mqtt.publish as publish
import paho.mqtt.client as mqtt
import thread

ACCESS_TOKEN = "your_fb_access token_here";

#hold sender id
history = dict()

set_cmd = 'floor1/room1/led1'
get_cmd = 'floor1/room1/temp1'

#respond to FB messenger
def reply(user_id, msg):
    data = {
        "recipient": {"id": user_id},
        "message": {"text": msg}
    }
    resp = requests.post("https://graph.facebook.com/v2.6/me/messages?access_token=" + ACCESS_TOKEN, json=data)
    print(resp.content)

#MQTT handler
def on_connect(mqttc, obj, flags, rc):
    print("rc: "+str(rc))
def on_message(mqttc, obj, msg):
    print(msg.topic+": "+str(msg.payload))
    if msg.topic in history:
        user_id = history[msg.topic]
  #reply FB messenger
        reply(user_id, msg.payload)
        history.pop(msg.topic, None)

mqttc = mqtt.Client()
mqttc.on_connect = on_connect
mqttc.on_message = on_message
mqttc.connect("localhost", 1883, 60)
mqttc.subscribe("floor1/#", 0)

#MQTT subscribe thread 
def mqtt_thread( threadName, delay):
   mqttc.loop_forever()
try:
   thread.start_new_thread( mqtt_thread, ("mqtt-thread", 0, ) )
except:
   print "Error: unable to start thread"

#Flask web server instance
app = Flask(__name__)

#handle GET request from Facebook
@app.route('/', methods=['GET'])
def handle_verification():
 if(request.args['hub.challenge'] == 'iotsharing.com'):
  return request.args['hub.challenge']
 else:
  return 'not matched'

#handle POST request from Facebook
@app.route('/', methods=['POST'])
def handle_incoming_messages():
    data = request.json
    print(data)
    sender = data['entry'][0]['messaging'][0]['sender']['id']
    message = data['entry'][0]['messaging'][0]['message']['text']
    print(message)
    if(message.startswith('set')):
        arr = message.split()
        l = len(arr)
        if(l == 3 and set_cmd == arr[1]):
            cmd = arr[1]
            val = '1' if arr[2]=='on' else '0'
            publish.single(cmd, val, hostname="localhost")
   #record command with sender id
            history[cmd+'/res'] = sender
            return 'ok'
    elif(message.startswith('get')):
        arr = message.split()
        l = len(arr)
        if(l == 2 and get_cmd == arr[1]):
            cmd = arr[1]
            publish.single(cmd, '', hostname="localhost")
   #record command with sender id
            history[cmd+'/res'] = sender
            return 'ok'
    reply(sender, 'invalid query')
    return "ok"

if __name__ == '__main__':
app.run(debug=True)
5. Steps to deploy
- MQTT broker run on Pi
- Open 2 Terminal on Pi:
   + Run "python chatbot.py" with Flask in it.
   + Run ngrok (5000 is the port Flask listen on): "./ngrok http 5000"

Copy the url in red box to Callback URL field that mention in step 2.2 Setup Facebook chat bot
Note: in case we shutdown ngrok and run it again the new url will be generated. We have to re-register this url to Facebook.
Choose Webhooks -> Edit Subscription
And then Unsubcribe and Subscribe the page again:
Now go to the Page that we created, choose About and Send Message
And type one of commands:
"set floor1/room1/led1 on"
"set floor1/room1/led1 off"
"get floor1/room1/temp1"

6. Result



Tuesday, January 16, 2018

Demo 39: ESP32/8266 multipart upload a file and download a file via HTTP

1. Introduction
In previous demos, I showed you how to use MQTT/MQTTS and how to update firmware OTA (TCP/UDP and HTTP). In this demo, I will show you another interesting topic. That is how to download and multi-part upload a file via HTTP.
Figure: multipart upload and download a file via http
Based on this demo, you can build an application that can download the new firmware or the configuration file to memory card that is attached to client board for using later. Or you can upload the log file that records occurred events in run time to a server.
This is the requirement of this demo: a ESP32 will download a file from internet and after finish downloading, upload it to local apache server. The downloaded file will be stored in sdcard.
In order to make it easy, I created a library here in github. You just install and use it.
Note: this library can be apply for ESP8266.
2. Software
The library is simple to use. It has 2 APIs: upload() and download() with parameters. The parameters are:
- url of uploading and downloading file.
- progress displaying callback function.
- response processing callback function.
- reading and writing data to storage (sdcard, SPIFFS, ...) callback functions.
You should change them accordingly to your application.
Note: The url parser of this library is quite simple and it will be updated later.
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
#include "UDHttp.h"
#include "mySD.h"


const char* ssid     = "dd-wrt";
const char* password = "0000000000";

File root;
//these callbacks will be invoked to read and write data to sdcard
//and process response
//and showing progress 
int responsef(uint8_t *buffer, int len){
  Serial.printf("%s\n", buffer);
  return 0;
}
//read data callback
int rdataf(uint8_t *buffer, int len){
  //read file to upload
  if (root.available()) {
    return root.read(buffer, len);
  }
  return 0;
}
//write data callback
int wdataf(uint8_t *buffer, int len){
  //write downloaded data to file
  return root.write(buffer, len);
}

void progressf(int percent){
  Serial.printf("%d\n", percent);
}

void setup() {
  // put your setup code here, to run once:
  Serial.begin(115200);

  WiFi.begin(ssid, password);

  while (WiFi.status() != WL_CONNECTED) {
      delay(500);
      Serial.print(".");
  }

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
    
  Serial.print("Initializing SD card...");
  if (!SD.begin(32, 14, 12, 27)) {
    Serial.println("initialization failed!");
    return;
  }
  Serial.println("initialization done.");
  SD.remove("test.pdf");
  {
    UDHttp udh;
    //open file on sdcard to write
    root = SD.open("test.pdf", FILE_WRITE);
    if (!root) {
       Serial.println("can not open file!");
       return;
    }
    //download the file from url
    udh.download("http://www.smart-words.org/linking-words/linking-words.pdf", wdataf, progressf);
    root.close();
    Serial.printf("done downloading\n");
  }
  {
    UDHttp udh;
    //open file on sdcard to read
    root = SD.open("test.pdf");
    if (!root) {
       Serial.println("can not open file!");
       return;
    }
    //upload downloaded file to local server
    udh.upload("http://192.168.1.107:80/upload/upload.php", "test.pdf", root.size(), rdataf, progressf, responsef);
    root.close();
    Serial.printf("done uploading\n");
  }
}

void loop() {
  // put your main code here, to run repeatedly:

}
And the php script was used for this demo:
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
<?php
$target_path = "uploads/";

$target_path = $target_path . basename( $_FILES['data']['name']); 

if(move_uploaded_file($_FILES['data']['tmp_name'], $target_path)) {
    echo "The file ".  basename( $_FILES['data']['name']). 
    " has been uploaded";
} else{
    echo "There was an error uploading the file, please try again!";
}
?>
Note: The variable $_FILES['data']['name'] has fixed 'data' field name (similar to html form ""). The downloaded file will be stored in sub-folder 'uploads' which is relative to folder where the php script located. Reference this post to install local apache server. 
3. Hardware
For harware connection please refer here.

4. Result
Figure: start downloading with progress displaying

Thursday, December 7, 2017

Demo 38: How to decode error/exception "CPU halted" of ESP on Arduino

1. Introduction
When developing software for ESP8266 or ESP32, you often face error(s) that caused the ESP halt.  At that moment, looking the Serial Monitor window, you will see the error/exception like below:
Figure: error that caused ESP CPU halt
It is not easy to know what was happening with our software. Actually, the string that is highlighted has meaning; it is the called stack of software when the CPU halted. There is a tool that supports you to decode the ESP error/exception above. It called EspExceptionDecoder. It is tool that is attached with Arduino IDE.
In order to install it, please follow these steps:
    - Install Arduino IDE with ESP8266/ESP32 core.
    - Download the tool here.
    - Create tools directory under Arduino directory if it is not exist.
    - Unpack the downloaded tool into tools directory (the path will look like /Arduino/tools/EspExceptionDecoder/tool/EspExceptionDecoder.jar).
    - Restart Arduino IDE.



Figure: Under Tools menu, new entry ESP Exception Decoder
2. Demo
I will make a simple program demo for this tool.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
void setup() {
  Serial.begin(115200);
  char *buf = NULL;
  buf[2] = 4;
  Serial.printf("val %d\n", buf[2]);
}

/* the forever loop() function is invoked by Arduino ESP32 loopTask */
void loop() {
}
Here the bug is that buf is not initialized buf buf[2] is assigned to 4.
Loading the software to ESP and look the Serial Monitor:
Figure: error in software
After that choosing Tools->ESP Exception Decoder, a window will occur.
 Figure: Exception Decoder window

Copy and paste the string "Backtrace: 0x400d05be:0x3ffc77c0 0x400d9b1a:0x3ffc77e0 " to it and you will see the output:
 Figure: the result after decoding stacktrace
So you see the CPU halted executing the code at line 4: buf[2] = 4; as we knew.
This is just a simple application. In real world, you may face a more difficult situation than this, but at least you could know where your application is halted.