1. Introduction
In this demo, I will show you how to make a camera live stream application with ESP32 Cam.
1.1 Problems of HTTP
- Request/Response
- Stateless
- Half duplex protocol
The web client sends request to web server, the web server send response and the connection close. If the the client want to know a continuous state change on the server, It has to send a request to server every specific time to get the state change on the server. this is polling. It is inefficient and waste resources
1.2 WebSocket
In order to solve the problems of HTTP, WebSocket was born. It is:
- Based on the TCP protocol
- Uses the HTTP protocol on the handshake phase
- The protocol identifier is ws (ws://iotsharing.com:80/)
- After the connection is established, it will be keep alive. So client and server can send messages to each other. It is full duplex protocol.
I used the camera module:
3.1 WebSocket Server for ESP32
We will use this WebSocket library. We will make a simple demo to get familiar with it. In this demo ESP32 will act as a WebSocket server, it will send the HTTP index page to web browser client (follow Demo 12). After loaded the index page, a javascript using jquery will create a WebSocket client that connects to WebSocket server. On server a counter will continuously sending counter value to client and display this value to web browser.
Note: we have 2 servers: simple HTTP Web server (follow Demo 12) and WebSocket server.
This library will catch some event from WebSocket. We will use CONNECTED event to send counter value to web browser.
3.2 Camera driver
The driver for OV2640 cam is available with ESP32 Arduino, you need to configure pins definitions, pixel_format, frame_size and jpeg_quality for camera then you can use it.
3.3 Combine WebSocket vs Camera
When WebSocket client connected to WebSocket server we start streaming the camera to client using sendBIN(). This function sends camera frame buffer to client. At client side we need to convert this buffer stream to base64 so that it can be displayed to
tag of HTML.
The full code:
4. Result
The display frame is not really smooth.
In this demo, I will show you how to make a camera live stream application with ESP32 Cam.
1.1 Problems of HTTP
- Request/Response
- Stateless
- Half duplex protocol
The web client sends request to web server, the web server send response and the connection close. If the the client want to know a continuous state change on the server, It has to send a request to server every specific time to get the state change on the server. this is polling. It is inefficient and waste resources
1.2 WebSocket
In order to solve the problems of HTTP, WebSocket was born. It is:
- Based on the TCP protocol
- Uses the HTTP protocol on the handshake phase
- The protocol identifier is ws (ws://iotsharing.com:80/)
- After the connection is established, it will be keep alive. So client and server can send messages to each other. It is full duplex protocol.
Figure: HTTP vs WebSocket (source)
Figure: WebSocket protocol
2. HardwareI used the camera module:
Figure: ESP32 CAM with OV2640 cam
3. Software 3.1 WebSocket Server for ESP32
We will use this WebSocket library. We will make a simple demo to get familiar with it. In this demo ESP32 will act as a WebSocket server, it will send the HTTP index page to web browser client (follow Demo 12). After loaded the index page, a javascript using jquery will create a WebSocket client that connects to WebSocket server. On server a counter will continuously sending counter value to client and display this value to web browser.
Note: we have 2 servers: simple HTTP Web server (follow Demo 12) and WebSocket server.
#include <WiFi.h> #include <WebSocketsServer.h> WebSocketsServer webSocket = WebSocketsServer(81); WiFiServer server(80); String index_html = "<html>\n \ <head>\n \ <title> WebSockets Client</title>\n \ <script src='http://code.jquery.com/jquery-1.9.1.min.js'></script>\n \ </head>\n \ <body>\n \ <div id='output'></div>\n \ </body>\n \ </html>\n \ <script>\n \ jQuery(function($){\n \ if (!('WebSocket' in window)) {\n \ alert('Your browser does not support web sockets');\n \ }else{\n \ setup();\n \ }\n \ function setup(){\n \ var host = 'ws://server_ip:81';\n \ var socket = new WebSocket(host);\n \ if(socket){\n \ socket.onopen = function(){\n \ }\n \ socket.onmessage = function(msg){\n \ showServerResponse(msg.data);\n \ }\n \ socket.onclose = function(){\n \ showServerResponse('The connection has been closed.');\n \ }\n \ }\n \ function showServerResponse(txt){\n \ document.getElementById('output').innerHTML = txt;\n \ }\n \ }\n \ });\n \ </script>"; void hexdump(const void *mem, uint32_t len, uint8_t cols = 16) { const uint8_t* src = (const uint8_t*) mem; Serial.printf("\n[HEXDUMP] Address: 0x%08X len: 0x%X (%d)", (ptrdiff_t)src, len, len); for(uint32_t i = 0; i < len; i++) { if(i % cols == 0) { Serial.printf("\n[0x%08X] 0x%08X: ", (ptrdiff_t)src, i); } Serial.printf("%02X ", *src); src++; } Serial.printf("\n"); } void webSocketEvent(uint8_t num, WStype_t type, uint8_t * payload, size_t length) { switch(type) { case WStype_DISCONNECTED: Serial.printf("[%u] Disconnected!\n", num); break; case WStype_CONNECTED: { int counter = 0; while(true){ counter++; String n = String(counter); webSocket.sendTXT(num, n); delay(1000); } } break; case WStype_TEXT: case WStype_BIN: case WStype_ERROR: case WStype_FRAGMENT_TEXT_START: case WStype_FRAGMENT_BIN_START: case WStype_FRAGMENT: case WStype_FRAGMENT_FIN: break; } } void setup() { Serial.begin(115200); WiFi.begin("I3.41", "xxx"); Serial.println(""); while (WiFi.status() != WL_CONNECTED) { delay(500); Serial.print("."); } Serial.println(""); String IP = WiFi.localIP().toString(); Serial.print("IP address: " + IP); index_html.replace("server_ip", IP); server.begin(); webSocket.begin(); webSocket.onEvent(webSocketEvent); } void http_resp(){ WiFiClient client = server.available(); if (client.connected() && client.available()) { client.flush(); client.print(index_html); client.stop(); } } void loop() { http_resp(); webSocket.loop(); }
3.2 Camera driver
The driver for OV2640 cam is available with ESP32 Arduino, you need to configure pins definitions, pixel_format, frame_size and jpeg_quality for camera then you can use it.
#define PWDN_GPIO_NUM 32 #define RESET_GPIO_NUM -1 #define XCLK_GPIO_NUM 0 #define SIOD_GPIO_NUM 26 #define SIOC_GPIO_NUM 27 #define Y9_GPIO_NUM 35 #define Y8_GPIO_NUM 34 #define Y7_GPIO_NUM 39 #define Y6_GPIO_NUM 36 #define Y5_GPIO_NUM 21 #define Y4_GPIO_NUM 19 #define Y3_GPIO_NUM 18 #define Y2_GPIO_NUM 5 #define VSYNC_GPIO_NUM 25 #define HREF_GPIO_NUM 23 #define PCLK_GPIO_NUM 22 void configCamera(){ camera_config_t config; config.ledc_channel = LEDC_CHANNEL_0; config.ledc_timer = LEDC_TIMER_0; config.pin_d0 = Y2_GPIO_NUM; config.pin_d1 = Y3_GPIO_NUM; config.pin_d2 = Y4_GPIO_NUM; config.pin_d3 = Y5_GPIO_NUM; config.pin_d4 = Y6_GPIO_NUM; config.pin_d5 = Y7_GPIO_NUM; config.pin_d6 = Y8_GPIO_NUM; config.pin_d7 = Y9_GPIO_NUM; config.pin_xclk = XCLK_GPIO_NUM; config.pin_pclk = PCLK_GPIO_NUM; config.pin_vsync = VSYNC_GPIO_NUM; config.pin_href = HREF_GPIO_NUM; config.pin_sscb_sda = SIOD_GPIO_NUM; config.pin_sscb_scl = SIOC_GPIO_NUM; config.pin_pwdn = PWDN_GPIO_NUM; config.pin_reset = RESET_GPIO_NUM; config.xclk_freq_hz = 20000000; config.pixel_format = PIXFORMAT_JPEG; config.frame_size = FRAMESIZE_QVGA; config.jpeg_quality = 9; config.fb_count = 1; esp_err_t err = esp_camera_init(&config); if (err != ESP_OK) { Serial.printf("Camera init failed with error 0x%x", err); return; } }
When WebSocket client connected to WebSocket server we start streaming the camera to client using sendBIN(). This function sends camera frame buffer to client. At client side we need to convert this buffer stream to base64 so that it can be displayed to
img.src = 'data:image/jpg;base64,'+window.btoa(binary);
#include "esp_camera.h" #include <WiFi.h> #include <WebSocketsServer.h> #define PWDN_GPIO_NUM 32 #define RESET_GPIO_NUM -1 #define XCLK_GPIO_NUM 0 #define SIOD_GPIO_NUM 26 #define SIOC_GPIO_NUM 27 #define Y9_GPIO_NUM 35 #define Y8_GPIO_NUM 34 #define Y7_GPIO_NUM 39 #define Y6_GPIO_NUM 36 #define Y5_GPIO_NUM 21 #define Y4_GPIO_NUM 19 #define Y3_GPIO_NUM 18 #define Y2_GPIO_NUM 5 #define VSYNC_GPIO_NUM 25 #define HREF_GPIO_NUM 23 #define PCLK_GPIO_NUM 22 WebSocketsServer webSocket = WebSocketsServer(81); WiFiServer server(80); uint8_t cam_num; bool connected = false; String index_html = "<html>\n \ <head>\n \ <title> WebSockets Client</title>\n \ <script src='http://code.jquery.com/jquery-1.9.1.min.js'></script>\n \ </head>\n \ <body>\n \ <img id='live' src=''>\n \ </body>\n \ </html>\n \ <script>\n \ jQuery(function($){\n \ if (!('WebSocket' in window)) {\n \ alert('Your browser does not support web sockets');\n \ }else{\n \ setup();\n \ }\n \ function setup(){\n \ var host = 'ws://server_ip:81';\n \ var socket = new WebSocket(host);\n \ socket.binaryType = 'arraybuffer';\n \ if(socket){\n \ socket.onopen = function(){\n \ }\n \ socket.onmessage = function(msg){\n \ var bytes = new Uint8Array(msg.data);\n \ var binary= '';\n \ var len = bytes.byteLength;\n \ for (var i = 0; i < len; i++) {\n \ binary += String.fromCharCode(bytes[i])\n \ }\n \ var img = document.getElementById('live');\n \ img.src = 'data:image/jpg;base64,'+window.btoa(binary);\n \ }\n \ socket.onclose = function(){\n \ showServerResponse('The connection has been closed.');\n \ }\n \ }\n \ }\n \ });\n \ </script>"; void configCamera(){ camera_config_t config; config.ledc_channel = LEDC_CHANNEL_0; config.ledc_timer = LEDC_TIMER_0; config.pin_d0 = Y2_GPIO_NUM; config.pin_d1 = Y3_GPIO_NUM; config.pin_d2 = Y4_GPIO_NUM; config.pin_d3 = Y5_GPIO_NUM; config.pin_d4 = Y6_GPIO_NUM; config.pin_d5 = Y7_GPIO_NUM; config.pin_d6 = Y8_GPIO_NUM; config.pin_d7 = Y9_GPIO_NUM; config.pin_xclk = XCLK_GPIO_NUM; config.pin_pclk = PCLK_GPIO_NUM; config.pin_vsync = VSYNC_GPIO_NUM; config.pin_href = HREF_GPIO_NUM; config.pin_sscb_sda = SIOD_GPIO_NUM; config.pin_sscb_scl = SIOC_GPIO_NUM; config.pin_pwdn = PWDN_GPIO_NUM; config.pin_reset = RESET_GPIO_NUM; config.xclk_freq_hz = 20000000; config.pixel_format = PIXFORMAT_JPEG; config.frame_size = FRAMESIZE_QVGA; config.jpeg_quality = 9; config.fb_count = 1; esp_err_t err = esp_camera_init(&config); if (err != ESP_OK) { Serial.printf("Camera init failed with error 0x%x", err); return; } } void liveCam(uint8_t num){ //capture a frame camera_fb_t * fb = esp_camera_fb_get(); if (!fb) { Serial.println("Frame buffer could not be acquired"); return; } //replace this with your own function webSocket.sendBIN(num, fb->buf, fb->len); //return the frame buffer back to be reused esp_camera_fb_return(fb); } void webSocketEvent(uint8_t num, WStype_t type, uint8_t * payload, size_t length) { switch(type) { case WStype_DISCONNECTED: Serial.printf("[%u] Disconnected!\n", num); break; case WStype_CONNECTED: cam_num = num; connected = true; break; case WStype_TEXT: case WStype_BIN: case WStype_ERROR: case WStype_FRAGMENT_TEXT_START: case WStype_FRAGMENT_BIN_START: case WStype_FRAGMENT: case WStype_FRAGMENT_FIN: break; } } void setup() { Serial.begin(115200); WiFi.begin("I3.41", "xxx"); Serial.println(""); while (WiFi.status() != WL_CONNECTED) { delay(500); Serial.print("."); } Serial.println(""); String IP = WiFi.localIP().toString(); Serial.print("IP address: " + IP); index_html.replace("server_ip", IP); server.begin(); webSocket.begin(); webSocket.onEvent(webSocketEvent); configCamera(); } void http_resp(){ WiFiClient client = server.available(); if (client.connected() && client.available()) { client.flush(); client.print(index_html); client.stop(); } } void loop() { http_resp(); webSocket.loop(); if(connected == true){ liveCam(cam_num); } }
The display frame is not really smooth.
6 Comments
do you know how can I do it as a tunneling?
to access it over NAT and Firewall?
I'm a beginner in Websocket protocol. is it necessary to upload web page every loop in arduino? I mean, can Javascript handle stream data? Thank you
i have a question how can I enhance image quality. like white balancing. brightness change.?
May I ask you a question??
Referring to this amazing code, I am going to leave the server external and use esp32-cam as a client.
I wrote the code and it works, but sending to the websocket takes a long time.
Only about 1 data per second is transmitted. Can you please help??