Streaming ESP32-CAM Images to Multiple Browsers via MQTT

In this tutorial, you'll learn how to publish images from an ESP32-CAM board to multiple browsers using MQTT (Message Queuing Telemetry Transport). This setup will enable you to create a platform that functions similarly to a live video stream, viewable by unlimited users. We will also show how to transform this stream of JPEG images into a continuous video using Python and ffmpeg for real-time encoding, resulting in one-hour segmented videos, each compactly viewable in about 4 minutes. Ideal for time-lapse videos and surveillance, this setup also introduces AI-powered face recognition, further enhancing surveillance efficiency by pinpointing moments of human presence in the footage for easy reference and review.

This hands-on tutorial includes ready-to-use source code! Curious why we chose Lua? Take a look at the tutorial: Why Smart C Coders Use Lua.

This tutorial builds on what you learned in the tutorial Your First Xedge32 Project.

Prerequisites

This hands-on tutorial teaches you how to use a ready-to-run ESP32 CAM Lua script. Simply install Xedge32 on your ESP32 CAM board, and you'll be set to execute the Lua script. Xedge32 works with most ESP32 and ESP32-S3 CAM boards and offers a suite of protocols, including MQTT.

Before diving in, make sure you have completed the following prerequisite tutorials:

• Purchase an ESP32 cam board such as the XIAO ESP32S3 Sense.
• Your First MQTT Lua Program: This tutorial introduces the basics of MQTT and how to write a simple Lua program to interact with MQTT.

By building on the knowledge gained from these foundational tutorials, you'll be better equipped to follow along with this tutorial.

Publishing ESP32-CAM Images via MQTT

In the MQTT CAM code, our primary focus is publishing images without subscribing to other events. This publishing operation is managed by a timer event, which publishes images based on the intervals specified.

Setting Up the Timer

First, let's create a timer object. This timer will trigger the publishImage function at specific intervals.

To interact with the ESP32 camera, initialize a camera object like so:

The cfg parameter represents a configuration table. Important: make sure it matches the settings for your particular ESP32-CAM module. See the Lua CAM API for details.

Handling MQTT Connection Status

For monitoring MQTT connections, use the following callback function:

The above function starts the timer when a successful MQTT connection is made. If the connection drops, it cancels the timer but will keep attempting to reconnect.

Image Publishing via Timer Callback

The core of the image publishing mechanism is the timer callback function, publishImage. This function captures an image using the camera object and publishes it via MQTT. The timer logic supports various timer types. Notably, this version operates as a Lua coroutine (akin to a thread). Within this coroutine, it continually loops and hibernates for the duration defined by the timer through coroutine.yield(true).

The above function maintains flow control by not publishing an image if two images already populate the MQTT client's send queue. The cam:read function can be time-consuming -- not in human time, but in terms of microcontroller operations. As such, we offload the task of reading from the CAM object onto a separate thread. While this step isn't strictly necessary, it enhances the performance of applications juggling multiple operations alongside reading from the CAM. For a deeper dive into the threading intricacies, you are encouraged to refer to the Barracuda App Server’s documentation on threading.

The following shows the complete MQTT CAM code:

While we have already covered the majority of the program's functionality, there are a few aspects we haven't touched upon yet:

• Topic and Broker Configuration:

  • local topic = "xedge32/espcam/USA/92629": Sets the MQTT topic where the images will be published. Change this topic to your address.
  • local broker = "broker.hivemq.com": Specifies the MQTT broker's address. The public HiveMQ broker is used in this example.

• ESP32 Camera Configuration (cfg): This block sets up the specific pin configurations and settings for your ESP32 CAM board. Replace these settings with those appropriate for your hardware.

• Creating the MQTT Client: The MQTT client is created with the require("mqttc").create(broker, onstatus) function, passing in the broker address and the onstatus callback.

• Creating the Timer Object for publishImage: The timer is created by calling ba.timer and passing in the publishImage callback, which will be activated at regular intervals. This is the mechanism that continually captures and publishes images.

Subscribing to CAM Images with a JavaScript-Powered HTML Client

To visualize the images published by the ESP32 camera, you can use an HTML client. The following client will subscribe to the same MQTT topic to which the camera is publishing images. The client runs purely in your web browser and does not require any server setup.

The entire code for the HTML client is shown below:

MQTT JavaScript client

At the top of the HTML file, the MQTT JavaScript library is imported to enable MQTT functionalities. This is found within the <script src=".......mqtt.min.js"></script> line.

Body Layout

The HTML body contains a <div> element with an id of "image-container" that will house the incoming images, and a <p> element with an id of "msg" that serves as a placeholder for status messages.

MQTT Configuration

In the JavaScript section, two constants topic and broker are defined. These must correspond to the topic and broker configurations in your mqttcam.xlua file.

Connecting to MQTT Broker

The client initiates an MQTT connection to the specified broker using the mqtt.connect() method. It uses a secure websocket port 8884 for this connection.

Handling Incoming Messages

Upon a successful connection, the client subscribes to the topic. Any incoming message on this topic is expected to be a binary JPEG image. The message is converted into a Blob and displayed as the source for the image element.

Frame Counter

A frameCounter variable keeps count of the incoming frames (or images) and displays this count as a text message below the displayed image.

By having this HTML file open in a web browser, you'll be able to visualize in real-time the images that are being published to the specified MQTT topic.

Preparing the Code

Step 1: Prepare the Lua Script as Follows:

1. As explained in the tutorial Your First Xedge32 Project, when the Xedge32 powered ESP32 is running, use a browser and navigate to the Xedge IDE.
2. Create a new Xedge app called "cam" and LSP enable the app.
3. Expand the cam app now visible in the left pane tree view.
4. Right-click the cam app and click New File in the context menu.
5. Type camtest.lsp and click Enter.
6. Open the camtest.lsp file at GitHub and click the copy raw file button.
7. Go to the Xedge IDE browser window and paste the content into the camtest.lsp file.
8. Important: Adjust the cfg settings in camtest.lsp to match your specific ESP32 CAM board. See the Lua CAM API for details.
9. Click Save and then Click Open to test your cam settings. Make sure you see the image generated by the LSP script before proceeding.
10. Right-click the cam app and click New File in the context menu.
11. Type mqttcam.xlua and click Enter.
12. Open the mqttcam.xlua file at GitHub and click the copy raw file button.
13. Go to the Xedge IDE browser window and paste the content into the mqttcam.xlua file.
14. Using the Xedge editor, update the topic variable xedge32/espcam/USA/92629 in the Lua script to your desired MQTT topic.
15. Important: Copy the cfg settings from camtest.lsp and replace the cfg settings in mqttcam.xlua with the settings you tested in step 9.
16. Click the Save & Run button to save and start the example.

Step 2: Prepare the HTML/JS File as follows:

1. Download mqttcam.html, open the file in any editor, and ensure the topic in the HTML file matches the topic you set in the Lua script.
2. Save the mqttcam.html file.
3. Open mqttcam.html: Double-click the mqttcam.html file or drag and drop it into your browser. Note: this file is designed to be opened directly from the file system. You do not need a web server to host this file.
4. Observe the Output: The webpage will display the images being published by the ESP32 CAM. The number of frames received will be displayed below the image.

Potential Issues with ESP32 CAM Boards and Solutions

ESP32 CAM boards are widely recognized for their versatility and affordability. However, they're not without their challenges. One of the significant issues users might face with the ESP32 CAM boards is interference between the camera read operation and the built-in WiFi module. Let's delve into the specifics:

Problem: Interference and WiFi Degradation

When the ESP32 CAM board is in operation, especially during the camera's read operation, it can generate noise. This noise interferes with the built-in WiFi, which results in:

• Reduced Range: The distance over which the WiFi can effectively transmit and receive data can be notably decreased.

• Decreased Throughput: The speed and efficiency at which data is transmitted over the WiFi network can be considerably hampered.

Solutions

To combat these issues, consider the following solutions:

1. Use a CAM Board with an External Antenna: Several ESP32 CAM boards come equipped with or support the use of an external antenna. By using such a board and connecting an external antenna, you can boost the WiFi signal strength and range, mitigating some of the interference caused by the camera operations.

2. Integrate the W5500 Ethernet Chip: If your application demands consistent and robust data transmission, consider incorporating the W5500 Ethernet chip. By utilizing Ethernet over WiFi, you are effectively sidestepping the interference issues associated with WiFi on the ESP32 CAM board. Xedge32 is equipped with integrated Ethernet drivers. When paired with hardware that supports it, like the W5500 chip, it can facilitate smooth and interference-free data transfer, ensuring that your application remains stable and efficient.

In conclusion, while the ESP32 CAM board is an excellent tool for a myriad of applications, it's crucial to be aware of its limitations and know how to circumvent them to ensure optimal performance.

Is this MQTT cam concept useless, or does it have practical applications? We've sparked a heated discussion. We'd love to hear your thoughts and insights. Join the conversation and give your input!

References

• Lua MQTT API
• Lua timer API
• Lua CAM API
• Your First Xedge32 Project
• Your First MQTT Lua Program

Converting MQTT JPEG Image Stream to Continuous Video

Turning a published stream of images into a regular video is straightforward with the right tools. You can efficiently create continuous video streams by leveraging a small Python script that subscribes to the MQTT image topic, combined with ffmpeg for real-time video encoding. This method has been successfully tested on a Raspberry Pi, but it should run on various host platforms, including Windows and Mac. The instructions here are tailored for Linux.

Before you begin, ensure the necessary packages are installed on your system. You can do this by running:

With the above dependencies in place, you're ready to execute the Python script. Make sure to adjust the image topic name to match the one you are using. Setting up a local MQTT broker is recommended for optimal performance, especially in a network with multiple CAM devices. The script provided below is designed to be easily adaptable for such setups:

The above Python script converts a continuous stream of JPEG images received from the MQTT image topic into video files. Each video file represents one hour of footage. Note that the ongoing video segment is not viewable until the hour elapses and the video file is finalized.

Python Script Details:

• The start_ffmpeg_process function initializes an ffmpeg subprocess that encodes incoming images into a video file.
• The on_message callback writes each received JPEG image to the ffmpeg subprocess.
• After one hour, the script closes the current ffmpeg process, finalizing the video file. It then starts a new ffmpeg process for the next segment.
• This cycle repeats, creating a series of one-hour video files.
• The video frame rate is set at 30 frames per second. Thus, assuming a capture rate of two images per second for an hour of image capturing, the total number of frames would be 3600 seconds x 2 images per second = 7200 images. Since these images are played back at 30 frames per second, the resulting video length will be 7200 images / 30 frames per second = 240 seconds, or 4 minutes long.

Run the script in the background as follows:

Video Surveillance and Face Detection

Computer Vision is a field of artificial intelligence that enables machines to derive meaningful information from digital images. The OpenCV library enables easy image recognition without requiring a detailed understanding of AI and Computer Vision. The library simplifies implementing object detection and recognition, making it an excellent option for applications ranging from security systems to factory supervision.

OpenCV provides pre-trained models for various classification tasks, including face detection. The following script uses haarcascade_frontalface_default.xml, a model trained at detecting frontal faces, which makes it particularly suitable for surveillance applications. In the final script, we will integrate this face detection functionality into our video script, enhancing the video script's ability to recognize and record human presence, thus offering a more complete surveillance solution.

Execute the following command to install OpenCV before running the script.

Face detection script:

The above script is designed to process a continuous stream of images sent over MQTT, detect human faces in each image using OpenCV, and save any images with detected faces. When messages (images) are received on the image topic, the on_message function triggers the detect_human function. In detect_human, each image is first converted from a JPEG format (as received in the MQTT payload) to a NumPy array using OpenCV's imdecode function. This is necessary because OpenCV processes images as NumPy arrays. The image is then converted to grayscale, a common preprocessing step in many computer vision tasks, including face detection. Grayscale simplifies the detection process by reducing the amount of data to be processed (color information is not usually necessary for detecting faces). The haarcascade_frontalface_default.xml classifier, a part of OpenCV's library, is specifically trained to detect human faces. It works by scanning the grayscale image for patterns or features typical of human faces. If any faces are detected, the script saves the original image (in color) with a timestamp in the filename, indicating when the face was detected.

Merging Video and Face Detection for Enhanced Surveillance

Our consolidated script unites video streaming with face detection capabilities, simplifying the operation of a surveillance system. The script saves images that capture human presence, each marked with a timestamp. This design allows users to quickly locate and view specific segments of video corresponding to the timestamped images, providing a quick reference to moments of interest.

Keep Learning:

Congratulations on completing this tutorial! But the learning doesn't stop here. Delve further into the world of embedded systems with our extensive collection of Xedge32 tutorials.

Posted in Xedge32

• 
• 
• 
• 
• 
• 
• 
• 

Engineers: We're Here to Support Your Vision:

Your time is valuable. Let our expert consulting services streamline your networking, security, and device management tasks. And for those moments of DIY inspiration, our tutorials offer unmatched guidance. Partner with Real Time Logic and witness the seamless blend of expertise and empowerment.