The most dependable element in our home automation system is an outside temperature sensor consisting of a DS18B20 digital thermometer placed outside connected to an ESP8266 installed inside the garage. It systematically reports the current temperature using MQTT messages that make their way to the Domoticz server and get stored in the latter's database. The minimal firmware running on the ESP recovers from the occasional power outages and from the rather more frequent interruption of the Wi-Fi network thanks to a new router from a new ISP provider. It does everything expected without providing a Web interface. Indeed it can be argued that there is no real need for a Web interface for most IoT device.

However, the XIAO ESP32C3 based Wi-Fi switch does have a Web interface. What's more, it uses a not very elegant mechanism to ensure that the data displayed by Web browsers connected to it remain current. Part 2 of this guide examines three different methods to ensure that the Web clients display current data with minimum latency and without resorting to a full reload of the Web page.

The source code for the PlatformIO projects / Arduino sketches presented in this post can be found in a GitHub Repository: sigmdel/xiao_esp32c3_wifi_switch. Three projects / sketches, named 04_ajax_update, 05_websocket_update and 06_sse_update, correspond to the three methods of updating data displayed on the Web page served by our Wi-Fi switch. In the end of course, a choice will have be made as to which mechanism to use.

Table of Content

1. Asynchronous JavaScript And XML (AJAX)
2. Web Sockets
3. Server-Side Events
4. Final Remarks

Asynchronous JavaScript And XML (AJAX)

Let's begin with the obligatory comment about the fact that the "XML" bit in AJAX is a bit of a red herring. To quote W3Schools.

AJAX just uses a combination of:

• A browser built-in XMLHttpRequest object (to request data from a web server)
• JavaScript and HTML DOM (to display or use the data)

[...] AJAX applications might use XML to transport data, but it is equally common to transport data as plain text or JSON text.

AJAX allows web pages to be updated asynchronously by exchanging data with a web server behind the scenes. This means that it is possible to update parts of a web page, without reloading the whole page.

That last paragraph shows that AJAX should be able to do exactly what we need. Furthermore, the confirmation that it will not be necessary to use XML is worth its weight in gold; anything to avoid dealing with XML is fine by me.

The HTML code of the Web page, let's call it index.html, sent to clients has to be modified in order to implement this technology. Remember that the HTML source is saved in the html_index[] string defined in the html.h file. The "magic" meta tag <meta http-equiv="refresh" content="5;url=/"> has to be removed because the Web client will request updates of the temperature, humidity, brightness and LED data explicitly. This is done with a Javascript function to be added in index.html that sends an HTTP request back to the Web server. The latter will respond with a short file containing the required data in four lines of text. The script must parse these lines and update the Web page. But how does it do that? Let's look at a simplified version of the script.

Sending a request to the Web server is a three-step operation: a) create an HTTP request object, b) define the HTTP method of the request and the URI of the request in the object's open method and then c) send the request back to the Web server. Obviously, we need to add a request handler in the Web server. Here it is.

The handler responds to the request from the Web client with a plain text file which contains the four strings of data that are displayed in the index.html file currently shown by the Web client. That response by the Web server is sent back to the Web client via the XMLHttpRequest which provides the Javascript the information it needs to update the Web page. Of course that begs the question of how the Web page is updated. Remember the reference to the HTML DOM above? It is the Document Object Model of the Web page, an in-memory tree whose nodes represent all objects found in the page along with their content and attributes. All Web browsers parse the HTML code received from a Web server to create that tree and use it to calculate the layout of the page and then display it on the screen. The DOM can be seen by clicking the F12 key in the browser. Click on the vignette on the right to see a bigger screen capture of the DOM for the index.html file. The tree is mostly collapsed, but the highlighted element is where the status of the LED is displayed. Note that it is a <span> tag with the led identity. The latter is how the element is found to update its content.

Placeholders for the substitution mechanism of AsyncWebServer remain in place so that the server will continue to send the index.html file with the current data. In addition, each data element is now an identified span tag so that it can be updated when processing the response from the XMLHttpRequest. Note that the third parameter of the open method is optional and its default value is true so it was not necessary to specify it. In the Mozilla documentation, it is identified as the optional async parameter:

An optional Boolean parameter, defaulting to true, indicating whether or not to perform the operation asynchronously. If this value is false, the send() method does not return until the response is received. If true, notification of a completed transaction is provided using event listeners. This must be true if the multipart attribute is true, or an exception will be thrown.

So an event handler, called onreadstatechange, must be added to the request before it is sent to the Web server.

As can be seen, it breaks up the four-line text sent by the Web server in response to the /state request into four strings. The handler then sets the innerHTML property of the corresponding DOM elements identified by their id equal to the matching string. The sensor data is thus updated without reloading the entire Web page. There remains the question of when this is done; what triggers the script in other words? A timer is used so that, much like before, the client browser will poll the server at regular intervals to get an update. Since there is no jarring visual impact when the data is updated, the frequency can be increased significantly. The timer is created with the setInterval() function of HTML DOM API.

Since Javascript is now used, one might as well use a XMLHttpRequest to inform the XIAO Web server that the Toggle button has been activated, instead of using a form method. The Web server response to the request can be the same as described above and in that way, the state of the LED can be updated very quickly instead of having to wait until the setInterval() next triggers. This gives a positive feedback to the user of the Web interface almost equivalent to the that obtained when pressing the physical button. Of course, the LED status of other Web clients will only be updated when they poll the Web server.

This long-winded explanation has been broken up into little bits and pieces. Let's show the modifications to the project in a more explicit fashion. First the changes to the HTTP code. To be clear, these are the principal changes to 02_basic_wifi_switch found in 04_ajax_update.

Do not forget that the refresh tag <meta http-equiv="refresh" content="5;url=/"> found in the HTTP code in 02_basic_wifi_switch/basic_wifi_switch/html.h has to be removed; that was the main point of this exercise. Next are the changes to the Web server code in main.cpp.

This technique avoids reloading the Web page on each update of the sensor data to be shown but each connected Web browser must still poll the XIAO Web server at regular intervals. Accordingly there is a latency which may be objectionable to some, but I would think quite acceptable to most, especially as one is free to set the time between XMLHttpRequests. As far as I can make out, AJAX is the technique used in Tasmota to update the Web interface at least back in version 9.1.

References

While I had heard about AJAX, I basically knew nothing about the technology. The following three references provided a gentle introduction with practical examples which were easily adaptable to the present context.

• AJAX Introduction, W3 Schools.
• ESP32/ESP8266: Control Outputs with Web Server and a Physical Button Simultaneously, Random Nerd Tutorial (Rui Santos).
• AJAX with ESP8266: Dynamic Web Page Update Without Reloading Debasis Parida.

When trying to figure out what the functions did, then the multilingual Mozilla XMLHttpRequest documentation proved very good.

Web Sockets

Again, I will differ to a higher authority for a definition of the concept of web sockets.

The WebSocket API is an advanced technology that makes it possible to open a two-way interactive communication session between the user's browser and a server. With this API, you can send messages to a server and receive event-driven responses without having to poll the server for a reply.

This holds out the possibility of having entirely event-driven updating of connected clients and the XIAO; no more timer-driven polling of the Web server with inherent latency. Actually, that is not just a possibility, it is exactly what will be achieved. Changes to the HTML code of the index.html file sent to clients is almost the same as done with XMLHttpRequest.

Each Web page element to be updated in situ is given a unique identifier as before. The Toggle button is also given a singularly unimaginative identifier: button and, contrary to the previous versions of the HTML code, it is not part of a form nor is the onClick event used. We will take care of the button with an event listener. That is done in the script that follows.

The gateway is the address of the Web socket server which will be added in the Web server running on the XIAO. Note how we are using the window location property to get the address of the XIAO. If the XIAO Web server were not using the default 80 TCP port, it would be better to use location.host since that property appends the port number. By the way, the gatetway is defined as a template literal which is needed for the string interpolation of ${window.location.hostname} within the backticks.

A listener for the window load event, called onLoad is then added. The onLoad handler will initialize the Web socket, websocket and the Toggle button listener when index.html is loaded into the web browser.

Initializing the web socket consists of creating the socket and connecting it to the socket server on the XIAO, and setting up three event handlers. Two are pretty much boilerplate: onOpen, onClose. The third event handler onMessage does the actual parsing of the data from the XIAO and updates the Web page just like the XMLHttpRequest response handler did in the previous section. The difference here is that the data will be sent by the XIAO while the data was requested from the XIAO by the Web client in the AJAX version.

The initButton() function creates a listener for the button click event so that it calls the toggle() function when Toggle is clicked. The toggle() function sends a single word message, toggle, back to the web socket server on the XIAO.

Changes to the XIAO firmware is rather more substantial than in the previous section. A web socket server must be added and it will now be the responsibility of the hardware to update the web clients in addition to Domoticz when the state of the button changes or new sensor data is available.

An asynchronous Web server is created as before, but in addition, an asynchronous Web socket is also created. AsyncWebSocket is part of the ESPAsyncWebServer library. Note that the address of the Web socket corresponds to the gateway address used in the script that will run on Web clients. The Web socket event handler contains boilerplate routines to handle Web socket connection and disconnection. The more important event in our case is when data is received from the Web client. That event handler is broken out as a separate function handleWebSocketMessage which verifies if data received is the word "toggle". In that case it calls the toggleLed() function to take care of the hardware.

There is also an updateAllClients() function. It is used in hardware.cpp to send the four-line text with the status of the button and the current values of the sensors to any connected Web client using the asynchronous Web socket. That means that we need to look at hardware.cpp which is also modified. There is not much to that. The line udpateAllClients() is added at the end of the setLed(), ReadTemp() and ReadLight() functions. In all three cases, this is done after the updateDomoticz... function, here is the shortest example.

In principle, the Web page data is updated every time the sensor data is read. Actually, all the data is sent via the Web socket whenever one datum is read. Perhaps a more sophisticated approach should be implemented to avoid sending all this redundant information. In other words, one could set up "temp", "light" and "led" messages in addition to the "toggle" message and create handlers for all of these. I find the updating of all the data at once to be quick enough to be imperceptible and I am quite happy to lazily take advantage of that.

References

Again, I found a couple of introductions to this new-to-me technology that contained practical examples easily adaptable to the present context.

• ESP32 WebSocket Server: Control Outputs (Arduino IDE), Random Nerd Tutorial (Rui Santos).
• ESP32 Async HTTP web server: websockets introduction, techtutorialsx.

And again, the Mozilla Developper Network (mdn) has detailed multilingual documentation about WebSocket and The WebSocket API.

Server-Side Events

According to the mdn Server-sent events documentation.

Traditionally, a web page has to send a request to the server to receive new data; that is, the page requests data from the server. With server-sent events, it's possible for a server to send new data to a web page at any time, by pushing messages to the web page.

To implement this push technology, an event source is needed. As with Web sockets, the ESPAsyncWebServer provides such a source object, AsyncEventSource and a handler in the Web server. Basically, this requires two lines of code in main.cpp.

Since this is a push technology, most of the work to be done from the server side is performed in hardware.cpp as events are generated. The exerpt below shows that nothing very complicated is done. Basically, whenever sensor data is read or when the button is released, an event is sent. So-called custom events are used here, meaning that the message sent is accompanied by a label, such as lightvalue.

The granularity of the events is an improvement. Messages sent to the Web clients will contain only the changed data, not the complete set as in the previous methods. To be fair, it was not necessary to do that previously; it was convenient as it minimized the coding to be done. Presumably, one could further diminish the amount of data transmitted by generating events only when the sensor value changes. In this case the data transmitted is not at all voluminous and I think it is worth sending the temperature at regular intervals. In Domoticz, the time of the last reading is displayed in the virtual sensor and that is a good way of verifying that the data is current and that the connection to the XIAO is working. Something similar with logging to a Web console using server-sent events will be added later.

Now we have to look at the implementation in the Web browser. Just as before the elements of the Web page that will be modified need to be identified.

And just as for the other two techniques, a script must be added to deal with events sent by the source.

I must admit that the initial line in the script caused me much grief. It's clearly a test that ensures that SSE is supported before creating an EventSource instance. First, the Double NOT (!!) construct was new to me. Why bother? After all predicate logic dictates that a and !!a have exactly the same logical value. The mdn explanation didn't help me too much, but a 14-year-old question on the not not operator in JavaScript (sounds like the start of a bad joke) did shed some light. I'll just quote the short answer: It converts Object to boolean. Basically, it is a typecast equivalent to Boolean(object). So !!window.EventSource returns true if the interface exists. But why test window.EventSource while the instance will be created with EventSource only. It turns out that the window object is global and its properties can be accessed without the window prefix (see window on mdn - I have so much to learn!). In other words the test could have been if (!!EventSource) or if (Boolean(EventSource)).

Interestingly, the search for the meaning of !!window.EventSource turned up two references (HTML SSE API by W3schools, and HTML5 Server-Sent Events and Examples by Eric Bruno) to a different way of testing for the presence of the interface.

Perhaps this would have been easier to understand, although hindsight is at play here. It must be pointed out that many examples found on the Web do not perform the test and assume that the EventSource interface exists. That's probably because SSE is broadly supported as attested by Server-sent events browser compatibility on mdn and Server-sent events on Can I use. That's enough on that opening statement, let's get on with the rest of the script.

When the EventSource instance, source, is created, it opens a persistant HTTP connection back to the Web server at the specified URL. The rest of the script consists of adding listeners with the addEventListener() method. These specify the function that will handle the event it is delivered based on its custom label. As with Web sockets, there are two boilerplate listeners that log the opening of the event source and the disconnection from it without doing anything else. The remaining four additions are for the custom events that the AsyncEventSource events can send. As can be seen, each of these functions updates the DOM element using the data sent by the events.send() method.

References:

As before, here a couple of practical introductions to Server-sent events.

• ESP32 Web Server using Server-Sent Events (Update Sensor Readings Automatically), Random Nerd Tutorial (Rui Santos).
• How to update Web page using ESP32 and Server-Sent Event (SSE), SwA (Francesco Azzola).

Final Remarks

There are obvious commonalities between these three techniques. They all required approximately the same changes to index.html. The various elements of the Web page that need to be updated have to be identified; how else could they be updated individually instead as part of the page as the latter is transmitted to the client as a whole? Elements of the DOM can be denoted by a name or an identifier, I chose to use identifiers.

The implementation of each technique in the client Web browser is done with a script. This is a problem if Javascript is disabled in the browser for security reasons. There is no obvious work around except to revert to automatic page reloads at fixed intervals. At the very least, there should be a warning displayed on the Web page when Javascript is not enabled. Theo Arend's firmware displays To use Tasmota, please enable JavaScript at the top of every page.

Of course, there are differences between the three technologies. AJAX pulls data from the server to keep the Web page up to date. The technique requests the data from the Web server at regular intervals and in that respect it is not much different from the page reload approach initially employed. However because only some elements of the Web page are changed, the flickering experienced when the complete page is redrawn is absent. Consequently, it is possible to poll the Web server at a higher frequency. Server-sent events do the exact opposite. SSE is a push methodology that minimizes updating of the Web page elements while doing that whenever the data changes. The reduction in latency is quite advantageous. Web sockets are the most advanced protocol since it is based on bidirectional communication. Consequently anything that can be done by AJAX or SSE can be accomplished with a Web socket.

One would think that Web socket should be the best choice, but I opted for SSE. One important reason is that it is sufficient for what is needed and it has some advantages over Web sockets. The most obvious is that it will reconnect automatically if the connection is lost for some reason. This does not occur when a Web socket gets disconnected. The onWsEvent handler in section 2, logged connections and disconnections but nothing else. Presumably, one could add code to reconnect in the latter case, but that would mean more work.

In the 06_sse_update project, activation of the Toggle button was done with an HTTP GET <XIAO_IP_ADDRESS>/toggle. This is not the best idea, because the Web server must send a reply and there's basically no choice but to send back the whole index.html page.

I wanted to experiment with the POST method, hoping to find a way around this unnecessary upload. Unfortunately, the ESP32-C3 crashed every time a POST request was received. I have yet to pinpoint the source of the problem. It could be that the problem has been solved with a newer version of ESPAsyncWebServer. It could be that it is a second ESP2-C3 bug in that library that is not well known. It is more likely that I am doing something wrong, although, in another program, I did test ESPAsyncWebServer with POST requests without problems when using a standard ESP32. The matter is important enough that I will investigate it further some time in the future.

In 08_ticker_hdw I returned to this problem and decided to use an XMLHTTPRequest to send an HTTP GET request at /toggle. That way,it did not matter how the Web server responded to the request. The Web page of all connected clients would still be updated to reflect the new state of the physical LED when the associated server-sent event was fired. This has been the solution since then (working on 10_with_config at the time of writing this).

It is a bit ironic, that I have settled on a combination of SSE to push sensor data changes and manual activation of the physical button to clients Web browsers and AJAX to respond to user actions in the Web interface. After all, it would seem to make more sense to use a single technology, the bidirectionnal Web socket. Perhaps it is a reflection of my lack of experience, but I feel that AJAX + SSE is simpler than Web socket. In comparisons between the three techniques, some favour Web socket to SSE + AJAX saying using XMLHTTPRequests is out of band. I'll concede the point if implementing a chat program in a Web browser for example. However, a click on a virtual button on a Web interface is "out of band" by its very nature since it is not done in response to anything that the Web server could have sent.

Learning about these three modes up updating the content of the web page has been instructional as I had hoped. In the upcoming iterations of the program, I want to improve the firmware from the point of view of the user, with better logging, avoiding blocking operations as much as possible and offering a sufficient level of user-defined settings that can be changed at run-time.