This is part 2 of a series of posts about installing a home automation system on a Raspberry Pi. It covers installing the major services that are needed for the home automation system:

• Domoticz a home automation application that is the hub of the system
• Eclipse mosquitto an MQTT broker that is used for communication between the hub and the IoT devices
• HA Bridge a service that enables local voice control of IoT devices with Dot Echo smart speakers
• NGINX a Web server that can be used to upload new firmware to the IoT devices

I recommend looking at the Andreas Spiess video , Pi Server based on Docker, with VPN remote access, Dropbox backup, Influx, Grafana, etc. on YouTube. While the intent is quite similar, the approach is rather different relying on Docker which is yet another level ofabstraction.

Table of Contents

1. Overview of the Home Automation System
2. Installing the Domoticz Home Automation Server
1. Unit File to Start and Stop Domoticz
2. Watchdog
3. Installing the mosquitto MQTT Broker
4. Installing the NGINX Web Server
5. Installing HA Bridge
   1. Installing the Java Runtime Environment
   2. Installing the ha-bridge Script
6. Configuring the Home Automation System
   1. Adding a Virtual Switch in Domoticz
   2. Configuring a Tasmota Device
   3. Setting up HA Bridge
7. More ...

Overview of the Home Automation System

It is probably best to start by providing a description of the home automation system that has been slowly built over the last eight years since moving into our current house. The following diagram is a simplified representation of the system as it is now.

Most Internet of Things (IoT) devices are either sensors or actuators. Sensors sample some aspect of the environment such as the temperature, humidity level, the light level, or the presence of someone, and so on. Actuators are really just on off switches that control lights, TVs, the garage door, or an irrigation system (an upcoming project). To be honest, sensors and actuators are rather stupid devices that need to be controlled by intelligent entities. This means there has to be an exchange of messages which in turn requires a data transport mechanism and a communication protocol. As can be seen data travels through the wireless local area networking (Wi-Fi) which was already in use for other purposes.

Having settled on the type of sensors and actuators and the wireless technology to use, I had to choose the home automation software and the hardware platform on which to run it. I chose to run Domoticz on a Raspberry Pi. While Domoticz might not have the popularity Home Assistant, it is, in my opinion, a very powerful yet lightweight application compatible with Linux, Windows and macOS although the later may require more effort. I have installed it without problems on many single-board computers such as a single core Raspberry Pi 1, an Orange Pi Zero, a La Frite AML-S805X-AC and a four coreRaspberry Pi 3 B running the latest version of Raspberrry Pi OS Lite used here.

It should be safe to claim that my spouse finds that this hobby of mine has improved some things about the house. Before, to turn the outside lights on when guests were leaving the house in the dark, two switches had to be activated, one by the front door and then another in the attached garage. Now when the balcony lights are turned on or off, the outside garage lights which illuminate the driveway are simultaneously turned off or on. It is very practical that the garage door is automatically shut after a few minutes if one of us forgets to do it when driving away. However there was some pushback when it came to some of the other devices. We all agreed that the light switch for the kitchen light was much too far away - one of the consequences of an open floor plan and bad planning. But no one, including me, thought that the solution was to open a tablet or intelligent phone, get to the Domoticz web interface, and finally click on the virtual kitchen light to control the real light. It was faster to walk to the other end of the kitchen to turn the light on or off. Voice control changed all that. Now everyone is at ease with given oral commands to "she who must not be named" to turn lights on or off, or to set their dimmer values. Domoticz scenes to turn the lights on temporarily as we make our way from the TV room in the basement to the bedroom on the top floor are turned on with voice commands every evening. The value of voice control in increasing the value and acceptability of the home automation system cannot be overstated.

There is a self-imposed constraint on our home automation system. It must be an independent local system with minimal reliance on remote servers. Obviously, smart speakers break that rule. They introduce risks which are not negligible. In addition to breaking updates mentioned in the previous sidebar, our experience has been that winter storms that knock out power in a wide area for more than a day or two are often accompanied by longer periods without Internet service. Every two or three years this has happened and it does look like the frequency can only increase. I wanted some mechanism to mitigate the problems. An old X10 wireless remote with 16 buttons is still available in the living room for that purpose. I also added yet another IR remote by the TV much to the chagrin of my spouse. I think I now have a better solution which I labelled Smart Button in the graph. So far the breadboard prototype sitting by my desktop keyboard has been working very well. By the way, this device relies entirely on MQTT to communicate with the home automation system. The hope is that it will grow into something even more practical, when these buttons advantageously replace seven Sonoff Basic providing local control for eight lights as well as remote control for other devices, including scenes and groups, set up in Domoticz.

Installing the Domoticz Home Automation Server

Before anything else, I would like to reiterate my gratitude to the Domoticz development team as well as to the community.

Before installing a major program, it is always a good idea to upgrade the system if it has not been done lately.

Even if Raspberry Pi OS had been installed and upgraded just a couple of weeks previously, this operation took a few minutes and upgraded no less than 20 packages. Of course, just what will be done on another Pi will depend on which packages were installed on the machine and the time elapsed since the last upgrade.

Installing Domoticz is truly simple when following The "easy" way instructions. It is a one line command which will pipe an installation script file to bash which is a command line interpreter.

After answering a couple of questions about ports and directories the process will be completed.

You will have noticed that the installation script required root privileges. As your mom taught you as a youngster, this is dangerous akin to accepting candy from a stranger. You could download the script, examine it and then execute it only after you are satisfied that it does nothing nefarious.

If happy with what has been found, then run the script to proceed with the installation.

The installation script will check for a number of needed packages and install them if absent.

• apt-utils
• whiptail
• git
• curl
• unzip
• wget
• sudo
• cron
• libudev-dev
• libsqlite3-0
• libcurl4
• libusb-0.1

If truly paranoid, then Domoticz should be compiled directly from the thoroughly audited source code. It does seem possible for mere mortals with patience and time on their hands to do that, but I have never undertaken that task and cannot comment on its difficulty.

No matter how it was installed, the service should be running. This is easily checked.

If an error about a missing Python library appears,

then install the python3-dev package. That package would have been installed if a virtual Python environment had been created in the first post of this series. See Virtual Python Environments in Setting up a Raspberry Pi as a Headless Computer in September 2021.

As a final check start a browser and go to the address specified. The home automation system will display the following page.

Of course, there will be no switches, scenes, temperature sensors and so on. When it is time, I will copy the database, various bash and python scripts from the Domoticz server that is currently running my home automation system and then restore the database to this new server. I have done this a couple of times in the past and it worked flawlessly.

Unit File to Start and Stop Domoticz

As described back in late 2019 in Home Automation Servers on Raspbian Buster Lite, I chose to use a systemd unit file to start and stop the domoticz.service instead of the domoticz.sh startup script installed in etc/init.d because there was a problem if a real time clock (RTC) had not been added to the Raspberry Pi (or in my case, when the RTC battery died). This problem no longer occurs in version 2021.1 (build 8547c5b7e dated 2021-04-17 12:29:11) of Domoticz. Unless you are interested in avoiding older init scripts for controlling services this subsection can be skipped as there is no longer a pressing reason to use a unit file. Nevertheless, the Linux page on the Domoticz Wiki does recommend using a unit file when the service is run on Linux distributions with systemd as the init program.

The first step is to create the unit file with nano or an editor of choice.

Here is an example unit file. The condition After=time-sync.target will ensure that the service is installed only once the system time has been synchronized. The parameter AmbientCapabilities=CAP_NET_BIND_SERVICE is needed if a TCP port less than 1024 is specified which is the case below where port 443 will be used for secured HTTP connections. See the Wiki page for more details, especially if an older version of Raspbian is being used.

Of course, the user name nostra, which appears three times, needs to be adjusted.

Now stop Domoticz if it is running and remove the old domoticz.sh script (saving it in case something goes wrong). Reload all service (daemon) unit files, start the Domoticz service and check its status to ensure that everything is correct.

The following command will instruct systemd to automatically start the service at boot time.

This command, which creates a symbolic link to the newly created Domoticz unit file, needs to be launched only once.

When booting up after power has been off for more than 10 minutes or so, do not worry if initially the domoticz service does not appear to be functioning. It will continue to repeat its start-up routine until the time is correctly synchronized. This works no matter if the injunction to wait until time synchronization with a SMTP server is active is in place or not. In other words, the unit file line shown in red, which was an important addition back in 2019, is no longer necessary.

Watchdog

Since home automation is the main task to be performed by the system, it made sense to add a watchdog that would restart the system should the home automation software stop functioning correctly. I have already discussed this in the post entitled Raspberry Pi and Domoticz Watchdog where more details can be found.

The first step is to create a Lua script that Domoticz will execute every minute. All it does is update the time stamp of a file named /tmp/domoticz.alive every time it is run.

Check the file time on a regular basis to ensure that it is updated every minute.

Next install the watchdog package.

The watchdog configuration file has to be modified. As usual, I used nano to do this.

As far as I can ascertain, the timeout value has to be 15 seconds, the default 60 seconds does not work.

Start the watchdog service and then wait over five minutes (300 seconds) to ensure that the system is not rebooted. Then stop the Domoticz service and the Raspberry Pi should be rebooted in about five minutes.

As this example shows, it will be necessary to stop the watchdog if Domoticz is suspended for any length of time otherwise the Raspberry Pi will reboot.

Installing the mosquitto MQTT Broker

An MQTT broker is a necessary part of my home automation system. It acts as the relay between the domoticz service and the IoT devices. Most of the latter are ESP828x devices running on firmware by Theo Arends: Tasmota. The mosquitto broker is available in the Rasperry Pi OS repository as can be seen here.

The latest version of the Eclipse Mosquitto package is 2.0.12, but version 1.5.7 is recent enough and it is much easier to install mosquitto from the repository than to try to install from the source or from an alternate repository. Indeed, [version 2.0].. is a big change with breaking behaviour changes in the broker so if it is installed be prepared to fiddle with the configuration to make it work with Domoticz (see the Mosquitto MQTT broker v2.0.0 topic in the Domoticz forum). A simple installation of the older broker and the optional utilities (to get mosquitto_sub and mosquitto_pub) can be done without a problem.

After I checked and found that the broker was running automatically.

To make sure that everything was installed properly, I subscribed to all topics in a terminal on the Raspberry Pi.

Then I sent a message to the broker from a terminal on my desktop computer.

If mosquitto is not installed on the desktop, the message could be published from a second terminal on the Raspberry Pi.

In either case, the message should show up in the first Raspberry Pi terminal.

Instead of monitoring exchanges with the mosquitto-clients, a chat type client which can publish as well as subscribe to multiple messages could be useful. While about to shamelessly plug my own lazmqttc which runs on Linux and Windows 10 (and probably macOS though I can't be sure), I ran into MQTT X which looks to be good. I will have to try it.

Installing the NGINX Web Server

While there is a stand-alone web server on the Raspberry Pi it is not a very busy because the two most used services, Domoticz and HA Bridge, run their own web server. Essentially, all the web server does is upload firmware to IoT devices that are updated directly from their own web interface, something that is rarely done. This is a Tasmota feature althouh I prefer uploading new firmware to Tasmota devices directly from my desktop. So the uploading firmware with the home automation system is mostly used for IoT devices running firmware that I have written and there is only three of those.

While Apache may be the "most popular" web server, I have used the LIGHTTPD and NGINX Open Source servers on small single-board computers because they are reputed to be "lighter" and more than adequate given the minimal requirements. In this section, I will show how to set up nginx, but there's no doubt something similar could be done with LIGHTTPD and Apache.

Installing the lightest version of NGINX in the Raspberry Pi OS repository is very simple. But before doing that, I recommend to once again upgrade the system if it has not been done lately.

Because the default HTTP port of the builtin web server started by HA Bridge is 80 and nginx uses the same default port, it is best to intall the web server before installing HA bridge. If HA bridge is already installed, just suspend it until the default HTTP port used by nginx is changed.

Opening the page in a web browser verified that the server is working.

Of course it may be necessary to use the IP address of the Raspberry Pi in Windows or in some other operating system.

No matter which Web server is installed, it will be quite useful to change the owner of the Web directory.

This way, the default user (nostra in this case) to will be able to add, delete and edit any file in the directory or any subdirectories that will be created.

Some may wonder why nostra is not a member of the www-data group. Quoting jojopi, [t]here appears to be a common misconception that everything to do with the web should be owned by www-data. Actually it is quite the opposite. Read the complete answer in the Raspberry Forum on the question of Re: /var/www/html permissions which I found quite cogent.

I then went on to create a directory to contain the Tasmota firmware to download to various ESP8266 IoT devices if needed.

While not needed at all, I decided to create a custom 404 error page. This page will be shown when a URL links to a non-existing file.

Then I created a similar 403 error page. This is what will be shown when a URL points to a subdirectory in /var/www/html that does not contain an index.html file or the equivalent, otherwise the content of the directory could be listed as a web page. Again this is not needed at all in this context, but I would suggest to do this with any outward facing web site.

Two entries must be added to the web server configuration file to display these pages when required. Here are the details for nginx.

Add the entries near bottom of the HTTP block.

While HA Bridge (the next server to be installed) runs very well once set up, it is a bit difficult to get it working at the start. For that reason, I much prefer not to change the default TCP port of its web interface. Consequently NGINX must listen to a TCP port other than the default 80. As usual this involves a small change to a configuration file.

Change port 80 at the start of the file

to something else. I chose 8888 (remember that Domoticz uses 8080 by default).

Restart the server for this to take effect.

Installing HA Bridge

As discussed above, HA Bridge is a shim that presents many of the virtual devices, scenes and groups in Domoticz as hardware devices that Amazon Dot smart speakers can control locally. Previously, the bridge between Alexa and Domoticz was handled by another single-board computer, but the Raspberry Pi 3 is quite capable of handling HA Bridge as well as all the other services mentioned in this series of posts.

Since HA Bridge is a Java script, the first step is to install the Java run-time environment.

Installing the Java Runtime Environment

The Raspberry Pi OS repository contains many Java packages. First there are two implementations: OpenJDK and Oracle Java (remember that Oracle purchased Java from Sun). Unless there is another pressing need, I would stay away from the Oracle implementation because of its more restrictive license. Even when looking only at the "free" OpenJDK implementation, there are no less than 8 available packages.

The jdk suffix identifies the Java Development Kit while jre is the Java Runtime Environment. The JDK contains the JRE and other components needed to create Java applications (more on the difference here). I decided to install the lightest runtime version, openjdk-8-jre. Presumably, if HA Bridge works in that environment then it would work in any of the other ones. As usual, installing a package from the repository is straightforward.

Installing the ha-bridge Script

Since Ha Bridge is a Java script, installation is not too complicated:

1. Create a directory to contain the script.
2. Copy the script to the directory.
3. Setup a symbolic link to the script.

Download the latest version of the script. It should be available in the releases page of the project github account. As can be seen, I am using a release candidate for the upcoming version 5.4.1. It added the capability to assign 9 octets ID numbers to the virtual Hue devices shown to Alexa. With the latest update to the Dot firmware, this was a necessary addition otherwise the devices are not discovered by Alexa.

The symbolic link is not absolutely required, but it will make it easier to update the script without having to change the unit file used to integrate the script into the systemd startup sequence. Create the unit file with an editor. As usual I use nano.

Do not forget to modify the user name, which in my case was nostra. It appears three times in the unit file.

All that remains now is to start the service.

It should now be possible to log into the web interface.

And, of course, it is empty.

Configuring the Home Automation System

All the essential services for a home automation system similar to ours are in place, but so far the system is an empty shell. Our HA system has been built over the course of many months and there is no practical way to describe it all in detail here. Instead, I will describe the installation of a virtual switch in Domoticz, which is the first step in adding a Tasmota IoT device into the HA system. In the following subsection, the virtual Domoticz sensor is added to the list of emulated devices in HA Bridge which is the first step in adding a Smart Home connected device in an Alexa account.

Adding a Virtual Switch in Domoticz

I have broken down the process of installing the first virtual switch in Domoticz, into 18 steps. Do not despair, the first 16 steps are required for setting the server location's latitude and longitude and for the one-time installation of the MQTT and Dummy "hardware". The geographical coordinates are used by Domoticz to calculate sunrise and sunset times. MQTT is needed to communicate with Tasmota enabled IoT devices and the Dummy hardware is a generic holder for virtual sensors, which can be many types of things including switches.

Most administrative tasks in Domoticz are done in its web interface. As already seen this is reached at http://damus.local:8080 or http://192.168.1.139:8080 (or whatever static IP address was assigned to the Raspberry Pi).

• Set the geographical coordinates of the HA server
  1. Click on Setup.
  2. Click on Settings.
  3. Click on the System tab (actually it will probably already be the open tab sheet).
  4. Enter the server's Latitude and Longitude in Location. Remember North latitude are positive, South are negative and East longitude are positive and West longitude are negative.

  

  5. Click on Apply Settings.
• Enable MQTT
  1. Click on Setup.
  2. Click on Hardware.
  3. In Name enter MQTT or whatever seems appropriate.
  4. In Type select MQTT Client Gateway with LAN interface for the drop-down list.
  5. In Remote Address enter 127.0.0.1 since the MQTT broker is running on the same machine.
  6. In Port enter 1883 unless the default port used by the broker was changed.
  7. Ensure that Publish Topic is set to Flat

  

  8. Scroll down to the bottom and click on the Add button. The MQTT client gateway should now appear in the top table.
• Add the Dummy hardware to contain all virtual sensors
  1. In Type select Dummy (Does nothing, use for virtual switches only).
  2. In Name enter Virtual or whatever seems appropriate
  3. Click on the Add button at the bottom.

     

     The Dummy hardware should be added to the top table with the MQTT gateway.
• Add a virtual switch
  1. Click on Create Virtual Sensors in the Dummy entry of the the hardware table.
  2. In the Create Virtual Sensor pop up window

     

     • Enter a name such as First Switch or something else appropriate in the Name field.
     • Select Switch as the Sensor Type from the drop-down list.
     • Click on the OK button

That is it. Now the Domoticz database contains one virtual device named First Switch which can be seen in the Switches tab.

Remember only these last two steps have to be repeated for each additional switch or virtual sensor that needs to be added. Again, this is done in the Setup / Hardware configuration page.

Configuring a Tasmota Device

Just to verify that everything works, the newly created virtual Domoticz switch is linked to a CE LA-WF3 Wi-Fi wall plug on which Tasmota 9.1.0 is installed. It was last used to control some lamps in the TV room so that it was possible to open its web interface. Settings need to be changed in order to bind the plug to the virtual device in the newly created Domoticz instance. These settings are the MQTT broker address, the Domoticz device number called its Idx and, optionally, the Syslog server address, log level and the device name. Since only one device has been added to the new HA database, it is not entirely surprising that the device number is 1. No matter, one should check just to be sure. The Idx can be found in the Devices table found in the Domoticz web interface: Setup / Devices.

The settings can be changed one at a time in various menus of the Configuration screen of the wall plug web interface. That gets tedious very quickly and is prone to errors; usually forgetting to set something. I prefer to use the Console which makes it possible to enter a single, multi-command string, exploiting the Backlog function.

When entering the Backlog command, do not forget the trailing semicolon. It must be there otherwise the last part, which in this case specifies the device index number in Domoticz, will not be executed.

To test this configuration, it is probably easier to plug a light into the wall plug at this point. Toggle the state of the virtual switch by clicking on the light bulb icon in the web interface.

Make sure that the physical wall plug (i.e. the light) follows suite. Then try in the other direction. Turn the light on and off using the wall plug switch (not the light switch if there is one) and check that the virtual device status is updated. Be aware that the update may not be instantaneous. The impatient can refresh the web page; F5 works in many web browsers.

Setting up HA Bridge

The first part of the following procedure needs to be done once only. It sets the host name or IP address of the server on which Domoticz runs. There is no requirement for Ha Bridge to be run on the same computer as the rest of the home automation system. Indeed, I have already described its installation on a separated Orange Pi Zero single board computer in HA Bridge on Armbian Working with Domoticz and Alexa.

The second part of the procedure adds all the virtual devices in Domoticz that can be presented as Hue bulbs to the Ha Bridge database of emulated devices. In practice, not all virtual Domoticz devices will be added, only those where it makes sense to use voice commands.

• Connecting to Domoticz
  1. Click on Bridge Control

     

  2. Scroll down to Domoticz Names and IP Addresses
  3. Enter the desired name for the Domoticz server and enter the IP address of the server. Since it is on the same Raspberry Pi 127.0.0.1 can be used but the statici IP address of the Pi could be entered.
  4. Click on the Add button.
  5. Scroll back up to top and click on Save and wait for the bridge to reinitialize. A message is displayed about that operation and once the bridge is intialized, the Bridge Devices tab will be displayed. There will be a new choice, Domoticz Devices, between Logs and Add/Edit.

     

     It might be necessary to refresh the web page (F5) to see that new choice.
• Adding a Domoticz Virtual Switch
1. Click on Domoticz Devices tab. The single virtual device in the Domoticz database should be visible.

   

The Bulk Add button at the bottom of the list can be used to add all selected devices at once in a surprisingly easy and efficient manner. Since there is only a single device in the current list, the point is moot.
2. Click on the Build Item button.

   In the Add/Edit tab that will be shown, you can scroll down to see the HTTP requests that the bridge will send to Domoticz to turn the light on and off when it receives a command from the Echo Dot telling the virtual Philip Bridge emulated by Ha Bridge to turn the corresponding Philip Hue bulb on or off.

   

3. Scroll back to the top of the window, and click on Update Bridge Device

   

• Testing

  It is possible to test that the HTTP requests created by the bridge work in the Bridge Devices tab sheet.

  

  Click on the Test ON and Test OFF while watching the First Switch virtual device in the Domoticz web interface. Of course, the light connected to the wall plug should also turn on and off in response to the button presses.

• Discovery by the Dot Echo

  In theory, this is not a complicated step. You have to check in Bridge Control that the UPNP IP Address is the IP address of the computer on which HA Bridge is running. This is usually set up correctly by the script itself. Then, further down, ensure that Unique ID to use 9 Octets is set to True (i.e. checked). This is not the default value. This new field was added in this very latest version of the script and it seems essential with newer versions of Alexa.

  The second step is to enable uPnP on the local network. On most residential routers supplied by an ISP it will be enabled by default although that is considered to be less than secure. The reason is that uPnP could be exploited to punch holes in the firewall without the user being aware. Does Skype still do that? It did in the past. Because of that, the capability is disabled in my router, which means that I have to remember to enable it again whenever I want to add a device in the list of Smart Home device in my Alexa account so that voice commands can be used to control them. And I have to remember to disable the uPnP when the addition of devices is completed.

  The final step is to have a Dot Echo find all devices in the house. I have found that it is best to disconnect all Dot devices except for the one that will perform a search for devices. Not all Echo devices work well, I use a 3rd generation Dot Echo. I usually issue a voice command to the speaker telling it to "search for devices". If the list of Smart Home devices is empty, this will usually work quite well.

  I have never been able to add a new device to a list of Smart Home devices that already contains some devices. Or if I have managed it, the list would then be poluted with many duplicate devices which clutters up the Alexa interface. The only way I have found to add a new device is to have Alexa discover all devices again. Here is, approximately, how I do it.

  • Clear the list of Smart home devices in Alexa. This is much more efficiently done in the Web interface while with smart phone applications it is a tedious task because devices have to be eliminated one by one.
  • Renumber the list of devices
    • in Bridge Control, set ID Seed (start numbering from this value)) to a new value
    • in Bridge Devices, press Renumber Devices
  • Ask the Echo Dot to find devices.

  I find that I have to wait a good while between each step and it seems like a good idea to power down the Dot Echo until it is time to proceed with the discovery. I also wait after powering up the Echo to ensure that it is completly reset and has lost all memory of the virtual Philips devices. It is almost as if I am anthropomorphizing the technology. It helps to be patient and obstinate because it usually takes a number of tries over a number of hours before success is attained. More details can be found in HA Bridge and Alexa Device Discovery but unfortunately the instructions are just as imprecise.

More ...

The reader will have noticed that there’s no real discussion of security in this configuration of the home automation system. The justification for this is that it is meant to be on a local area network that will never be exposed to the Internet. It could be on its own virtual LAN for example. Obviously, strict enforcement of this embargo would defeat part of the purpose of a HA system. Many consider remote access to devices about the house to be very useful. In the next instalment of this guide, the installation of a virtual private network server will be covered. That offers a way of securely accessing the HA system which is not itself secure.