May 30, 2017
Workflow when Working with a Headless Raspberry Pi Flashing a Sonoff in the Arduino IDE

In the first post of this series, I showed how to install Domoticz, a home automation server on a Raspberry Pi. In this step, I will install an MQTT broker on a Raspberry Pi 3. This is a communication protocol which will be used as glue between Domoticz and ESP8266 based hardware that I am using to control devices.

For reference, I am following my own previous posts Add an MQTT Broker to the Raspberry Pi and Add MQTT to Domoticz for the most part. Nevertheless, I hope to improve on the latter. While this installation is done on a Raspberry Pi 3, it should be possible to follow the instructions and install the broker on any model of the Raspberry Pi.

Table of Contents

  1. What is MQTT?
  2. Installing Mosquitto MQTT
  3. Testing Mosquitto MQTT
  4. Adding a Virtual Switch in Domoticz
  5. Enabling MQTT in Domoticz
  6. Testing MQTT in Domoticz

  1. What is MQTT?
  2. The Message Queue Telemetry Transport, MQTT for short, is a messaging protocol which has gained momentum for establishing communication between devices in the fast-growing "Internet of Things" sector. Interestingly, the protocol was developed back in 1999, just as the IoT was attracting some interest.

    MQTT uses a publish-subscribe paradigm. We all understand point-to-point connections between peers. It is the basis of telephone conversations between two people. The client-server connection is another well-known point-to-point link. To display a web page, a web browser, the client, requests the page from an HTML server that may send the page or may refuse to do so. In these types of links, both parties must intentionally address each other. With MQTT, devices can publish data or subscribe to receive data from a "broker". Some devices do both. When a device publishes data, it does not need to know which devices want to read it. A device that has subscribed, receives data when it becomes available without having to request it from a particular device. Instead the broker acts as a clearing house; it receives all published data organized by "topics", and sends out that data to all subscribers of specific "topics".

    Using a broker means that publishing data from one to many is possible at the same time as reading data from many to one. Here is a typical arrangement:

    Icons made by EpicCoders from www.flaticon.com is licensed by CC 3.0 BY

    The air conditioner is registered with the MQTT broker to receive all data on the topics "temperature" and "humidity". The temperature and humidity sensors are not registered to receive data, they only publish data to the broker.

    Every ten minutes, the temperature sensor publishes the room temperature. In practice this means it sends a single number, say 23, under the topic "temperature" to the broker indicating that it is currently 23° C. Similarly, every fifteen minutes the humidity sensor sends data, 62 for example, under the topic "humidity" to the broker signifying that the humidity factor is 62%. In turn, the broker passes on these messages to the air-conditioning unit. It decides on the basis of these two measurements if it needs to turn on or off. This is the Internet of Things at work.

    A home automation program with built in MQTT support can be added to the system to provide "intelligence" and control. The architecture of the system could be something like this:

    Icons made by EpicCoders from www.flaticon.com is licensed by CC 3.0 BY

    The home automation (ha) program is registered with the MQTT broker to receive all data on the topics "temperature", "pressure" and "air conditioner status". The air conditioner is registered with the MQTT broker to receive all data on the topic "air conditioning".

    As before, the temperature and humidity sensors publish data to the broker. But now the ha program is registered to receive that information. It decides if the air conditioner should be turned on or off by taking into consideration the sensor data, the time of day, the delay since the last time the unit turned on or off, the presence of people in the house and so on. It instructs the air conditioner to turn on or off by publishing to the topic "air conditioning". The air conditioner publishes changes to its on or off state under the "air conditioner status" topic. Because the ha program subscribes to that topic, it knows when the air conditioning has been turned on or off by the homeowner.

    Things are not quite that simple, of course. MQTT is a "transport" protocol, it says nothing about the content of messages. Is that 23 from the temperature sensor 23° centigrade or 23° Fahrenheit? Unless the temperature and humidity sensors are manufactured by the same firm that makes the air conditioner, and unless one of these devices hosts an MQTT broker, there is very little chance that it would work "out of the box". For true IoT, where one can buy sensors from one producer, appliances from another, and a bridge with home automation software from a third firm, there is a need for a universal IoT message syntax that MQTT could transport.

    While some effort has been expanded on this subject (see Homie, an MQTT convention for the IoT by Marvin Roger), something widely used has yet to emerge as far as I know. Nevertheless, there is a good case for using MQTT. It is supported in Domoticz. Using a standard protocol is always a better idea than creating an ad hoc solution. But as will become evident, a lot of "glue" has to be applied to cobble something together.

  3. Installing Mosquitto MQTT
  4. The only real requirement about the host of the MQTT broker is that the computer be always on and connected to the local area network. The Raspberry Pi hosting the Domoticz server is probably the best choice for most. Luckily, one of the best known open source implementation of MQTT, Mosquitto, runs on the little board. Following the instructions, I was able to install the broker without problems.

    To use the Mosquitto Debian repository at Mosquitto.org, you must import the repository package signing key and then make the repository available to apt:

    pi@rpi2b:~ $ wget http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key pi@rpi2b:~ $ sudo apt-key add mosquitto-repo.gpg.key pi@rpi2b:~ $ cd /etc/apt/sources.list.d/ pi@rpi2b:/etc/apt/sources.list.d $ sudo wget http://repo.mosquitto.org/debian/mosquitto-jessie.list

    The package can now be installed and, to clean up, the Mosquitto repository key can be erased.

    pi@rpi2b:/etc/apt/sources.list.d $ cd $home pi@rpi2b:~ $ sudo apt-get udpate pi@rpi2b:~ $ sudo apt-get install mosquitto pi@rpi2b:~ $ rm mosquitto-repo.gpg.key

    The broker is now installed. Make sure that it is there:

    pi@domo:~ $ systemctl status mosquitto ● mosquitto.service - Mosquitto MQTT Broker Loaded: loaded (/lib/systemd/system/mosquitto.service; disabled) Active: active (running) since Tue 2017-05-30 12:06:43 ADT; 9min ago Docs: man:mosquitto(8) https://mosquitto.org/ Main PID: 1547 (mosquitto) CGroup: /system.slice/mosquitto.service └─1547 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

    Reboot and make sure the Mosquitto was started by asking systemctl for the status of mosquitto as above. If the mosquitto.service did not start, you could try enabling it:

    pi@domo:~ $ sudo systemctl enable mosquitto.service Synchronizing state for mosquitto.service with sysvinit using update-rc.d... Executing /usr/sbin/update-rc.d mosquitto defaults Executing /usr/sbin/update-rc.d mosquitto enable
    Hopefully, Mosquitto will now start automatically when booting Raspbian.

    While not essential, you may want to install the Mosquitto clients on the Raspberry Pi:

    pi@domo:~ $ sudo apt-get install mosquitto-clients
    Example use of the clients will be found in the next section.

  5. Testing Mosquitto MQTT
  6. Now that an MQTT broker is installed, I recommend that you read Elliot Williams' excellent instructions at Hackaday as an introduction to using the protocol. Or you can follow along here right away.

    I installed Mosquitto clients on my Ubuntu desktop using the Software Center. It can be done with Synaptic or even with apt-get in a terminal window. The keyboard combination that opens a terminal in Ubuntu is Alt Ctrl T.

    michel@hp:~ $ sudo apt-get install mosquitto-clients
    Instead you could open ssh sessions with the Raspberry Pi and use its Mosquitto clients if they are installed.

    The clients are two programs: mosquitto_sub to subscribe to the broker in order to receive messages and mosquitto_pub to publish messages to the broker. We will open two terminals, one that subscribes to messages, and one to send messages. To avoid confusion, I changed the title to Subscribe in one terminal (look in the Terminal menu) and to Publish in the other.

    In the Subscribe terminal subscribe to all topics that begin with "home" with the MQTT broker on the Raspberry Pi at IP address 192.168.0.22:

    michel@hp:~$ mosquitto_sub -h 192.168.0.22 -v -t "home/#"
    As can be seen in the image, nothing much happens. The program is running but nothing will appear until the MQTT broker gets a message under the "home" topic.

    In the Publish terminal, publish a message to the MQTT broker

    michel@hp:~$ mosquitto_pub -h 192.168.0.22 -t "home/temperature" -m "21"

    The broker will send this message on to the mosquitto_sub process still running in the Subscribe terminal. That process will display the received message, topic first, message after:

    Play with this. Open a third terminal and subscribe to the same topic. Then publish a new message and you will see both subscribing clients will display the message. Publish to other subtopics such as "home/humidity". Try publishing a message with "test/temperature" as a topic.

    To exit the subscribe clients, press the Ctrl C key combination.

    MQTT Topics

    From the above, it should be obvious that MQTT topics are hierarchical and represented in a fashion quite similar to full file paths (using the Unix '/' separator). And, somewhat like filenames, there are wildcards:

    # which represents all lower level topics. Hence subscribing to home/# means that the broker will send all messages with topics such as home, home/humidity, home/temperature, home/temperature/centigrade, home/temperature/fahrenheit and so on.
    + which represents a single level in the hierarchy of topics. Hence subscribing to home/+/switches means that messages with topics such as home/bedroom/switches and home/kitchen/switches will be received, but not home/switches nor home nor switches.

    For obvious reasons, you cannot have an '/', '#' or '+' as part of a topic. It is not a good idea to include these characters in a user name or client id (see Security advisory CVE-2017-7650 at mosquitto.org. You can use UTF-8 encoded text in topics such as /saison/été. At least this works in Linux. How it works in Windows, which uses 16 bit Unicode instead of UTF-8, I have yet to check.

    When debugging MQTT messaging, it may be useful to subscribe to all topics:

    michel@hp:~$ mosquitto_sub -h 192.168.0.22 -v -t "#"

  7. Adding a Virtual Switch in Domoticz
  8. So far no devices have been added to the Domoticz database on the Raspberry Pi 3. It will be difficult to test MQTT in conjunction with the home automation program without at least one virtual switch. The procedure to add a virtual switch, which eventually will be linked to a physical switch is almost exactly the same as the one followed in X10 On/Off Switches with a CM11A Interface in Domoticz.

    The first step is to create a "dummy" hardware; Domoticz does not know how to handle the WiFi switches that I will be installing later on.

    1. Click on the Setup tab.
    2. Select Hardware in the drop-down menu.
    3. Enter a name for the hardware; I chose Virtual.
    4. For the Type select Dummy (Does nothing, use for virtual switches only) in the drop-down list.

    5. Add virtual hardware
    6. Click on the Add button. The added hardware will be added in a table at the top of the page:
    7. This step is done only once.

    To add a virtual switch

    1. Click on the Create Virtual Sensors in the newly created Virtual hardware.
    2. Fill in the Name field, select Switch for Sensor Type in the Create Virtual Sensor window:

    A message will pop up confirming the creation of the sensor and stating that it can be found in the devices' list. To see the list, click on the Setup tab and then select Devices.

    The newly created switch can also be found on the Switches tab.

    Clicking on the virtual switch bulb icon, found on the Switches tab or in the list of devices, will toggle its image to the on or off state. Try it and you will see a message pop up for a couple of seconds saying that the device has been turned on or off as appropriate. Of course, nothing actually happens because this virtual switch is not connected to anything yet. How this works is shown in the next section.

  9. Enabling MQTT in Domoticz
  10. The first step in adding MQTT capabilities to Domoticz is to add the 'hardware'. The procedure is similar to adding the dummy hardware in the previous secion

    1. Click on the Setup tab.
    2. Select Hardware in the drop-down menu.
    3. Enter a name for the hardware. Again showing how imaginative I am, I chose MQTT.
    4. For the Type select MQTT Client Gateway with LAN interface in the drop-down list.
    5. Set the Remote Address and Port of the MQTT broker. In my case, the broker is on the same host as the Domoticz server so I used localhost but I could have used the full IP address 192.168.0.22 or perhaps 127.0.0.1. I set the Port to 1883 which is the Mosquitto MQTT default port which I not changed.
      Add virtual hardware Make sure that Publish Topic is set to out; I think that is the default.
    6. Click on the Add button. The added hardware will be included along with the Virtual hardware in the table at the top of the page.

    From this point on, the Domoticz server will publish MQTT messages on the domoticz/out topic whenever the state of a device changes. At the same time, it subscribes to the domoticz/in topic and will take action whenever it receives a message with the correct syntax.

  11. Testing MQTT in Domoticz
  12. Open a terminal and subscribe to all messages, then toggle the test switch on and off in the Switches tab to see the MQTT messages generated by Domoticz:

    michel@hp:~$ mosquitto_sub -h 192.168.0.22 -v -t "#" domomoticz/out { "Battery" : 255, "RSSI" : 12, "dtype" : "Light/Switch", "id" : "00014051", "idx" : 1, "name" : "Test Switch", "nvalue" : 1, "stype" : "Switch", "svalue1" : "0", "switchType" : "On/Off", "unit" : 1 } domoticz/out { "Battery" : 255, "RSSI" : 12, "dtype" : "Light/Switch", "id" : "00014051", "idx" : 1, "name" : "Test Switch", "nvalue" : 0, "stype" : "Switch", "svalue1" : "0", "switchType" : "On/Off", "unit" : 1 }

    These are typical JSON formatted Domoticz messages. For details about the API see Domoticz API/JSON URL's. The WiFi switches switch will have to handle these to turn itself on or off. As will be seen, this looks much more daunting than it is thanks to the work of many persons that have developed open source libraries for the ESP8266.

    Of course the WiFi switch will have to send MQTT messages to the broker to inform it that someone has manually turned the switch on or off. The topic for those types of messages is domoticz/in because Domoticz subscribes to that topic. To be clearer, I assume that the software module implementing the 'MQTT hardware' does the subscribing when started.

    To continue the test, the typical message that a switch could send when manually turned on will be published to the MQTT broker. I am assuming that the virtual switch that was installed in the previous section has an index (idx) of 1. The index is shown in the devices list. To see the list, click on the Setup tab and then select Devices . If it is not 1, then adjust the value when publising the following MQTT message. michel@hp:~$ mosquitto_pub -h 192.168.0.22 -t "domoticz/in" -m '{ "idx" : 1, "nvalue" : 1}'

    The subscribing terminal will show that the broker got the message domoticz/in { "idx" : 1, "nvalue" : 1}

    The log shows that Domoticz got the message.

    2017-05-31 13:14:38.446 MQTT: Topic: domoticz/in, Message: { "idx" : 1, "nvalue" : 1}
    To see the log click on the Setup tab and then select Log. Then return to the Switches tab, and verify that the bulb icon of the virtual switch is set to the on state. This takes a little while so be patient.

    To turn the virtual switch state to off, a similar MQTT message must be sent: michel@hp:~$ mosquitto_pub -h 192.168.0.22 -t "domoticz/in" -m '{ "idx" : 0, "nvalue" : 0}' Again adjust the idx value to reflect the correct index of the virtual switch.

    There remains to programme the Sonoff WiFi switch to work with MQTT and Domoticz. This is already done in previous French language posts (see Flashing a Sonoff with the Arduino IDE) which I will translate "very soon" and perhaps update to the newer version of Theo Arends firmware at the same time.

Workflow when Working with a Headless Raspberry Pi Flashing a Sonoff in the Arduino IDE