May 23rd, 2017

As mentioned in the first post, Domoticz on a Raspberry Pi - Déjà Vu All Over Again , most of my X10 power line devices will soon have a new home. The system will be built around an "old" Raspberry Pi Model B Rev 2 connected to a CM11A serial computer interface and an undetermined number of X10 lamp, appliance modules and power sockets. It will probably also include some sensors. This blog is mostly a repetition of previous posts, but I hope to provide a better guide this second time around.

The X10 module used to test the hardware and software is an old Radio Shack plug'n power Appliance module 61-2684B which is a rebranded AM466 Appliance module. Any on/off device should work with the scripts presented here including the WS13A, XPS3 and XPS4 Decorator Relay Wall Switch, the new WS469 Push Button Relay Wall Switch, the SR227 and PAO11 SuperSocket Receptacle, the SC546A Remote Chime and so on. The scripts will handle the on off functions of dimmers, but not their dimming functions, presets and so on. That will be covered in the next post.

Table of Contents

  1. Install Heyu
  2. Troubleshooting Communications with the CM11A
  3. Problematic USB-serial Cable
  4. Initial Delay
  5. Removing Timers and Macros Stored in the CM11A
  6. Add X10/Heyu to Domoticz
  7. Scripting X10 Switches

  1. Install Heyu
  2. There is no direct support for the CM11A computer interface in Domoticz. A "gateway" called Heyu can be installed. It is actually a home automation program of its own. Installation of Heyu is done by getting the source code, compiling it and installing the resulting binary.

    Before installing the gateway, connect a USB-RS232 cable to the Raspberry Pi. Check that it has been found by the system. If it is the only USB-serial adaptor connected to the Raspberry Pi, it should show up as ttyUSB0:

    pi@domopole:~/heyu/heyu-2.10 $ ls /dev/tty* /dev/tty /dev/tty19 /dev/tty3 /dev/tty40 /dev/tty51 /dev/tty62 ... /dev/tty14 /dev/tty25 /dev/tty36 /dev/tty47 /dev/tty58 /dev/ttyUSB0
    If it is not possible to connect the converter cable, then answer dummy instead of /dev/ttyUSB0 below. It will be necessary to update the serial port later in the x10config.

    This is the procedure to retrieve and unpack an archive of version 2.10 of the software. There are more recent versions, but I had problems with them when I first tried this a few months back. I did not check if these newer versions would work with the latest version of Raspbian.

    pi@domopole:~ $ mkdir heyu pi@domopole:~ $ cd heyu pi@domopole:~/heyu $ wget -O heyu.tgz https://github.com/HeyuX10Automation/heyu/archive/v2.10.tar.gz pi@domopole:~/heyu $ tar xf heyu.tgz pi@domopole:~/heyu $ cd heyu* pi@domopole:~/heyu/heyu-2.10 $

    Now there remains the compilation and installation. The first step creates a make file with the Configure script. If a full installation of Heyu is acceptable then the just enter ./Configure. This is what I did just a few months ago. However the default build includes a lot of functionality that is not used. Remember, Heyu is a home automation program able to run scheduled events and handle more devices than just the CM11A. Here it is used as a CM11A driver only. Since resources are at a premium on a Raspberry Pi, I decided to read the file called README.INSTALL to obtain the leanest version of Heyu possible.

    pi@domopole:~/heyu/heyu-2.10 $ ./Configure linux -nocm17a -noext0 \ > -norfxs -norfxm -nodmx -noore -nokaku -flags=32 -counters=32 -timers=32 ... pi@domopole:~/heyu/heyu-2.10 $ make ... plenty of time to do something else! ... ** Now become root and run 'make install' ** pi@domopole:~/heyu/heyu-2.10 $ sudo make install mkdir -p -m 755 /usr/local/bin cp heyu /usr/local/bin chgrp root /usr/local/bin/heyu chmod 755 /usr/local/bin/heyu chown root /usr/local/bin/heyu ./install.sh I did not find a Heyu configuration file. Where would you like the sample Heyu configuration file installed? 1. In directory /home/pi/.heyu/ 2. In subdirectory .heyu/ under a different user home directory 3. In directory /etc/heyu (for system-wide access) 4. No thanks, I'll take care of it myself Choice [1, 2, 3, 4] ? 1 Creating directory /home/pi/.heyu The sample configuration file will be installed as /home/pi/.heyu/x10config I will add the TTY port for your CM11 to the config file Specify /dev/ttyS0, /dev/ttyS1, etc., or the word dummy To which port is the CM11 attached? tty tty ./install.sh: 198: [: tty: unexpected operator I could not find the device you specified. Please try again. Specify /dev/ttyS0, /dev/ttyS1, etc., or the word dummy To which port is the CM11 attached? /dev/ttyUSB0 Setting uid:gid = 1000:1000 for /home/pi/.heyu/x10config Changing TTY permissions to 777 The directory /var/tmp/heyu was created with the permissions 777. The permissions for the SPOOL directory (/var/tmp/heyu) are OK The permissions for the directory /var/lock were set to 1777 cat install.sh >install chmod a+x install pi@domopole:~/heyu/heyu-2.10 $
    According to the above, the binary heyu is installed in the /usr/local/bin directory which is in the executable search path:
    pi@domopole:~/heyu/heyu-2.10 $ echo $PATH /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/games:/usr/games

    To check that the installation is working correctly, connect the CM11A to the USB-serial cable, and a lamp to an X10 appliance module. You can use a lamp module, but for this initial device, I will only deal with the on and off functions. Plug the CM11A and the module in nearby receptacles. Taking no chances, I plugged the module into the feedthrough power receptacle of the CM11A which will eliminate any possible phase problem. Set the address of the X10 module; I chose B4. To make things simpler, turn on the lamp manually. You may have to do that more than once for the lamp to light.

    pi@domopole:~/heyu/heyu-2.10 $ cd $home pi@domopole:~ $heyu info starting heyu_relay Heyu version 2.10 Configuration at /home/pi/.heyu/x10config Powerline interface on /dev/ttyUSB0 Firmware revision Level = 7 Interface battery usage = Unknown Raw interface clock: Wed, Day 327, 00:03:28 (--> Civil Time: Wed 23 Nov 2016 00:03:28 AST) No schedule has been uploaded by Heyu. Housecode = A 0 = off, 1 = on, unit 16.......8...4..1 Last addressed device = 0x0000 (0000000000000000) Status of monitored devices = 0x0000 (0000000000000000) Status of dimmed devices = 0x0000 (0000000000000000) pi@domopole:~ $ heyu on b4 pi@domopole:~ $ heyu off b4
    Note the first line about starting the relay. We'll come back to it. Also, take note of the location of the configuration file called x10config.

  3. Troubleshooting Communications with the CM11A
  4. I had problems: heyu info would return an error saying it could not communicate with the CM11A. Since I was confident it could not be a mains phase problem, the source had to be the CM11A or the USB-serial cable. That was odd because both were working just a couple of days ago albeit with the Raspberry Pi 3 and not with this older board.

    Trying to narrow the problem down, I did a Web search and quickly found a message in Yahoo! Groups by Charles Sullivan which provided a bash script with built in instructions to test the serial communication with the CM11A.


    #! /bin/bash
    
    # Display hex output from CM11A requesting time update
    # Author: Charles Sullivan
    # URL: https://groups.yahoo.com/neo/groups/heyu_users/conversations/messages/2729
     
    if (($# != 1)) ; then
    echo "Usage: $0 serial_port"
    exit
    fi
    
    stty -F $1 4800 cs8 raw cread clocal -parenb -cstopb -echo
    
    echo ""
    echo "Make sure the heyu_relay is NOT running ('heyu stop')."
    echo "Unplug the CM11A from AC power for about 5 seconds,"
    echo "then plug it back in and wait for at least 5 seconds."
    echo "The CM11A will start sending its clock update request"
    echo "(0xa5) about once every second. If the CM11A is connected"
    echo "to port "$1", lines like this will be displayed:"
    echo " 0000000: a5 ."
    echo " 0000001: a5 ."
    echo " 0000002: a5 ."
    echo "Otherwise, it is connected to a different port."
    echo " "
    echo "Now waiting for input from port: "$1
    cat $1 | xxd -g1 -c1
    

    Before testing with the script as described below, I performed a mechanical test by which I mean that I used a power line controller to turn off and on the B4 module. I was happy that worked. At that point, I should have run the heyu info command again, but I didn't and instead continued with the script as follows.

    1. pi@domopole:~ $ cd heyu pi@domopole:~/heyu $ nano cm11a_check.sh
      copy and paste the above script
    2. Save the script: press Ctrl O (letter O not the number 0) simultaneously and then press Enter to save the file with its current file name. Exit nano by pressing Ctrl X.

    3. Make the script executable and launch it, but before doing that stop the heyu_relay which would normally be listening to the serial port:
      pi@domopole:~/heyu $ heyu stop pi@domopole:~/heyu $ chmod +x cm11a_check.sh pi@domopole:~/heyu $ ./cm11a_check.sh /dev/ttyUSB0
    4. Follow the instructions and wait some time after plugging the CM11A back into the power socket. Hopefully, it will start sending data, 0xa5, on the serial line.
    5. Press Ctrl C to exit the script.

    I did see the incoming time update requests, 0xA5, from the CM11A which was a good sign; both the CM11A and the serial connection (the incoming side in any case) seemed to be working. I continued by starting the heyu monitor:

    pi@domopole:~/heyu $ heyu monitor 05/23 01:18:57 Monitor started 05/23 01:19:16 CM11A experienced an AC power interruption. It requested a time update. 05/23 01:19:18 CM11A experienced an AC power interruption. It requested a time update. 05/23 01:19:20 CM11A experienced an AC power interruption. It requested a time update. 05/23 01:19:20 Powerfail signal received.
    That was more good news, because when the CM11A stopped requesting time updates it meant that it had obtained a time update from Heyu. As a matter of fact running the test
    pi@domopole:~/heyu $ heyu info pi@domopole:~/heyu $ heyu on b4 pi@domopole:~/heyu $ heyu off b4
    gave the expected result. The Heyu gateway was successfully installed.

    In retrospect, the real problem was probably the CM11A and the solution was turning the lamp module on and off with another X10 controller. This is not the first time that I have noticed that the CM11A is confused when initially connected to the power line. Often, things are put right when it detects a power line command from another controller. This is actually a well known problem and solution. Nevertheless, I find the script useful and I will keep it around.

  5. Problematic USB-serial cable
  6. Of course X10 commands take some time to travel along the power line. So don't expect instantaneous responses. However what I actually got was something like a three or four second delay between invoking Heyu and the lamp or device module reacting. The program provides an explanation for the excessive delay:

    pi@domopole:~ $ heyu on b4 RI serial line may be stuck. RI serial line may be stuck. pi@domopole:~ $ heyu off b4 RI serial line may be stuck. RI serial line may be stuck.
    It turns out that the culprit is the USB-serial cable I used. It was cheap for a reason. The question is addresses in the Heyu FAQ:
    If getting a long delay when switching lights on and off and report of 
    Q: I get the message "RI serial line may be stuck"
       after a long delay.
    
    A: This is a problem with some adapters using an older Prolific chip.
       The workaround is to put the directive 'CHECK_RI_LINE  NO' in your
       Heyu configuration file.
    

    The proposed solution, adding a directive in the configuration file works. Both the installation script and the heyu info command say where the configuration file is located.

    pi@domopole:~ $ nano /home/pi/.heyu/x10config
    ... # HOUSECODE A # A B C D E F G H I J K L M N O P # Workaround for "RI serial line may be stuck" problem with older Prolific chip # based USB-serial adapter: CHECK_RI_LINE NO ...

  7. Initial Delay
  8. Whenever a Heyu command is issued, the program starts the relay (heyu_relay) if it is not already running. It takes a few seconds to start this service. Thus when a first "on" or "off" command is issued after a reboot, there will be a considerable delay before it is executed. It would be better to start the relay during the boot process. This can be done following the advice in X10 with CM11a wiki:

    To restart Heyu at 10 minutes before the end of every hour (because it sometimes fails) and whenever the Rasberry Pi is rebooted, edit the crontab file as follows:

    pi@domopole:~ $ crontab -e ... select the editor you want to use, the default is nano which has been used throught this guide.
    # For more information see the manual pages of crontab(5) and cron(8) # # m h dom mon dow command 50 * * * * /usr/local/bin/heyu start @reboot /usr/local/bin/heyu start

    I have not found Heyu particularly flaky, but the above advice means that a problem could resolve itself after an hour. This could be very useful when away for an extended period.

  9. Removing Timers and Macros Stored in the CM11A
  10. A final pointer: remove all timers and macros that could still be stored in the CM11A. I did not do this and for a while I was wondering what was turning on and off some of the living room lights at seemingly random times. It had been a couple of years since I had last used the CM11A but it still contained some timers.

    pi@domopole:~ $ heyu erase Erasing all blocks of data on the eeprom ................................................................ pi@domopole:~ $

  11. Add X10/Heyu to Domoticz
  12. Reference: Domoticz wiki: X10 with CM11a

    There is no native support for Heyu in Domoticz so it will be necessary to add devices "manually" and to handle them with scripts.

    The first step is to create a "dummy" hardware representing the gateway. Click on the Setup tab and then select Hardware. For a name, I chose Heyu. For the Type click on Dummy (Does nothing, use for virtual switches only) in the drop down list.
    Add dummy hardware
    Don't forget to click on the Add button. This step is done only once. The added hardware will be added in a table at the top of the page:

    To add a virtual switch to represent the physical X10 appliance module with the B4 address click on the Create Virtual Sensors in Heyu hardware. The Create Virtual Sensor window, as shown below, will be displayed. Fill in the Name field, select Switch for Sensor Type and press the OK button.

    A message will pop up confirming the creation of the sensor and stating that it can be found in the devices list. To see the list, click on the Setup tab and then select Devices.

    Clicking on the bulb icon will toggle the X10 module on or off. Try it and you will see a message pop up for a couple of seconds saying that the device has been turned on or off as appropriate. After a short while the colour of the bulb icon will change indicating if the device is on or off. But nothing will happen to the X10 module and the connected lamp.

    If you click on the Switches tab, you will see that the newly created On/Off virtual switch Last Lamp is visible. Again, clicking on its bulb icon will toggle the switch virtually on or off but do nothing to the actual lamp.

    Remember that we have created a virtual sensor/switch which does not know anything about the hardware it supposedly controls. We have to create scripts to handle the hardware.

    There is another way to create a virtual switch that may seem more natural:

    1. Click on the Switches tab.
    2. Click on the Manual Light/Switch at the top left of the page.
    3. Fill in the correct information in the Add Manual Light/Switch Device window.
    4. Click on the Add Device button.

    This does not mean that Domoticz handles X10 hardware. Scripts must still be written. It does mean that the house and unit codes will be saved in the database. They can be seen in the device list. I feel this may be counterproductive because these values cannot be edited easily if at all. The device codes and the database values can easily get out of sync.

    In the first discussion about using X10 modules, I went on at length about the need to plan carefully when allocation house and unit codes to modules. That was because I had been using this second way of creating virtual switches while moving devices around trying to get things to work. By the time I completed that post, I had switched to the "virtual sensor" route where house and unit codes are ignored. Duh...

  13. Scripting X10 Switches
  14. Rules about Lua scripts are quite strict. A Lua script for a virtual sensor

    The following creates a script to turn the "Test Lamp" connected to the appliance module B4 on and off with Domoticz.

    pi@domopole:~ $ cd domoticz/scripts/lua pi@domopole:~/domoticz/scripts/lua $ nano script_device_heyu_b4.lua

    commandArray = {} NAME = 'Test Lamp' ADDR = 'B4' if devicechanged[NAME] then cmd = string.lower(devicechanged[NAME]) if (cmd == 'on') or (cmd == 'off') then cmd = 'sudo -u pi /usr/local/bin/heyu '.. cmd .. ' ' .. ADDR os.execute(cmd) end end return commandArray

    Download: script_device_heyu_b4.lua to directory ~/domoticz/scripts/lua of the Raspberry Pi server. The download version is a bit longer because it has debug code. This should work right away without rebooting or restarting Domoticz. Test it.

    I will quickly explain the script. Each time a switch is toggled in Domoticz, the server executes every script with a file name that matches script_device_*.lua in the /home/pi/domoticz/scripts/lua directory. In addition, the server fills in a number of global variables that are made available to each lua script when it is executed. The most important variables are the tables named devicechanged, otherdevices, uservariables and timeofday. Scripts can do things on their own and can also return the action they want the server to perform in a table variable named commandArray.

    A script must begin by creating an empty commandArray and must end by returning that variable. After the creation of the variable, a script will test if it should do something. Since our script is meant to perform an action only when the virtual switch 'Test Lamp' is toggled, it verifies if devicechanged['Test Lamp'] contains something. If it does not, then the script does nothing else. If it does contain something, it places the content in the local variable cmd converting the string to lower case. Then if cmd is equal to 'on' or 'off' it will use the os.execute procedure to call on Heyu to turn on or off the B4 module. Our script returns an empty commandArray because Domoticz already updates the status of the virtual switch and displays the correct icon to reflect that status.

    There is a missing piece. Domoticz does not know enough to update its database and the status of the virtual switch when I turn the lamp on or off with an X10 controller. Fortunately, Heyu has a scripting mechanism that can be used to notify Domoticz of a change in status of a module. This is what should happen in principle:

    In practice care must be exercised to avoid creating a loop because in that last step Domoticz does not just update its database, it also performs the associated action. In other words it executes the script script_device_heyu_b4.lua which can restart the sequence. This problem is discussed at length in the forums and solutions have been suggested. The solution I have adopted is specific to Heyu using its own script trigger conditions.

    The index number (idx) of the virtual switches must be used in the Heyu scripts. That can be found in the list of devices. If you recall, a click on the Setup tab and then selecting Devices bring up the list:

    The index number is to the right of the bulb icon. It is 1 for 'Test Lamp'.

    The next step is to edit the Heyu configuration file to

    Open a terminal and launch an ssh session with the Raspberry Pi. Then edit the x10config and restart Heyu.

    pi@domopole:~ $ nano .heyu/x10config
    # Start the Heyu Engine daemon automatically (needed to execute scripts) START_ENGINE AUTO # Define the (writeable) directory where the Heyu state engine daemon # (started with 'heyu engine') is to write its log file 'heyu.log.'. # The default is 'NONE', indicating no log file is to be written. # LOG_DIR NONE # LOG_DIR /home/pi Read the rest of the config file for important information about the size of log files if you create one! # Script to update Domoticz device on receiving power line message # SCRIPT B4 on rcvi :: curl "http://192.168.0.22:8080/json.htm?type=command&param=udevice&idx=1&nvalue=1" SCRIPT B4 off rcvi :: curl "http://192.168.0.22:8080/json.htm?type=command&param=udevice&idx=1&nvalue=0"
    Download: x10config-v1.txt to directory /home/pi/.heyu and rename x10config.

    Restart Heyu so that it will reread the changed configuration file.

    pi@domopole:~ $ heyu stop pi@domopole:~ $ heyu start starting heyu_relay starting heyu_engine pi@domopole:~ $
    Note how heyu_engine, which handles events and scripts, is now being loaded when the service starts.

    Test everything, turn test lamp on and off with Domoticz and with an X10 controller. It may take a while, but when using the latter, the bulb icon in the Web interface should reflect the state of the lamp indicating that Domoticz is updating its status in the database.

    If you open the Heyu monitor on the Raspeberry Pi, you will see what happens.

    pi@domopole:~ $ heyu monitor 05/23 22:16:44 Monitor started turning the lamp on with Domoticz 05/23 22:17:40 sndc addr unit 4 : hu B4 (_no_alias_) 05/23 22:17:41 sndc func On : hc B turning the lamp off with Domoticz 05/23 22:18:48 sndc addr unit 4 : hu B4 (_no_alias_) 05/23 22:18:49 sndc func Off : hc B turning the lamp on with an X10 controller 05/23 22:19:55 rcvi addr unit 4 : hu B4 (_no_alias_) 05/23 22:19:55 rcvi func On : hc B 05/23 22:19:56 sndc addr unit 4 : hu B4 (_no_alias_) 05/23 22:19:57 sndc func On : hc B turning the lamp off with an X10 controller 05/23 22:23:25 rcvi addr unit 4 : hu B4 (_no_alias_) 05/23 22:23:25 rcvi func Off : hc B 05/23 22:23:26 sndc addr unit 4 : hu B4 (_no_alias_) 05/23 22:23:27 sndc func Off : hc B

    The command is repeated when turning the lamp off or on with the X10 controller. As explained above, the Heyu script informs the Domoticz server that the lamp status has changed through an HTTP request. When the server updates its database, it executes the lua script to turn on or off the lamp. This is why there could be an infinite loop if Heyu script trigger "rcvi" had not been used. Heyu does not fire the on and off scripts when it sends the command to the CM11A for transmission on the power line ("sndc"), it only fires them when it is notified by the CM11A ("rcvi") that the latter heard an on or off command sent by another device over the power line.