2021-10-19
md
Additional Servers on a Raspberry Pi Based Home Automation System
Setting up a Raspberry Pi as a Headless Computer (September 2021) (back to part I)-> <-Home Automation System on a Raspberry Pi (part II)
 Draft of the third part of a projected multi-part guide 

This is part 3 of the series of posts about installing a home automation system around Domoticz on a Raspberry Pi. It covers installing additional services that I find useful, but that are not truly required for the home automation system. This is still a work in progress but I wanted to make it available sooner rather than later, especially because of the improvement in the presentation about video streaming. Expect additions and corrections in the upcoming weeks.

Table of Contents

  1. Installing the WireGuard VPN Server
  2. Installing the Syncthing File Synchronization Application
  3. Installing the MJPG-Streamer Video Streamer
    1. Using a Webcam
    2. Controlling the Video Streaming Service
    3. Integration into Domoticz
    4. Secured Streaming
    5. Using a Raspberry Pi Camera Module
  4. Installing the Radicale Calendar and Contact Server
  5. Installing the chrony Network Time Client and Server
    1. NTP Client
    2. NTP Server
    3. Tasmota NTP Settings

Installing the WireGuard VPN Server toc

One of the perks of having a home automation system is that it allows one to control devices in the house from outside. Perhaps the simplest way of going about that is to use port forwarding which involves opening a TCP or UDP port of the local area network (LAN) for each service or device to be reached from the outside world. This can become an administrative nightmare if many ports have to be forwarded, and it does introduce security risks. For the last two years or so, I have been using a virtual private network (VPN). The point of hosting a VPN on the local area network is that it makes it possible to securely access all resources on the LAN while opening a single port (or UDP port in the case of Wireguard). Once the connection to the VPN server is established (this is often referred to "establishing or opening a tunnel") from outside the LAN, all IP addresses on the LAN can be reached and the "tunnel" takes care of encrypting the IP traffic meaning that unsecured protocols like HTTP or telnet (anyone still use that?) can be used safely.

Initially, I used OpenVPN (probably PiVPN; be careful, the advice about running downloaded bash scripts applies here too) with success, but I quickly switched to WireGuard and I am pleased with the results. Lately the wireguard package has become available in the official repository

nostra@damus:~ $ sudo apt-cache policy wireguard wireguard: Installed: (none) Candidate: 1.0.20200827-1~bpo10+1 Version table: 1.0.20200827-1~bpo10+1 500 500 http://archive.raspberrypi.org/debian buster/main armhf Packages

so installation of WireGuard is now as simple as can be.

nostra@damus:~ $ sudo apt install wireguard -y ... The following additional packages will be installed: wireguard-tools The following NEW packages will be installed: wireguard wireguard-tools 0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded. Need to get 85.1 kB of archives. After this operation, 302 kB of additional disk space will be used. ...

There is not much to check until the server and clients have been configured. At a minimum verify that the wg-quick utility is installed and that is configuration directory ias created although it is empty.

nostra@damus:~ $ which wg-quick /usr/bin/wg-quick nostra@damus:~ $ sudo ls -l /etc/wireguard total 0

Let's create an empty configuration file and then bring up the VPN server and see that it is running.

nostra@damus:~ $ sudo touch /etc/wireguard/wg0.conf nostra@damus:~ $ wg-quick up wg0 [#] ip link add wg0 type wireguard [#] wg setconf wg0 /dev/fd/63 [#] ip link set mtu 1420 up dev wg0 nostra@damus:~ $ sudo wg interface: wg0 listening port: 38343 nostra@damus:~ $ wg nostra@damus:~ $ ip a ... 4: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000 link/none nostra@damus:~ $ sudo systemctl status wg-quick@wg0 ● wg-quick@wg0.service - WireGuard via wg-quick(8) for wg0 Loaded: loaded (/lib/systemd/system/wg-quick@.service; disabled; vendor preset: enabled) Active: inactive (dead) Docs: man:wg-quick(8) man:wg(8) https://www.wireguard.com/ https://www.wireguard.com/quickstart/ https://git.zx2c4.com/wireguard-tools/about/src/man/wg-quick.8 https://git.zx2c4.com/wireguard-tools/about/src/man/wg.8

Because WireGuard has not been configured, the virtual network interface has been created but it is inactive. Enabling automatic start of the wg0 interface at boot time makes sense on a server.

nostra@damus:~ $ sudo systemctl enable wg-quick@wg0 Created symlink /etc/systemd/system/multi-user.target.wants/wg-quick@wg0.service → /lib/systemd/system/wg-quick@.service.

At this point, it makes sense to stop the server and proceed with the configuration of the VPN.

nostra@damus:~ $ wg-quick down wg0 [#] ip link delete dev wg0

This remaining task is now the most complex part of setting up the virtual network. Luckily, some kind and clever individuals have provided scripts that simplify the whole procedure considerably. I will shamelessly suggest reading my post entitled Installing and Configuring WireGuard on Raspberry Pi OS (September 2021) for details on configuring the VPN server and on installing "clients" or "peers" that will access this self-hosted VPN server.

Installing Syncthing toc

In the previous edition of this post, I showed how to install Synchthing a decentralized file synchronization program saying that I wanted to use it to synchronize directories that contained scripts and the Domoticz database. As it turned out, I never did use Synchthing for backing up the many scripts on the Raspberry Pi. And lately I have stopped using Synchthing for backing up the Domoticz. Nevertheless, here is an update to how to install the service because I think it could be useful. The installation procedure has not changed much

Using apt-cache policy, it became clear that Syncthing in the Debian and Raspbian repository are rather out of date.

nostra@damus:~ $ apt-cache policy syncthing syncthing: Installed: (none) Candidate: 1.0.0~ds1-1 Version table: 1.1.4~ds1-4 150 150 http://deb.debian.org/debian unstable/main armhf Packages 1.0.0~ds1-1 500 500 http://raspbian.raspberrypi.org/raspbian buster/main armhf Packages

So I more or less followed the instructions at Syncthing, Debian/Ubuntu Packages to get the latest stable version (1.18).

nostra@damus:~ $ curl -s https://syncthing.net/release-key.txt | sudo apt-key add - OK nostra@damus:~ $ echo "deb https://apt.syncthing.net/ syncthing stable" | sudo tee /etc/apt/sources.list.d/syncthing.list deb https://apt.syncthing.net/ syncthing candidate nostra@damus:~ $ sudo apt-get update Get:1 http://deb.debian.org/debian unstable InRelease [139 kB] ... Fetched 14.4 MB in 19s (747 kB/s) Reading package lists... Done nostra@damus:~ $ printf "Package: *\nPin: origin apt.syncthing.net\nPin-Priority: 990\n" | sudo tee /etc/apt/preferences.d/syncthing Package: * Pin: origin apt.syncthing.net Pin-Priority: 990 nostra@damus:~ $ apt-cache policy syncthing syncthing: Installed: (none) Candidate: 1.18.2 Version table: 1.18.2 990 990 https://apt.syncthing.net syncthing/stable armhf Packages 1.18.1 990 990 https://apt.syncthing.net syncthing/stable armhf Packages 1.0.0~ds1-1 500 500 http://raspbian.raspberrypi.org/raspbian buster/main armhf Packages nostra@damus:~ $ sudo apt install syncthing ... The following NEW packages will be installed: syncthing 0 upgraded, 1 newly installed, 0 to remove and 5 not upgraded. Need to get 9,192 kB of archives. After this operation, 21.5 MB of additional disk space will be used. ...

Now start the program manually to verify that it functions.

nostra@damus:~ $ syncthing [monitor] 16:30:23 INFO: We will skip creation of a default folder on first start [start] 16:30:23 INFO: syncthing v1.18.2 "Fermium Flea" (go1.17 linux-arm) deb@build.syncthing.net 2021-08-22 18:04:47 UTC [noupgrade] [start] 16:30:23 INFO: Generating ECDSA key and certificate for syncthing... [start] 16:30:23 INFO: Default folder created and/or linked to new config [start] 16:30:23 INFO: Default config saved. Edit /home/nostra/.config/syncthing/config.xml to taste (with Syncthing stopped) or use the GUI [start] 16:30:23 INFO: Archiving a copy of old config file format at: /home/nostra/.config/syncthing/config.xml.v0 [NTGKU] 16:30:26 INFO: My ID: NTGKU7Z-EROOV7Z-AZCOREN-VO5GFWP-WTK6W65-EPUPRZB-ZZWNXFK-TXL4JQH [NTGKU] 16:30:27 INFO: Single thread SHA256 performance is 16 MB/s using minio/sha256-simd (16 MB/s using crypto/sha256). [NTGKU] 16:30:28 INFO: Hashing performance is 15.08 MB/s [NTGKU] 16:30:28 INFO: Running database migration 1... [NTGKU] 16:30:28 INFO: Running database migration 2... [NTGKU] 16:30:28 INFO: Running database migration 3... [NTGKU] 16:30:28 INFO: Running database migration 5... [NTGKU] 16:30:28 INFO: Running database migration 6... [NTGKU] 16:30:28 INFO: Running database migration 7... [NTGKU] 16:30:28 INFO: Running database migration 9... [NTGKU] 16:30:28 INFO: Running database migration 10... [NTGKU] 16:30:28 INFO: Running database migration 11... [NTGKU] 16:30:28 INFO: Running database migration 13... [NTGKU] 16:30:28 INFO: Running database migration 14... [NTGKU] 16:30:28 INFO: Running database migration 16... [NTGKU] 16:30:28 INFO: Running database migration 17... [NTGKU] 16:30:28 INFO: Running database migration 19... [NTGKU] 16:30:28 INFO: Compacting database after migration... [NTGKU] 16:30:28 INFO: Overall send rate is unlimited, receive rate is unlimited [NTGKU] 16:30:28 INFO: No stored folder metadata for "default"; recalculating [NTGKU] 16:30:28 INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) starting [NTGKU] 16:30:28 INFO: TCP listener ([::]:22000) starting [NTGKU] 16:30:28 INFO: Using discovery mechanism: global discovery server https://discovery.syncthing.net/v2/?noannounce&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW [NTGKU] 16:30:28 INFO: Using discovery mechanism: global discovery server https://discovery-v4.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW 2021/09/25 16:30:28 failed to sufficiently increase receive buffer size (was: 176 kiB, wanted: 2048 kiB, got: 352 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details. [NTGKU] 16:30:28 INFO: Using discovery mechanism: global discovery server https://discovery-v6.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW [NTGKU] 16:30:28 INFO: Using discovery mechanism: IPv4 local broadcast discovery on port 21027 [NTGKU] 16:30:28 INFO: QUIC listener ([::]:22000) starting [NTGKU] 16:30:28 INFO: Using discovery mechanism: IPv6 local multicast discovery on address [ff12::8384]:21027 [NTGKU] 16:30:28 INFO: Loading HTTPS certificate: open /home/nostra/.config/syncthing/https-cert.pem: no such file or directory [NTGKU] 16:30:28 INFO: Creating new HTTPS certificate [NTGKU] 16:30:28 INFO: Ready to synchronize "Default Folder" (default) (sendreceive) [NTGKU] 16:30:28 INFO: Completed initial scan of sendreceive folder "Default Folder" (default) [NTGKU] 16:30:28 INFO: GUI and API listening on 127.0.0.1:8384 [NTGKU] 16:30:28 INFO: Access the GUI via the following URL: http://127.0.0.1:8384/ [NTGKU] 16:30:28 INFO: My name is "damus" [NTGKU] 16:30:42 INFO: Detected 1 NAT service [NTGKU] 16:30:47 INFO: quic://0.0.0.0:22000 detected NAT type: Port restricted NAT [NTGKU] 16:30:47 INFO: quic://0.0.0.0:22000 resolved external address quic://174.116.200.129:22000 (via stun.syncthing.net:3478) [NTGKU] 16:31:27 INFO: Joined relay relay://199.195.251.28:22067 CtrlC shutting down the application [monitor] 16:32:15 INFO: Signal 2 received; exiting [NTGKU] 16:32:15 INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) shutting down [NTGKU] 16:32:15 INFO: QUIC listener ([::]:22000) shutting down [NTGKU] 16:32:15 INFO: TCP listener ([::]:22000) shutting down [NTGKU] 16:32:15 INFO: Exiting

Since I want to control the application through its Web interface from my desktop computer, I need to modify the configuration as explained in the FAQ.

nostra@damus:~ $ nano .config/syncthing/config.xml

Locate the gui entry and change the <address> value from 127.0.0.1:8384 to 0.0.0.0:8384.

<gui enabled="true" tls="false" debugging="false"> <address>0.0.0.0:8384</address> <apikey>...

Restart syncthing

nostra@damus:~ $ syncthing

and note how the message

[NTGKU] 16:30:28 INFO: GUI and API listening on 127.0.0.1:8384

changed to

[NTGKU] 16:50:02 INFO: GUI and API listening on [::]:8384

which means it will be possible to open the Syncthing web interface from the desktop using a Web browser pointed to 192.168.1.139:8384. There was a warning about the lack of security, so I followed the instruction to add a user and password to access the GUI interface.

At the same time, I disabled all connections except for Local Discovery because I want to limit the activity of this application to the local network.

After shutting down the service from the GUI (Actions / Shudown), I restarted it from the command line

nostra@damus:~ $ syncthing [start] 16:50:00 INFO: syncthing v1.18.2 "Fermium Flea" (go1.17 linux-arm) deb@build.syncthing.net 2021-08-22 18:04:47 UTC [noupgrade] [NTGKU] 16:50:00 INFO: My ID: NTGKU7Z-EROOV7Z-AZCOREN-VO5GFWP-WTK6W65-EPUPRZB-ZZWNXFK-TXL4JQH [NTGKU] 16:50:01 INFO: Single thread SHA256 performance is 16 MB/s using minio/sha256-simd (16 MB/s using crypto/sha256). [NTGKU] 16:50:02 INFO: Hashing performance is 14.99 MB/s [NTGKU] 16:50:02 INFO: Overall send rate is unlimited, receive rate is unlimited [NTGKU] 16:50:02 INFO: Using discovery mechanism: IPv4 local broadcast discovery on port 21027 [NTGKU] 16:50:02 INFO: Using discovery mechanism: IPv6 local multicast discovery on address [ff12::8384]:21027 [NTGKU] 16:50:02 INFO: Ready to synchronize "Default Folder" (default) (sendreceive) [NTGKU] 16:50:02 INFO: TCP listener ([::]:22000) starting 2021/09/25 16:50:02 failed to sufficiently increase receive buffer size (was: 176 kiB, wanted: 2048 kiB, got: 352 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details. [NTGKU] 16:50:02 INFO: QUIC listener ([::]:22000) starting [NTGKU] 16:50:02 INFO: Completed initial scan of sendreceive folder "Default Folder" (default) [NTGKU] 16:50:02 INFO: GUI and API listening on [::]:8384 [NTGKU] 16:50:02 INFO: Access the GUI via the following URL: http://127.0.0.1:8384/ [NTGKU] 16:50:02 INFO: My name is "damus"

and saw that it was no longer connecting to outside relay servers.

If you wanted syncthing to start automatically when the Raspberry Pi is booted, then enable the service as follows.

nostra@damus:~ $ sudo systemctl enable syncthing@nostra.service Created symlink /etc/systemd/system/multi-user.target.wants/syncthing@nostra.service → /lib/systemd/system/syncthing@.service.

About the failed to increase receive buffer message

Just before starting the QUIC listener, Syncthing emits an error message.

[NTGKU] 16:50:02 INFO: TCP listener ([::]:22000) starting
2021/09/25 16:50:02 failed to sufficiently increase receive buffer size (was: 176 kiB, wanted: 2048 kiB, got: 352 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
[NTGKU] 16:50:02 INFO: QUIC listener ([::]:22000) starting

Following the link you will find a solution to the problem that presumably would apply in Raspberry Pi OS. I did not follow through immediately with the recommendation to use sysctl to set a higher limit for the UDP buffer size. Then I noticed that the error was no longer shown when Syncthing was restricted to the local network. Does that mean that the QUIC protocol is used only during the discovery of remote relays? Perhaps it is only used when syncing files using remote relays? If both these things are true, it would make sense that the error does not need to be addressed when the service is used over the LAN only.

For information on setting up Syncthing on other computers and Android tablets and sharing directories see the documentation.

Installing the MJPG-streamer Video Streamer toc

Ostensibly, this guide is about setting up Domoticz on a Raspberry Pi. To be truthful, the video capabilities of the home automation software are minimal. Up to now at least, Domoticz is only capable of displaying single JPEG images or snapshots taken by supported cameras. Video streaming is actually just a series of snapshots taken at short intervals. I suspect that many would want more from a video camera connected to the Pi. So below, I rehash a previous post on streaming a USB webcam connected how to a Raspberry Pi over HTTP. It just happens that the streamer can also provide snapshots that Domoticz can use.

MJPG-streamer takes JPGs from Linux-UVC compatible webcams, filesystem or other input plugins and streams them as M-JPEG via HTTP to webbrowsers, VLC and other software. It is the successor of uvc-streamer, a Linux-UVC streaming application with Pan/Tilt. The author, Tom Stöveken, has abandonned the project but the code has been forked many times. However, the fork by jacksonliam is the "designated successor".

There is no binary package in the repository, mjpg_streamer has to be built from the source code and before that can be done some prerequisites must be installed.

nostra@damus:~ $ sudo apt-get install cmake libjpeg8-dev -y ... upgraded, 8 newly installed, 0 to remove and 0 not upgraded. Need to get 4,914 kB of archives. After this operation, 23.2 MB of additional disk space will be used. ... nostra@damus:~ $ git clone https://github.com/jacksonliam/mjpg-streamer.git Cloning into 'mjpg-streamer'... remote: Enumerating objects: 2964, done. remote: Total 2964 (delta 0), reused 0 (delta 0), pack-reused 2964 Receiving objects: 100% (2964/2964), 3.48 MiB | 4.29 MiB/s, done. Resolving deltas: 100% (1885/1885), done. nostra@damus:~ $ cd mjpg-streamer/mjpg-streamer-experimental/ nostra@damus:~/mjpg-streamer/mjpg-streamer-experimental $ make [ -d _build ] || mkdir _build ... make[1]: Leaving directory '/home/nostra/mjpg-streamer/mjpg-streamer-experimental/_build' nostra@damus:~/mjpg-streamer/mjpg-streamer-experimental $ sudo make install make -C _build install ... make[1]: Leaving directory '/home/nostra/mjpg-streamer/mjpg-streamer-experimental/_build'

Let's locate the executable and its input and output libraries.

nostra@damus:~/mjpg-streamer/mjpg-streamer-experimental $ cd ~ nostra@damus:~ $ which mjpg_streamer /usr/local/bin/mjpg_streamer nostra@damus:~ $ ls /usr/local/lib/mjpg-streamer input_file.so input_raspicam.so output_file.so output_rtsp.so input_http.so input_uvc.so output_http.so output_udp.so

The input module to use for USB webcam is input_uvc.so, while the output module is output_http.so. As far as I can determine the UDP and RTSP output modules are not functionnal and I have not tried the other modules.

Using a Webcam toc

We are now in a position to test the video streaming service with default values. Plug a USB webcam into the Raspberry Pi and enter the following command. Since both MJPG-Streamer and Domoticz use TCP port 8080 by default, it is necessary to change one of those. I decided to change the TCP streaming port to 8085.

nostra@damus:~ $ mjpg_streamer -o "output_http.so -p 8085" -i "input_uvc.so" MJPG Streamer Version: git rev: 310b29f4a94c46652b20c4b7b6e5cf24e532af39 i: Using V4L2 device.: /dev/video0 i: Desired Resolution: 640 x 480 i: Frames Per Second.: -1 i: Format............: JPEG i: TV-Norm...........: DEFAULT UVCIOC_CTRL_ADD - Error at Pan (relative): Inappropriate ioctl for device (25) UVCIOC_CTRL_ADD - Error at Tilt (relative): Inappropriate ioctl for device (25) UVCIOC_CTRL_ADD - Error at Pan Reset: Inappropriate ioctl for device (25) UVCIOC_CTRL_ADD - Error at Tilt Reset: Inappropriate ioctl for device (25) UVCIOC_CTRL_ADD - Error at Pan/tilt Reset: Inappropriate ioctl for device (25) UVCIOC_CTRL_ADD - Error at Focus (absolute): Inappropriate ioctl for device (25) UVCIOC_CTRL_MAP - Error at Pan (relative): Inappropriate ioctl for device (25) UVCIOC_CTRL_MAP - Error at Tilt (relative): Inappropriate ioctl for device (25) UVCIOC_CTRL_MAP - Error at Pan Reset: Inappropriate ioctl for device (25) UVCIOC_CTRL_MAP - Error at Tilt Reset: Inappropriate ioctl for device (25) UVCIOC_CTRL_MAP - Error at Pan/tilt Reset: Inappropriate ioctl for device (25) UVCIOC_CTRL_MAP - Error at Focus (absolute): Inappropriate ioctl for device (25) UVCIOC_CTRL_MAP - Error at LED1 Mode: Inappropriate ioctl for device (25) UVCIOC_CTRL_MAP - Error at LED1 Frequency: Inappropriate ioctl for device (25) UVCIOC_CTRL_MAP - Error at Disable video processing: Inappropriate ioctl for device (25) UVCIOC_CTRL_MAP - Error at Raw bits per pixel: Inappropriate ioctl for device (25) o: www-folder-path......: disabled o: HTTP TCP port........: 8085 o: HTTP Listen Address..: (null) o: username:password....: disabled o: commands.............: enabled

Despite all the error messages that have to do with non-existent capabilities of my old webcam, the video stream can be seen on a web browser at the following address: http://192.168.1.139:8085/?action=stream. Single JPEG images, snapshots, can be obtained with the following HTTP request: http://192.168.1.139:8085/?action=snapshot. Of course, the local host name of the Pi, damus.local, could be used instead of the IP address, if the operating system and web browser support mDNS.

The output_http module has another ouput option that will provide access to a number of web pages designed to show the video stream or snapshots in various ways. Use this command.

nostra@damus:~ $ mjpg_streamer -o "output_http.so -p 8085 -w /usr/local/share/mjpg-streamer/www" -i "input_uvc.so"

The simple URL to access these pages is http://192.168.1.139:8085/ or http://damus.local:8085/. There is a menu on the left of the page. The screenshot below shows the Static page which displays a snapshot taken with the webcam when the HTTP request for the page is received.

And yes, what you are seeing is a fuzzy image of the Raspberry Pi 3 B taken with a webcam while a camera module is also connected to the Pi. More about that later. Note that there is a Control page. It is worth looking at it because the input module has many options which can have a great deal of impact on the performance and quality of the video stream. It will pay to experiment with those. Here is the list of the options.

nostra@damus:~ $ mjpg_streamer -i "input_uvc.so --help" MJPG Streamer Version: git rev: 310b29f4a94c46652b20c4b7b6e5cf24e532af39 --------------------------------------------------------------- Help for input plugin..: UVC webcam grabber --------------------------------------------------------------- The following parameters can be passed to this plugin: [-d | --device ].......: video device to open (your camera) [-r | --resolution ]...: the resolution of the video device, can be one of the following strings: QQVGA QCIF CGA QVGA CIF PAL VGA SVGA XGA HD SXGA UXGA FHD or a custom value like the following example: 640x480 [-f | --fps ]..........: frames per second (camera may coerce to different value) [-q | --quality ] .....: set quality of JPEG encoding [-m | --minimum_size ].: drop frames smaller then this limit, useful if the webcam produces small-sized garbage frames may happen under low light conditions [-e | --every_frame ]..: drop all frames except numbered [-n | --no_dynctrl ]...: do not initalize dynctrls of Linux-UVC driver [-l | --led ]..........: switch the LED "on", "off", let it "blink" or leave it up to the driver using the value "auto" [-t | --tvnorm ] ......: set TV-Norm pal, ntsc or secam [-u | --uyvy ] ........: Use UYVY format, default: MJPEG (uses more cpu power) [-y | --yuv ] ........: Use YUV format, default: MJPEG (uses more cpu power) [-fourcc ] ............: Use FOURCC codec 'argopt', currently supported codecs are: RGB24, RGBP [-timestamp ]..........: Populate frame timestamp with system time [-softfps] ............: Drop frames to try and achieve this fps set your camera to its maximum fps to avoid stuttering [-timeout] ............: Timeout for device querying (seconds) [-dv_timings] .........: Enable DV timings queriyng and events processing --------------------------------------------------------------- Optional parameters (may not be supported by all cameras): [-br ].................: Set image brightness (auto or integer) [-co ].................: Set image contrast (integer) [-sh ].................: Set image sharpness (integer) [-sa ].................: Set image saturation (integer) [-cb ].................: Set color balance (auto or integer) [-wb ].................: Set white balance (auto or integer) [-ex ].................: Set exposure (auto, shutter-priority, aperature-priority, or integer) [-bk ].................: Set backlight compensation (integer) [-rot ]................: Set image rotation (0-359) [-hf ].................: Set horizontal flip (true/false) [-vf ].................: Set vertical flip (true/false) [-pl ].................: Set power line filter (disabled, 50hz, 60hz, auto) [-gain ]...............: Set gain (auto or integer) [-cagc ]...............: Set chroma gain control (auto or integer) ---------------------------------------------------------------

Of course, the parameters will have to be adjusted according to the webcam used. Here are the basic settings for two old models I have used:

Logitech C270
-i "input_uvc.so -n -f 10 -r 1280x720"
gives a 1280 by 720 pixel video stream at 10 frames per second.
Microsoft VX2000
-i "input_uvc.so -n -softfps 10"
gives a 640 by 480 pixel video stream at 10 frames per second although the video is captured at 30 frames per second.

Controlling the Video Streaming Service toc

In the currently running home automation system, a single Logitech C270 is permanently connected to the Raspberry Pi. It is turned of or on with a bash script as described in a previous post. This time around I wanted to do something similar but using a systemd unit file and the systemctl utility.

[Unit] Description=A server for streaming Motion-JPEG from /dev/video0 - Microsoft LifeCam VX-2000 (045e:0761) After=network.target [Service] ExecStart=/usr/local/bin/mjpg_streamer -i 'input_uvc.so -n -softfps 10' -o 'output_http.so -p 8085 -w /usr/local/share/mjpg-streamer/www' Restart=always RestartSec=10 [Install] WantedBy=multi-user.target

This file should be stored in the /etc/systemd/system/. Since it must belong to root, I created the file with the usual sudo nano mjpg_streamer.service command. A couple of comments about this unit file are in order.

If this were the typical service, it would be automatically started when the Pi is booted. In other words, the service needs to be enabled.

nostra@damus:~ $ sudo systemctl enable mjpg_streamer.service Created symlink /etc/systemd/system/multi-user.target.wants/mjpg_streamer.service → /etc/systemd/system/mjpg_streamer.service.

It is a simple matter to reboot the Pi and the verify that the camera is on which is kind of obvious because its LED remains on once the Pi has started. One could check by looking at the status of the service.

nostra@damus:~ $ sudo systemctl status mjpg_streamer ● mjpg_streamer.service - A server for streaming Motion-JPEG from /dev/video0 - Microsoft LifeCam VX-2000 (045e:0761) Loaded: loaded (/etc/systemd/system/mjpg_streamer.service; enabled; vendor preset: enabled) Active: active (running) since Sat 2021-10-16 12:37:51 ADT; 2min 25s ago Main PID: 566 (mjpg_streamer) Tasks: 4 (limit: 1935) CGroup: /system.slice/mjpg_streamer.service └─566 /usr/local/bin/mjpg_streamer -i input_uvc.so -n -softfps 10 -o output_http.so -p 8085 Oct 16 12:37:51 damus mjpg_streamer[566]: o: HTTP Listen Address..: (null) Oct 16 12:37:51 damus mjpg_streamer[566]: o: username:password....: disabled Oct 16 12:37:51 damus mjpg_streamer[566]: o: commands.............: enabled Oct 16 12:37:51 damus mjpg_streamer[566]: MJPG-streamer [566]: www-folder-path......: disabled Oct 16 12:37:51 damus mjpg_streamer[566]: MJPG-streamer [566]: HTTP TCP port........: 8085 Oct 16 12:37:51 damus mjpg_streamer[566]: MJPG-streamer [566]: HTTP Listen Address..: (null) Oct 16 12:37:51 damus mjpg_streamer[566]: MJPG-streamer [566]: username:password....: disabled Oct 16 12:37:51 damus mjpg_streamer[566]: MJPG-streamer [566]: commands.............: enabled Oct 16 12:37:51 damus mjpg_streamer[566]: MJPG-streamer [566]: starting input plugin input_uvc.so Oct 16 12:37:51 damus mjpg_streamer[566]: MJPG-streamer [566]: starting output plugin: output_http.so (ID: 00)

Of course the real test is to stream the video with a web browser at damus.local:8085/?action=stream which should work even when the -w /usr/local/share/mjpg-streamer/www option is not included in the unit file.

The streamer can be stopped and started with the usual commands.

nostra@damus:~ $ sudo systemctl stop mjpg_streamer nostra@damus:~ $ sudo systemctl start mjpg_streamer

My spouse is adament that cameras inside the house should not be on, spying on us, as it were. Because I concur, the service was disabled after the above tests were completed.

nostra@damus:~ $ sudo systemctl disable mjpg_streamer Removed /etc/systemd/system/multi-user.target.wants/mjpg_streamer.service.

It is a simple matter to start and stop the streamer when needed such as when we leave the house. More on that later. Before moving on to that topic, a warning is in order. This configuration is only meant for a single USB webcam that is permanently connected to the Pi. If the camera is unplugged and then plugged back in, there is a high probability that the video stream will not work without manually stopping and then starting the service.

Integration into Domoticz toc

Adding a webcam to Domoticz is quite simple. It involves clicking on the menu choices, Setup, More Options, Cameras, and finally Add Camera to get to a form that must be filled out.

Provide an appropriate name for the camera, chose HTTP for the protocol, enter the Raspberry Pi IP address or the local host name as shown if supported, the TCP port specified for the output_html.so module, and finally the query part of the URL to obtain a single image from the streamer which, as we have seen, is ?action=snapshot. Don't forget to click on the Add button at the button.

Once the webcam has been added, it will show up in the Cameras table.

If the webcam icon to the right of the camera name in the table is clicked, a pop-up window will display a pseudo-video stream from the webcam. What is shown is actually a series of independent snapshots taken one at a time. If the still camera icon is clicked then a snapshot is taken and a pop-up menu will appear in which the user can choose to save the file on the device used to open the Domoticz web interface or to open it directly with an image viewer.

Snapshots can be taken and the pseudo-video stream can be viewed through the Domoticz web interface while the video stream at damus.local:8085/?action=stream is concurrently viewed in a browser.

Of course, nothing will be visible if the mjpg_streamer.service is not running. We can set up a Domoticz virtual switch to take care of turning the service on and off. That virtual switch will need the help of a script to perform its task but that is not too difficult to create. Here is one such script.

#!/bin/bash case "$1" in start) if systemctl is-active --quiet mjpg_streamer.service then echo "mjpg_streamer already running" else sudo systemctl start mjpg_streamer.service echo "mjpg_streamer started" fi ;; stop) if systemctl is-active --quiet mjpg_streamer.service then sudo systemctl stop mjpg_streamer.service echo "mjpg_streamer stopped" else echo "mjpg_streamer is not running" fi ;; *) echo "Usage: $0 {start|stop}" exit 1 ;; esac exit 0

The script named webcam (available for download) is saved in the /home/nostra/.local/bin directory. Don't forget to make it executable.

nostra@damus:~ $ chmod +x .local/bin/webcam

Before moving on, test the script.

nostra@damus:~ $ webcam start mjpg_streamer started nostra@damus:~ $ webcam start mjpg_streamer already running nostra@damus:~ $ webcam stop mjpg_streamer stopped nostra@damus:~ $ webcam stop mjpg_streamer is not running nostra@damus:~ $ webcam strt Usage: /home/nostra/.local/bin/webcam {start|stop}

Now, let's create a virtual switch. This has been covered in the first part of this guide in the section Adding a Virtual Switch in Domoticz. But quickly, click on the following sequence of buttons.

Then in the Create Virtual Sensor window

To complete the installation, go to the Switches tab.

Click on the Edit button of the Webcam virtual switch.

The important task here is to specify the On Action and Off Action fields. In both cases, the webcam script is run, the only difference being the command line option which is start when turning on the streamer and stop when turning it off. Be careful there are 3 forward slashes between script: and home/.... That's because in typical request syntax, the action to perform is script:// and the fully qualified path to the script begins with an / also.

I also changed the Switch Icon from the default light bulb to the Generic on/off icon. Custom icons could be defined, say webcam48_On.png, webcam48_Off.png, but how to install them is outside the scope of this section.

Do not forget to press the Save button. The next time you want to take a snapshot or view the pseudo-video stream with the Domoticz web interface, clik on Setup / More Options / Cameras. If there is not preview in the webcam entry as shown here

then go back to the Switches tab and turn the webcam on by clicking on its icon.

Secured Streaming toc

Looking at the help for the HTTP output module shows that a user name and password can be required to see the video output.

nostra@damus:~ $ mjpg_streamer -o "output_http.so --help" MJPG Streamer Version: git rev: 310b29f4a94c46652b20c4b7b6e5cf24e532af39 --------------------------------------------------------------- Help for output plugin..: HTTP output plugin --------------------------------------------------------------- The following parameters can be passed to this plugin: [-w | --www ]...........: folder that contains webpages in flat hierarchy (no subfolders) [-p | --port ]..........: TCP port for this HTTP server [-l ] --listen ]........: Listen on Hostname / IP [-c | --credentials ]...: ask for "username:password" on connect [-n | --nocommands ]....: disable execution of commands --------------------------------------------------------------- output_init() return value signals to exit

Unfortunately, the web server does not use the secure HTTPS protocol. Instead, everything is done with the HTTP protocol without encoding; even the password and the user name will be sent in plaintext. So enabling this feature does not really provide security. In other words, I would never open up port 8085 for access from outside the LAN.

Accessing the video stream from outside the local network through a VPN (as shown in the first section of this post) is secure. If this approach cannot be used, then setting up a secure reverse proxy is certainly possible with NGINX that is already installed on the Pi, if the first post in this guide has been followed.

What is a Reverse Proxy? Show

SSL Certificate

There is a prerequisite before the secure HTTPS protocol can be enabled in NGINX, or any other web server for that matter. An SSL certificate must be present on the Pi. This certificate can be "self-signed" or it can be a full fledged SSL certificate obtained from a certificate authority. If you want the latter, then look into obtaining a free certificate from the Let's Encrypt nonprofit authority. NGINX has a blog on the topic: Update: Using Free Let’s Encrypt SSL/TLS Certificates with NGINX.

If you have installed Domoticz, perhaps following the instructions in the first post in this guide, then a self-signed certificate is already available. Instead, one could use the self signed snake-oil certificate which the ssl-cert package installed by default on the Pi. Failing that one could always create one. The details of that will not be covered here, there are numerous guides on the Web.

Reverse Proxy

We have already modified the default nginx configuration file, default.conf, in the /etc/nginx/sites-available directory. That was set up to change the default HTTP TPC port used by HTTP to avoid a conflict with another server. Now a second configuration file will be added in the same directory.

nostra@damus:~ $ cd /etc/nginx/sites-available nostra@damus:/etc/nginx/sites-available $ sudo nano streamer-proxy.conf

Here is the content of the file. Notice that the SSL certificate supplied with Domoticz is being used.

server { listen 8085 ssl; ssl_certificate /home/nostra/domoticz/server_cert.pem; ssl_certificate_key /home/nostra/domoticz/server_cert.pem; location / { proxy_pass http://127.0.0.1:8085; } }

Replace the two ssl_certificate... entries with the snakeoil certificate or any other SSL certificate if desired or needed.

ssl_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem; ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;

A symbolic link to the newly created file must be created in the /etc/nginx/sites-available directory from which nginx reads its configuration.

nostra@damus:/etc/nginx/sites-available $ sudo ln -s /etc/nginx/sites-available/streamer-proxy.conf /etc/nginx/sites-enabled/

Restart the server and check that everything works.

nostra@damus:/etc/nginx/sites-available $ cd nostra@damus:~ $ sudo systemctl restart nginx.service nostra@damus:~ $ sudo systemctl status nginx.service nginx.service - A high performance web server and a reverse proxy server Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled) Active: active (running) since Sun 2021-10-15 12:38:20 ADT; 10s ago Docs: man:nginx(8) Process: 1072 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Process: 1073 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS) Main PID: 1074 (nginx) Tasks: 5 (limit: 1935) CGroup: /system.slice/nginx.service ├─1074 nginx: master process /usr/sbin/nginx -g daemon on; master_process on; ├─1075 nginx: worker process ├─1076 nginx: worker process ├─1077 nginx: worker process └─1078 nginx: worker process Oct 15 12:38:20 damus systemd[1]: Starting A high performance web server and a reverse proxy server... Oct 15 12:38:20 damus systemd[1]: Started A high performance web server and a reverse proxy server.

Just to test, open a web browser and go to the following site: https://damus.local:8085/?action=stream. Note that HTTPS is used, not HTTP. The first time this site is reached, the web browser will complain that the site is not secure unless a CA signed SSL certificate is used. To get access to the video stream,t will be necessary to create an exception. Do not worry, the problem is not with the security of the site per se, full TLS encryption will be used. The web browser will show the site as insecure because it does not recognize the authority of a self-signed certificate. I actually don't mind setting up the exception in the web browser. Furthermore that ominous warning by the web browser just might discourage someone with a little bit too much curiosity.

Outside Access

To have easy access to the video stream, other things must be put in place. I have already described the extra steps needed after setting up a server. The TCP port which is listened to by the reverse proxy must be forwarded to the proxy by the router. See port forwarding. And then see Creating a dynamic domain name, because it is much cleaner to use a dynamic host name instead of trying to find the public IP address of the local area network which I would imagine is next to impossible to do from outside the LAN.

Using a Raspberry Pi Camera Module toc

It is entirely possible to use a Raspberry Pi camera module plugged into the CSI connector of the Pi with a flat flex cable while one or indeed more than one webcam is connected to the Pi. The camera module will not work until its kernel driver is enabled. That is a one-time operation easily done with the Raspeberry Pi configuration utility.

nostra@damus:~ $ sudo raspi-config

Answer Yes when asked to confirm that the camera is to be enabled and when asked to reboot.

First let's test that the Raspberry Pi camera module is working by taking a snapshot. Of course, the module has to be connected with the Pi as explained above./p>

nostra@damus:~ $ which raspistill /usr/bin/raspistill nostra@damus:~ $ raspistill -o image.jpg total 2256 -rw-r--r-- 1 nostra nostra 2300813 Sep 25 19:46 image.jpg ...

Adding the camera module to Domoticz is quite simple and similar to adding the webcam.

The camera module will then appear alongside the webcam in the Cameras table.

It is possible to stream the video output from the camera module. Look up Application-specific Settings, raspivid in the Raspberry Pi documentation. As a starting point try raspivid -v -t 0 -l -o tcp://0.0.0.0:8087 and then read the stream at tcp://damus.local:8087 in a media player such as Celluloid (formerly GNOME MPV, installed by default in Mint 20.1) or VLC. It must be pointed out that both raspistill and raspivid are both based on the legacy Raspicam camera stack that may soon be deprecated.

It is also possible to stream using MJPG-Streamer using -i "input_raspicam.so -fps 10 -x 1280 -y 720" will give a video output equivalent to that obtained with the Logitech Webcam.

nostra@damus:~ $ mjpg_streamer -o "output_http.so -p 8087" -i "input_raspicam.so -fps 10 -x 1280 -y 720" MJPG Streamer Version: git rev: 310b29f4a94c46652b20c4b7b6e5cf24e532af39 i: fps.............: 5 i: resolution........: 640 x 480 i: camera parameters..............: Sharpness 0, Contrast 0, Brightness 50 Saturation 0, ISO 0, Video Stabilisation No, Exposure compensation 0 Exposure Mode 'auto', AWB Mode 'auto', Image Effect 'none' Metering Mode 'average', Colour Effect Enabled No with U = 128, V = 128 Rotation 0, hflip No, vflip No ROI x 0.000000, y 0.000000, w 1.000000 h 1.000000 o: www-folder-path......: disabled o: HTTP TCP port........: 8085 o: HTTP Listen Address..: (null) o: username:password....: disabled o: commands.............: enabled i: Starting Camera Encoder Buffer Size 81920

I find there's an annoying amount of stuttering and using the default dimensions of the camera modules (-x 1920 -y 1080) is better. I have a lot of experimenting to do yet.

Installing the Radicale Calendar and Contact Server toc

Radicale is a "Free and Open-Source CalDAV and CardDAV Server" that is surprisingly easy to install. It is a way of self-hosting calendars and contact lists instead of relying on email accounts. To be honest, testing out Radicale can be very easy, but I did run into problems when it came time to create a viable permanent installation. As usual, someone else solved that conundrum about a year ago: see bomben question How to install radicale (CardDAV & CalDAV) on a headless RaspberryPi? and answer. So here's my take on those instructions.

As bomben suggested, the system should be updated and upgraded. I was surprised at the time it took to do this at this point, as Raspberry Pi OS had been updated just a week or two before.

nostra@damus:~ $ sudo apt update && sudo apt upgrade -y ... The following packages were automatically installed and are no longer required: python3-atomicwrites python3-dateutil python3-radicale python3-tz python3-vobject Use 'sudo apt autoremove' to remove them. The following packages will be upgraded: base-files debconf debconf-i18n debconf-utils distro-info-data firmware-atheros firmware-brcm80211 firmware-libertas firmware-misc-nonfree firmware-realtek libatk-wrapper-java libatk-wrapper-java-jni libgssapi-krb5-2 libk5crypto3 libkrb5-3 libkrb5support0 libntfs-3g883 libraspberrypi-bin libraspberrypi-dev libraspberrypi-doc libraspberrypi0 libssl1.1 linux-libc-dev ntfs-3g openssl psmisc python3-debconf raspberrypi-bootloader raspberrypi-kernel raspberrypi-sys-mods rpi-eeprom syncthing tzdata 33 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Need to get 148 MB of archives. After this operation, 2,759 kB of additional disk space will be used. ... nostra@damus:~ $ sudo apt autoremove Reading package lists... Done Building dependency tree Reading state information... Done The following packages will be REMOVED: python3-atomicwrites python3-dateutil python3-radicale python3-tz python3-vobject 0 upgraded, 0 newly installed, 5 to remove and 0 not upgraded. After this operation, 953 kB disk space will be freed. Do you want to continue? [Y/n] y

While there is usually no harm in running apt autoremove, you may not need to do it. As can be seen, I had tried installing the radicale package, but I had run into some problems with it so it was removed. Radicale is a Python 3 script that will be installed with pip3 which is not installed by default in the Lite version of Raspberry Pi OS.

nostra@damus:~ $ apt-cache policy python3-pip python3-pip: Installed: (none) Candidate: 18.1-5+rpt1 Version table: 18.1-5+rpt1 500 500 http://archive.raspberrypi.org/debian buster/main armhf Packages 18.1-5 500 500 http://raspbian.raspberrypi.org/raspbian buster/main armhf Packages nostra@damus:~ $ sudo apt install python3-pip Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: gir1.2-glib-2.0 libgirepository-1.0-1 python3-asn1crypto python3-cffi-backend python3-crypto python3-cryptography python3-dbus python3-entrypoints python3-gi python3-keyring python3-keyrings.alt python3-secretstorage python3-setuptools python3-wheel python3-xdg Suggested packages: python-crypto-doc python-cryptography-doc python3-cryptography-vectors python-dbus-doc python3-dbus-dbg gnome-keyring libkf5wallet-bin gir1.2-gnomekeyring-1.0 python-secretstorage-doc python-setuptools-doc The following NEW packages will be installed: gir1.2-glib-2.0 libgirepository-1.0-1 python3-asn1crypto python3-cffi-backend python3-crypto python3-cryptography python3-dbus python3-entrypoints python3-gi python3-keyring python3-keyrings.alt python3-pip python3-secretstorage python3-setuptools python3-wheel python3-xdg 0 upgraded, 16 newly installed, 0 to remove and 0 not upgraded. Need to get 1,674 kB of archives. After this operation, 8,498 kB of additional disk space will be used.

The Radicale script can now be installed.

nostra@damus:~ $ sudo python3 -m pip install --upgrade radicale Looking in indexes: https://pypi.org/simple, https://www.piwheels.org/simple Collecting radicale Installing collected packages: python-dateutil, defusedxml, passlib, vobject, radicale Successfully installed defusedxml-0.7.1 passlib-1.7.4 python-dateutil-2.8.2 radicale-3.0.6 vobject-0.9.6.1

Let's test the server by manually starting it.

nostra@damus:~ $ python3 -m radicale --config "" --server-hosts 0.0.0.0:5232 --storage-filesystem-folder=~/.var/lib/radicale/collections

This is a server, so do not expect to see anything. The server will run until interrupted (with CtrlC) or any other method. Wait until the test is completed before interrupting it.

Some explanations about this command might be in order.

So just open a web browser on a machine on the same local area network as the Raspberry Pi and open the Radicale login page at damus.local:5232. A new user can be created by entering a user name and a password in that login screen. You can then create or import an address book ar calendar in the next page entitled Collections. Once you have finished testing the server, get rid of the directory.

nostra@damus:~ $ rm -r .var

Let's go on with the "permanent" installation of the server. It will be run by a user called radicale which does not have a login for security reasons. That new user will be a member of a group also named radicale.

nostra@damus:~ $ sudo useradd --system --home-dir / --shell /sbin/nologin radicale checking: nostra@damus:~ $ cat /etc/passwd | grep radicale radicale:x:999:995::/:/sbin/nologin nostra@damus:~ $ cat /etc/group | grep radicale radicale:x:995:

A single command can reverse this. Note how the empty radicale group is automatically removed.

nostra@damus:~ $ sudo deluser radicale Removing user `radicale' ... Warning: group `radicale' has no more members. Done. checking: nostra@damus:~ $ cat /etc/passwd | grep radicale nostra@damus:~ $ cat /etc/group | grep radicale nostra@damus:~ $

Now lets create a directory belonging to radicale to hold the calendars and contacts.

nostra@damus:~ $ sudo mkdir -p /var/lib/radicale/collections nostra@damus:~ $ sudo chown -R radicale:radicale /var/lib/radicale/collections nostra@damus:~ $ sudo chmod -R o= /var/lib/radicale/collections checking: nostra@damus:~ $ sudo ls -l /var/lib/radicale/collections total 4 drwxr-x--- 3 radicale radicale 4096 Oct 12 23:12 collection-root

Let's create a minimal configuration file. I am using the nano editor as usual.

nostra@damus:~ $ sudo mkdir /etc/radicale nostra@damus:~ $ sudo nano /etc/radicale/config

Since IPv6 is disabled on the Pi, the server is bound to only IPv4 addresses, but you may want to bind to all addresses if IPv6 is enabled. And we point the

[server] # Bind all addresses #hosts = 0.0.0.0:5232, [::]:5232 # Bind all IPv4 addresses hosts = 0.0.0.0:5232 [storage] filesystem_folder = /var/lib/radicale/collections

I think it may not be necessary to specify the filesystem_folder given the systemd unit file about to be created. However it causes no harm and it will allow for manual launching of the server with the command sudo python -m radicale. Time to create the unit file, using nano once again.

nostra@damus:~ $ sudo nano /etc/systemd/system/radicale.service

Here is the file.

[Unit] Description=A simple CalDAV (calendar) and CardDAV (contact) server After=network.target Requires=network.target [Service] ExecStart=python3 -m radicale Restart=on-failure User=radicale # Deny other users access to the calendar data UMask=0027 # Optional security settings PrivateTmp=true ProtectSystem=strict ProtectHome=true PrivateDevices=true ProtectKernelTunables=true ProtectKernelModules=true ProtectControlGroups=true NoNewPrivileges=true ReadWritePaths=/var/lib/radicale/collections [Install] WantedBy=multi-user.target

Let's enable the service so that it will automatically be started whenever systemd initializes the system.

nostra@damus:~ $ sudo systemctl enable radicale Created symlink /etc/systemd/system/multi-user.target.wants/radicale.service → /etc/systemd/system/radicale.service.

We can manually start the server and check its status.

nostra@damus:~ $ sudo systemctl start radicale nostra@damus:~ $ sudo systemctl status radicale ● radicale.service - A simple CalDAV (calendar) and CardDAV (contact) server Loaded: loaded (/etc/systemd/system/radicale.service; enabled; vendor preset: enabled) Active: active (running) since Tue 2021-10-12 23:10:20 ADT; 4s ago Main PID: 7042 (python3) Tasks: 1 (limit: 1935) CGroup: /system.slice/radicale.service └─7042 python3 -m radicale Oct 12 23:10:20 damus systemd[1]: Started A simple CalDAV (calendar) and CardDAV (contact) server.

If you compare this long-winded description of the installation process to its source, it will be clear that all the instructions pertaining to the encrypted passwords have been removed. Again, this server is meant to be run behind a firewall and the only remote access to it will be done through WireGuard which will be encrypt all traffic end to end.

Installing the chrony Network Time Client and Server toc

All Tasmota devices are set up to use a remote network time server (using SNTP) to obtain the time. Having all these devices synchronizing with an outside server is a problem given the objective to have a home automation system that is running locally. Consequently, I want to make the Raspberry Pi act as an SNTP server. The Pi will have a hardware clock connected to it, so that even when access to the Internet is lost, it can be a reasonable accurate time server. Certainly accurate enough to turn lights on and off automatically. Adding the hardware clock will be left to the next part of this guide, for the remainder of this section, there will be no RTC connected to the Pi.

The default time synchronization service on Raspberry Pi OS, systemd-timesyncd, is stricly an NTP client and cannot be used as the NTP server. But it does work nicely with some servers. Here is part of the configuration file of the service.

nostra@damus:~ $ cat /usr/lib/systemd/system/systemd-timesyncd.service.d/disable-with-time-daemon.conf [Unit] ConditionFileIsExecutable=!/usr/sbin/ntpd ConditionFileIsExecutable=!/usr/sbin/openntpd ConditionFileIsExecutable=!/usr/sbin/chronyd ConditionFileIsExecutable=!/usr/sbin/VBoxService

Because VBoxService looks very much like something that would be used in a Virtual Box virtual machine only, it seems that should any one of the other three NTP servers be installed, systemd-timesyncd will not start. The reference server is surely ntpd by the Ntp Project, while OpenNTPD is the lightweight contender from OpenBSD but ported to Linux. I have seen chrony included in other Linux distribution. Looking at its features (see Comparison of NTP implementations done by chrony itself), it seems to be a good choice.

NTP Client toc

The chrony version 3.4 package available in the repository is three years out of date, something that is not unusual for Debian based releases. Nothwithstanding that defect, I installed that older version.

nestor@domus:~ $ sudo apt install chrony Reading package lists... Done ... Setting up chrony (3.4-4+deb10u1) ... Creating '_chrony' system user/group for the chronyd daemon… Creating config file /etc/chrony/chrony.conf with new version Creating config file /etc/chrony/chrony.keys with new version Created symlink /etc/systemd/system/chronyd.service → /lib/systemd/system/chrony.service. Created symlink /etc/systemd/system/multi-user.target.wants/chrony.service → /lib/systemd/system/chrony.service. ...

I decided to reboot, and check the situation with the new NTP package. As expected, the systemd-timesyncd.service is no longer active and the new service is operational. The systemd time control utility timedatectl is a bit confused, but the chronyc control utility should be used instead.

nestor@domus:~ $ sudo systemctl status systemd-timesyncd.service Warning: The unit file, source configuration file or drop-ins of systemd-timesyncd.service changed on disk. Run 'systemctl daemon-reload' to reload units. ● systemd-timesyncd.service - Network Time Synchronization Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled) Drop-In: /usr/lib/systemd/system/systemd-timesyncd.service.d └─disable-with-time-daemon.conf Active: inactive (dead) Condition: start condition failed at Tue 2021-10-19 14:46:19 ADT; 1min 16s ago └─ ConditionFileIsExecutable=!/usr/sbin/chronyd was not met Docs: man:systemd-timesyncd.service(8) Oct 19 14:46:19 domus systemd[1]: Condition check resulted in Network Time Synchronization being skipped. nestor@domus:~ $ sudo systemctl status chrony.service ● chrony.service - chrony, an NTP client/server Loaded: loaded (/lib/systemd/system/chrony.service; enabled; vendor preset: enabled) Active: active (running) since Sun 2020-04-12 19:52:55 ADT; 1 years 6 months ago Docs: man:chronyd(8) man:chronyc(1) man:chrony.conf(5) Process: 560 ExecStart=/usr/sbin/chronyd $DAEMON_OPTS (code=exited, status=0/SUCCESS) Process: 582 ExecStartPost=/usr/lib/chrony/chrony-helper update-daemon (code=exited, status=0/SUCCESS) Main PID: 569 (chronyd) Tasks: 2 (limit: 1935) CGroup: /system.slice/chrony.service ├─569 /usr/sbin/chronyd -F -1 └─580 /usr/sbin/chronyd -F -1 Apr 12 19:52:55 domus systemd[1]: Starting chrony, an NTP client/server... Apr 12 19:52:55 domus chronyd[569]: chronyd version 3.4 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +SCFILTER +SIGND +A Apr 12 19:52:55 domus chronyd[569]: Frequency 12.333 +/- 8.264 ppm read from /var/lib/chrony/chrony.drift Apr 12 19:52:55 domus chronyd[569]: Loaded seccomp filter Apr 12 19:52:55 domus systemd[1]: Started chrony, an NTP client/server. Apr 12 19:53:01 domus chronyd[569]: Selected source 45.33.84.208 Apr 12 19:53:01 domus chronyd[569]: System clock wrong by 47933189.739940 seconds, adjustment started Oct 19 14:39:31 domus chronyd[569]: System clock was stepped by 47933189.739940 seconds nestor@domus:~ $ date Tue 19 Oct 14:41:33 ADT 2021 nestor@domus:~ $ timedatectl -a Local time: Tue 2021-10-19 14:42:55 ADT Universal time: Tue 2021-10-19 17:42:55 UTC RTC time: n/a Time zone: America/Halifax (ADT, -0300) System clock synchronized: yes NTP service: inactive RTC in local TZ: no nestor@domus:~ $ chronyc tracking Reference ID : 4A06A849 (t2.time.gq1.yahoo.com) Stratum : 3 Ref time (UTC) : Tue Oct 19 17:50:09 2021 System time : 0.006536143 seconds fast of NTP time Last offset : -0.000269975 seconds RMS offset : 0.012076506 seconds Frequency : 13.057 ppm fast Residual freq : +0.958 ppm Skew : 26.224 ppm Root delay : 0.110977933 seconds Root dispersion : 0.010177141 seconds Update interval : 64.6 seconds Leap status : Normal nestor@domus:~ $ chronyc sources -a 210 Number of sources = 4 MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^+ clock.fmt.he.net 1 6 377 33 -9448us[-9448us] +/- 54ms ^+ 38.229.53.9 2 6 377 33 +41ms[ +41ms] +/- 122ms ^* t2.time.gq1.yahoo.com 2 6 377 35 -10ms[ -10ms] +/- 66ms ^+ t1.time.bf1.yahoo.com 2 6 377 35 -4094us[-4094us] +/- 61ms

There is an unnecessary Run 'systemctl daemon-reload' to reload units warning from the systemd-timesyncd.service which can be safely ignored. The source servers used by chrony were shown in that last command for later comparison when the NTP server pool will be changed.

From past experience I know that power failures accompanied by longer loss of Internet access can play havoc with the time on the home automation system. In those particular instances, it is worthwhile to set the time manually in those circumstances. When this is done, chrony will update the system clock very slowly (like OpenNTP and other clients perhaps). I tend to be impatient and thankfully, the service can be configured to update the clock quickly. As usual, that is done by editing a configuration file.

nestor@domus:~ $ sudo nano /etc/chrony/chrony.conf

At the end of the file, change makestep 1 3 to makestep 1 -1

... # Step the system clock instead of slewing it if the adjustment is larger than # one second, but only in the first three clock updates. #makestep 1 3 # On a computer without an RTC it might be necessary to allow the step on any clock update. # https://chrony.tuxfamily.org/faq.html#_is_chronyd_allowed_to_step_the_system_clock makestep 1 -1

While editing the configuration file, I also changed to the NTP server pool at the very beginning of the file to the Canadian pool that is ostensibly closer.

# Welcome to the chrony configuration file. See chrony.conf(5) for more # information about usuable directives. #pool 2.debian.pool.ntp.org iburst pool 2.ca.pool.ntp.org iburst ...

I then restarted the system and set the date and time manually to test that it worked. Note that it was necessary to enable the settime functionality before using it.

nestor@domus:~ $ sudo systemctl restart chrony.service nestor@domus:~ $ sudo chronyc manual on 200 OK nestor@domus:~ $ date; sudo chronyc settime 2020-12-27 18:12; date Tue 19 Oct 15:16:42 ADT 2021 200 OK Clock was 25560282.00 seconds fast. Frequency change = 1000000.00ppm, new frequency = 500000.00ppm Sun 27 Dec 18:12:00 AST 2020 nestor@domus:~ $ date Sun 27 Dec 18:12:04 AST 2020 ... nestor@domus:~ $ date Sun 27 Dec 18:12:15 AST 2020 ... nestor@domus:~ $ date Tue 19 Oct 15:17:33 ADT 2021

It was possible to set the date back a few months, meaning that manual adjustment of the time will be possible when access to the Internet is lost again. With a working Internet connection, it did not take long before the NTP client updated the system date and time to the correct value as can be seen. But again, this is because of the change in the makestep values. With the default values, the adjustment would have been stretched over a long period.

nestor@domus:~ $ chronyc sources -a 210 Number of sources = 4 MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^* rocinante.baxterit.net 2 6 377 1 -488us[ -730us] +/- 52ms ^+ rwhois.dargalsolutions.c> 2 6 377 1 +505us[ +505us] +/- 76ms ^+ up2.com 2 6 377 2 -13ms[ -14ms] +/- 111ms ^+ bras-base-toroon2638w-gr> 2 6 377 2 -1499us[-1742us] +/- 63ms

Changing the source did not bring about substantial differences as far as I can make out.

nestor@domus:~ $ sudo systemctl restart chrony.service

If you check the status there should be no error.

NTP Server toc

Bringing up the NTP server is remarkably easy as it comes down to a one-line addition in the configuration file.

nestor@domus:~ $ sudo nano /etc/chrony/chrony.conf

# setup NTP server listening to all IPv4 request from the LAN allow 192.168.1.0/24

Restart the service for the change to take effect.

Tasmota NTP Settings toc

When the Tasmota firmware boots, it sets its real-time clock at 00:00:00. It will then try to get the correct time from an NTP server, but this will be a hit-or-miss proposition given the default settings. It goes without saying that the Tasmota firmware needs the correct real time to trigger timers correctly. The time stamp of entries shown in the web interface console also uses the real-time clock value. Now that we have a local NTP server, we will attempt to use it with the Wi-Fi wall plug running Tasmota firmware already installed in the test home automation system.

The firmware stores up to 3 NTP server host names or IP addresses named ntpserverx (x=1,2,3). When Tasmota wants to know the time, it tries each URL in order stopping when it gets the current time.

[...] the time sync request is forced to NTPSERVER1. If can't connect, it tries NTPSERVER2. And finally NTPSERVER3. Ensure that these parameters are set appropriately and that the device can reach at least one of these time servers.
(Source: Timers trigger at the wrong time)

To start the test, I removed the default URL names and restarted the wall plug thus ensuring that the device had no valid time. This could easily be done in the web interface console of the device, but I chose to use my own MQTT client lazmqttc.

TX: [cmnd/tasmotas/NtpServer] - RX: [stat/Test/RESULT] - {"NtpServer1":"pool.ntp.org","NtpServer2":"nl.pool.ntp.org","NtpServer3":"0.nl.pool.ntp.org"} TX: [cmnd/tasmotas/NtpServer1] - 0 RX: [stat/Test/RESULT] - {"NtpServer1":""} TX: [cmnd/tasmotas/NtpServer2] - 0 RX: [stat/Test/RESULT] - {"NtpServer2":""} TX: [cmnd/tasmotas/NtpServer3] - 0 RX: [stat/Test/RESULT] - {"NtpServer3":""}

Be careful, do not use the tasmotas topic on a running home automation system, replace it with the topic of a single Tasmota module until everything is thoroughly tested. The point is moot in our example installation since there is only one device in place.

The Tasmota real time clock will not work properly until the time zone is set up correctly. The geographical coordinates of the device should also be set since they are needed to calculate sunset and sunrise times used in timers. Here is the pertinent information for my time zone which is AT (Atlantic Time). The offset between local time and UTC (coordinated universal time) depends on the date because we advance the clock by one hour in the winter.

Here is the information about the Tasmota commands for setting for the start of the standard and daylight times.

    TimeStd H,W,M,D,h,T

    TimeDST H,W,M,D,h,T

        H = hemisphere (0 = northern hemisphere / 1 = southern hemisphere)
        W = week (0 = last week of month, 1..4 = first .. fourth)
        M = month (1..12)
        D = day of week (1..7 1 = sunday 7 = saturday)
        h = hour (0..23)
        T = timezone (-780..780) (offset from UTC in MINUTES - 780min / 60min=13hrs)

The proper Tasmota commands for the Atlantic time zone are as follows.

    TimeStd 0,1,11,1,2,-240
        H = O    : north
        W = 1    : first week
        M = 11   : November
        D = 1    : Sunday
        h = 2    : 02:00 (2 AM)
        T = -240 : AST is UTC-4 hours

    TimeDst 0,2,3,1,2,-180
        H = O    : north
        W = 2    : first week
        M = 3    : March
        D = 1    : Sunday
        h = 2    : 02:00 (2 AM)
        T = -180 : ADT is UTC-3 hours

Tasmota has to be told to use the settings for standard and daylight time periods with the Timezone 99 command and also it is best to make sure that NTP is enabled (Time 0) even if it is the default value. With all that I was able to set up a single Backlog command to take care of everything at once.

Backlog TimeStd 0,1,11,1,2,-240; TimeDst 0,2,3,1,2,-180; Timezone 99; NtpServer1 192.168.1.88; NtpServer2 2.ca.pool.ntp.org; latitude 46.107174; longitude -64.806497; time 0;

Instead of the IP address of the Raspberry Pi, its local host name damus.local could be entered as the first NTP server (NtpServer1 damus.local;) in the Backlog command.

Here is the exchange of MQTT messages as recorded in lazmqttc.

TX: [cmnd/tasmotas/backlog] - TimeStd 0,1,11,1,2,-240; TimeDst 0,2,3,1,2,-180; Timezone 99; NtpServer1 damus.local; NtpServer2 latitude 46.107174; longitude -64.806497; time 0; RX: [stat/Test/RESULT] - {"TimeStd":{"Hemisphere":0,"Week":1,"Month":11,"Day":1,"Hour":2,"Offset":-240}} RX: [stat/Test/RESULT] - {"TimeDst":{"Hemisphere":0,"Week":2,"Month":3,"Day":1,"Hour":2,"Offset":-180}} RX: [stat/Test/RESULT] - {"Timezone":99} RX: [stat/Test/RESULT] - {"NtpServer1":"domus.local"} RX: [stat/Test/RESULT] - {"NtpServer2":"2.ca.pool.ntp.org"} RX: [stat/Test/RESULT] - {"Latitude":46.107170} RX: [stat/Test/RESULT] - {"Longitude":-64.806496} RX: [stat/Test/RESULT] - {"Time":"1970-01-01T00:01:15","Epoch":75}

And here is what was displayed on the wall plug console. At the start, the time is 00:00:00 because I had just restarted the device after clearing the ntpserver<x> settings. After about a minute and 14 seconds, the backlog message was sent with the MQTT client and echoed as stat messages to the MQTT broker (as seen above) and shown on the console (as shown below). After a minute or so, I entered a time command in the console and the result was the correct local time.

00:00:00 CFG: Loaded from flash at F7, Count 117 00:00:00 QPC: Count 1 00:00:00 Project tasmota Test Version 9.1.0(tasmota)-2_7_4_5 00:00:00 WIF: Connecting to AP1 COROBRUN-2 Channel 11 BSSId 00:FC:8D:4F:71:B8 in mode 11N as Test... 00:00:01 WIF: Connected 00:00:02 DNS: Initialized 00:00:02 HTP: Web server active on Test.local with IP address 192.168.1.104 00:00:04 DNS: Query done. MQTT services found 0 00:00:04 MQT: Attempting connection... 00:00:04 MQT: Connected 00:00:04 MQT: tele/Test/LWT = Online (retained) 00:00:04 MQT: cmnd/Test/POWER = 00:00:04 MQT: tele/Test/INFO1 = {"Module":"CE LA-WF3","Version":"9.1.0(tasmota)","FallbackTopic":"cmnd/DVES_F14117_fb/","GroupTopic":"cmnd/tasmotas/"} 00:00:04 MQT: tele/Test/INFO2 = {"WebServerMode":"Admin","Hostname":"Test","IPAddress":"192.168.1.104"} 00:00:04 MQT: tele/Test/INFO3 = {"RestartReason":"Software/System restart"} 00:00:04 MQT: stat/Test/RESULT = {"POWER":"ON"} 00:00:04 MQT: stat/Test/POWER = ON 00:00:04 MQT: domoticz/in = {"idx":1,"nvalue":1,"svalue":"","Battery":100,"RSSI":9} 00:00:06 QPC: Reset 00:00:07 MQT: tele/Test/STATE = {"Time":"1970-01-01T00:00:07","Uptime":"0T00:00:09... 00:01:14 MQT: stat/Test/RESULT = {"TimeStd":{"Hemisphere":0,"Week":1,"Month":11,"Day":1,"Hour":2,"Offset":-240}} 00:01:14 MQT: stat/Test/RESULT = {"TimeDst":{"Hemisphere":0,"Week":2,"Month":3,"Day":1,"Hour":2,"Offset":-180}} 00:01:14 MQT: stat/Test/RESULT = {"Timezone":99} 00:01:14 MQT: stat/Test/RESULT = {"NtpServer1":"domus.local"} 00:01:14 MQT: stat/Test/RESULT = {"NtpServer2":"2.ca.pool.ntp.org"} 00:01:15 MQT: stat/Test/RESULT = {"Latitude":46.107170} 00:01:15 MQT: stat/Test/RESULT = {"Longitude":-64.806496} 00:01:15 MQT: stat/Test/RESULT = {"Time":"1970-01-01T00:01:15","Epoch":75} wait a while until Tasmota obtains the time from the chrony NTP server on the Pi 19:41:44 CMD: time 19:41:44 MQT: stat/Test/RESULT = {"Time":"2021-10-19T19:41:44"}

It works. Given that success, I installed chrony on the "production" home automation system and sent the backlog MQTT message to all the tasmotas devices. I checked that they all got the current time from the NTP server.

~ $ sudo chronyc clients Hostname NTP Drop Int IntL Last Cmd Drop Int Last =============================================================================== 192.168.1.100 2 0 12 - 20m 0 0 - - 192.168.1.126 2 0 10 - 20m 0 0 - - 192.168.1.122 2 0 10 - 20m 0 0 - - 192.168.1.124 2 0 10 - 20m 0 0 - - 192.168.1.120 2 0 10 - 20m 0 0 - - 192.168.1.118 2 0 10 - 20m 0 0 - - 192.168.1.114 2 0 10 - 20m 0 0 - - 192.168.1.116 2 0 10 - 20m 0 0 - - 192.168.1.108 2 0 10 - 20m 0 0 - - ...

This did not change anything substantial since none of the devices used timers, but at least the entries in their console screen will have meaningful time stamps and my Tasmota devices will no longer be unnecessarily polling NTP servers on the Internet.

Setting up a Raspberry Pi as a Headless Computer (September 2021) (back to part I)-> <-Home Automation System on a Raspberry Pi (part II)