It was time to replace the venerable Raspberry Pi (Model B Plus Rev 1.2) that has been the main workhorse of our home automation system for over a year. A Raspberry Pi 3 B will be used from now on. The old Model B+ will replace an even older Model A that was also hosting a home automation system. This is the first post recording the details of the installation of a home automation system based built around Domoticz on both of these Raspberry Pi models with Raspbian Buster Lite as the operating system.

The installation is described over three posts. The first one is about installing and updating the operating system. The second is about installing the most important servers that I use in my home automation system. The third is about peripherals added to the Raspberry Pi.

I have done this before, but with older versions of the operating system and the home automation software. Some things have changed and hopefully I have learned a few things since those earlier installations.

This series of posts could be of interest mostly to new fans of small single-board computers, in this case the Raspberry Pi. Assuming that many of these would be new to Linux, I have added sidebars entitled Information for neophytes in which I ramble about what that I think could be of help to beginners.

Table of Contents

1. New To Linux
2. Already Have Raspbian Running?
   1. Saving User Scripts and Environment
   2. Saving Python Scripts and Environment
   3. Saving Cron Data
   4. Saving Domoticz Data
   5. Saving Web Server Data
   6. Saving WireGuard Data
   7. Saving Packages with apt_clone
3. Getting Buster on an SD card
4. Start with Monitor and Keyboard
5. Headless Start the Easy Way
6. Headless Start the Hard Way
   1. Enable ssh
   2. Enable Wi-Fi (optional)
   3. Set Static IP Addresses (optional)
   4. Initial Boot
7. Basic Configuration
8. Change the Default User Name and Hostname
9. Updating the Operating System
10. Syslog
11. Working Directories

New to Linux

GNU/Linux or simply Linux is a mature open source Unix-like operating system which can run on numerous hardware platforms including Intel, AMD and ARM 32 and 64-bit processors. Many, including myself, see Linux distributions such as Debian and Manjaro (and their many variants: Ubuntu, Mint and so on) as viable alternatives to the ubiquitous Windows and Mac OS commercial operating systems. Right now, you are probably using one of these systems to read this. In all of these GUI systems, most everything can be done with the usual point and click mouse operations with the occasional data entry into text windows. To be fair, some arcane settings will require "dropping down" into a terminal or console (emulating a physical computer terminal) to issue commands at a prompt as in the "old days".

The CPM/80 ">" prompt, when pip was not a Python utility!

The Windows 10 cmd ">" prompt.

Mac OS terminal "$" prompt (on a borrowed Mac).

The Ubuntu (Linux) terminal "$" prompt (on my desktop).

Raspbian is a Debian based Linux distribution for the Raspberry Pi. It is probably the most commonly used distribution on this family of single board computers. Raspbian comes in three flavours, two with a GUI desktop similar to what I have on my Ubuntu desktop machine and Raspbian Lite which does not have a graphic user interface. It is suitable for machines that are to be used for servers that are not used for personal tasks like sending emails or looking at YouTube videos. This is the version that I installed on the Raspberry Pi that will host the home automation system. Since there is no GUI, setting up the system has to be done with text commands.

The Raspbian LXTerminal "$" prompt (on Raspbian Desktop).

I cannot go into a lengthy tutorial about Linux commands. This post is already too long, but the real reason there will be no detailed discussion of Linux command is that I am not knowledgeable enough to undertake the task. Instead I will defer to two sites with lists of the most important commands, which will prove to be a good starting point for the newcomer.

• Linux commands by the Raspberry Foundation.
• 57 Raspberry Pi commands that everyone should know by Patrick at RaspberryTips

Do not worry, there is no need to memorize 57 commands before starting. Here is the list of Linux commands found in this post, which is far from using all those commands listed above.

Be careful with some of these commands. I have lost work when I carelessly typed a command such as reboot without looking at the prompt. Instead of shutting down a Raspberry Pi, I entered the command in a Ubuntu terminal with the obvious consequence that my desktop computer became very quiet very quickly. You have been warned.

Already Have Raspbian Running?

If Raspbian is already running on the Raspberry Pi, then it would be a good idea to back up configuration files, images, scripts and so on for the major programs that will be reinstalled later.

If you have a Linux-based desktop computer, then the SD card can be kept and placed in a reader. It will then be possible to access everything in the old system. Better yet, make a backup and mount the image, leaving the SD card in a safe place.

If you are upgrading to Buster from an older version of Raspbian on the same Rapsberry Pi, you could try in-situ upgrading. If it works then in principle, the configuration for all the installed packages will be in place. While the Raspberry Pi Foundation does give instructions on doing this, it gives the following warning.

We do not recommend upgrading an existing Stretch (or earlier) system to Buster – we can’t know what changes everyone has made to their system, and so have no idea what may break when you move to Buster. However, we have tested the following procedure for upgrading, and it works on a clean version of the last Stretch image we released. That does not guarantee it will work on your system, and we cannot provide support (or be held responsible) for any problems that arise if you try it. You have been warned – make a backup!

Source: Buster – the new version of Raspbian

Past experience tells me that I cannot take the SD card from the older Raspberry Pi, stick it into the SD slot of the newer Pi and then try to update the operating system. So this is a tabula rasa install of Buster for the most part. Nevertheless, it is quite useful to save some configuration files, scripts, etc. for re-use in the new system or at the minimum for comparison with the new system.

Saving User Scripts and Environment

As shown near the end of this post, I created a hidden local directory in which I saved useful Bash scripts. In addition two hidden files that define the user environment were modified. All this is saved for future reference.

Saving Python Scripts and Environment

It's almost as if I had planned it; the Python scripts that are used by the home automation system are in a Python virtual environment named syspy . It is very simple matter to create a list of Python packages added to the virtual environment to be able to recreate it elsewhere.

I saved the following files to recreate the virtual environment later on.

Saving Cron Data

Here's another area where fixes and tweaks have been accumulated and which are easily forgotten. The list of user defined scheduled tasks, called cron jobs in Linux lingo, can be redirected into a text file to be saved for later reference.

Jobs created as root can be saved in a similar manner.

There were no root cron jobs on my original Raspberry Pi.

Saving Domoticz Data

Here is the data that I saved. The last two are only useful if switch icons were added manually. If they were uploaded using the Custom Icons in the Setup/More Options menu, then they were saved in the database. Even then it might be worthwhile to save the uploaded images. They can be identified by the file owner who is root.

Saving Web Server Data

The Web server is used for only one purpose: serve binary files for over-the-air updates of home automation wireless devices. There is one subdirectory per device type in the /var/www/html directory. I saved everything in the html directory.

Files saved:

Saving the nginx configuration file is almost more work than editing the file when configuring the server as it involved only the addition of a single line.

Saving WireGuard Data

I saved the WireGuard configuration file wg0.conf in the /etc/wireguard directory. On some other systems there could be more than one file and all should be saved.

I also saved the content of the /home/pi/wg_config directory. That the script and information about all the peers that connect to the virtual network server on the system used to create the configuration file.

Saving Packages with apt_clone

Michael Vogt (mvo5) has created a package called apt-clone that saves a list of all installed packages on Debian based systems including Raspbian. Later the packages can be easily re installed in one go after installing the operating system and apt-clone itself. It is not unlike the use pip freeze to preserve the Python environment for later use. More information can be found in a post by Oltjano Terpollari: How to Clone Packages in Debian Derived Systems and elsewhere on the Web.

This will be a first-time use of this package. First I installed it, created a hidden directory ~/.clones and had apt-clone create the list of installed packages in that directory.

I have no experience with using apt-clone when moving to a newer version of Debian but it does work when cloning a current version of the OS.

Getting Buster on an SD card

The home automation system cannot be offline while I am experimenting with updates. Instead everything is being done on a Raspberry Pi Model 3 B. Once that new system is running, I hope to do the same thing on the old Model B which I have found more than adequate as a home automation server.

As usual, I downloaded the latest Lite version of Raspbian:

  Raspbian Buster Lite
  Minimal image based on Debian Buster
  Version:            September 2019
  Release date:       2019-09-26
  Kernel version:     4.19
  Size:               435 MB

from https://www.raspberrypi.org/downloads/raspbian/. Of course you can download and install a desktop version of Raspbian and it will be possible to do everything described in these posts.

I followed the instructions to copy the image onto a micro-SD card in Installing operating system images. Installing the current version of balenaEtcher in a Linux system is quite simple. Extract the application, balenaEtcher-1.5.59-x64.AppImage from the downloaded archive and copy it to ~/.local/bin (it could just about anywhere in the home directory). Delete any old application desktop file. Lately, these are named appimagekit-balena-etcher-electron.desktop and are stored in the /home/michel/.local/share/applications/ directory. Start the application and let it install a desktop file. From then on, balenaEtcher can be started in the usual way. I recommend checking the application settings (clig on the cog wheel) and disable the Auto-unmount on success option.

As before, it was a breeze to copy the operating system image onto the SD card with the application. It is not even necessary to decompress the downloaded archive containing the image file.

All these file management chores, copying, deleting, extracting from archives and so on, can all be done with "point and click" ease using Double Commander. It is one of the first utilities I install in all the Linux distributions I use. It even works in Windows, macOS and FreeBSD. Highly recommended. Furthermore, it is written in Free Pascal/Lazarus, my favourite programming language.

Start with Monitor and Keyboard

By far the easiest way to boot and configure Raspbian on a Raspberry Pi is with a direct connection to the Linux console using a monitor and keyboard. A mouse can also be connected, but it is not needed as Buster Lite does not have a graphical interface. If it is possible to connect the Pi to the local area network with an Ethernet cable, then updating the system and installing packages will be very simple indeed.

Put in the SD card with the Debian image and turn the machine on. Expect lots of things to scroll by the screen and after a while enter the default password when prompted. Do not worry if nothing shows up as you type, that's a safety feature.

Once you have typed in the invisible password, press the Enter key to tell Linux that you have finished your bit. It will read what you have typed in and either complain that the password is wrong or it will welcome you and show the '$' prompt.

If this is the first time that you are setting up a Raspberry Pi, I would strongly suggest that you follow what preceded and not bother with the "headless" stuff in the following sections. Later on when you see an ssh command given from a Linux desktop, or a Windows machine:

it means that I am connecting to a headless Pi over the local area network. Ignore these commands if you have a direct console connection and skip to the Basic Configuration section.

Of course if an extra keyboard and monitor are not available, then you will have to find another way to reach a headless Raspberry Pi. There is an easy way to open a session with a Raspberry Pi that is not connected to a keyboard and monitor, and there is a hard way. Both are described below.

Headless Start the Easy Way

If you have a USB to Serial adapter that operates on 3.3V and you know how to use a terminal program such as Screen, tmux or C-Kermit, then it is relatively simple to connect to a Rapsberry Pi. Here is the pinout.

If the Raspberry Pi is an older model without built-in Bluetooth, then there is no further preparation needed. If the Raspberry Pi is a a newer model, then the config.txt file found in the boot partition of the micro SD card containing the Raspbian image has to be modified. Assuming the SD card is still in the desktop computer reader, its boot partition is should be seen as a drive connected to the system. If not, remove the card from the reader, wait awhile and insert it in the reader again. The partition should be found by the system because its file system is the ubiquitous FAT32 which most operating system can read.

Add the following lines near the end of the file.

List the serial devices on the system before connecting the USB to serial (tty) adapter to the desktop computer

Now connect the serial end of the the USB to Serial adapter to the Raspberry Pi which should not be under power and then connect it to the desktop USB port. List serial devices to find the newly added adapter, start up the terminal program connecting to that port and then apply power to the Raspberry Pi. You should then see the Linux boot messages which end with a login prompt.

If using Screen the terminal should be launched with the following command on the desktop.

If you are here, then ignore the ssh commands such as

that you may see later on in this post unless you have connected to the Raspberry Pi over the local area network, which is not necessary when there is a direct serial connection with the Pi. Skip ahead to the Basic Configuration section.

Headless Start the Hard Way

The Raspberry Pi is used as a headless server with no monitor and no keyboard. I am rather lazy find a hassle to dig out a monitor and keyboard just for the initial configuration of such a system. And my USB to serial adapters have a way of disappearing. So this is my preferred method to set up a Raspberry Pi, but not the one I recommend.

Since connectivity with the local network will be by Wi-Fi when the home automation system will be in operation, it would be great if that could also be set up before the first boot. Well, it can... maybe. Wi-Fi can be enabled without problems on newer Raspberry Pi's that have a built-in Wi-Fi (Raspberry Pi Model 3 and newer). On older Raspberry Pi it may be possible, but that requires a Wi-Fi USB dongle that is supported by the Linux kernel; see Various Hardware with Raspbian Buster Lite, USB Wi-Fi Dongles for some details.

Furthermore a static IP addresses can be assigned to the Ethernet network interface. It is also possible to do that for the Wi-Fi interface if the desktop (or portable) computer used to create the SD image is running Linux or Mac OS (I have not verified this claim on a Mac).

The setup is in two or three parts. I am assuming that the (micro) SD card containing the Raspbian image is still connected to the desktop computer and its boot partition is visible in the desktop system. If not, remove the card from the reader, wait awhile and insert it in the reader again. The partition should show up as its file system is the ubiquitous FAT32 which most operating system can read. If the desktop is running Linux or MacOS the second, much bigger partition, rootfs will also show up. Windows will not read an Ext4 partition.

Enable ssh

This approach entails connecting to the Raspberry Pi over the local network using the SSH protocol. By default the ssh service is not enabled in Raspbian Buster. But, this is easily remedied. If there is a file named ssh in the boot partition when the Raspberry Pi is first booted, then the service will be enabled and started. The content of the ssh file does not matter. Indeed, it can be an empty file.

Enable Wi-Fi (optional)

What is required is to create the wpa_supplicant configuration file with Wi-Fi credentials. I used nano from the command line, but any text editor could probably be used. The file, wpa_supplicant.conf, goes in the boot partition alongside the ssh file created in the previous state. You have to adjust the two-letter country code and the network credentials. The quotation marks around the SSID and password are necessary.

Set Static IP Address (optional)

This not strictly necessary but it is highly recommended to set static addresses for the network interfaces that will be used. First, this Raspberry Pi will be a server which needs to be easily found on the local network. Having a static IP address will accomplish this. Otherwise, the IP address may change when the Pi is rebooted. As the Domoticz installation script says:

    Domoticz is a SERVER so it needs a STATIC IP ADDRESS to function properly.

The other reason is that it will make it much simpler to find a headless Raspberry Pi to complete its basic configuration, especially for those that use a Windows desktop or portable computer.

I will make two points before going on to show how to proceed.

1. Make absolutely sure that the static address about to be assigned to the Raspberry Pi is not used by another device connected to the local network.
2. Make absolutely sure that the static address about to be assigned to the Raspberry Pi is not used by another device connected to the local network.

The static address can always be changed later.

There are two ways of setting a static IP address. The "easy but incorrect way" that will only work with an Ethernet connection to the Raspberry Pi. It is the only way I know that can be used by those that are using a Windows desktop. The "harder and correct way" will work with either an Ethernet connection or a Wi-Fi connection if it has been set up correctly as explained above.

Easy but incorrect

A simple addition to the text file /boot/cmdline.txt is all that is needed.

Add "ip=192.168.1.100" (or whatever the IP should be) without the quotation marks at the end of the single long line of commands the file contains.

That is all that is needed right now. It will be better to clean things up later when the Raspberry Pi is configured.

Strangely (or not?) if Wi-Fi credentials have been defined, the Wi-Fi interface will not be enabled unless the wired connection is in place when powering the Pi for the first time. In that case, the Ethernet interface will have the static IP address specified in cmdline.txt, the Wi-Fi IP address will be set by the LAN dhcp server.

Harder but correct

This requires access to the rootfs partition which, on my Ubuntu desktop machine, is mounted in the /media/michel directory alongside the boot partition. Sometimes there is a problem trying to cd into the rootfs. This happens when the SD card is changed quickly. Start the disk application, find the SD card in the list of drives in the left pane and mount both the boot and rootfs partitions by selecting the partition in the listt of Volumes and then clicking on the left most button with the right pointing triangle. If the button shows a square, the partition is already mounted.

The mount point will be displayed on the Contents line at the bottom. In my case it is /media/michel/rootfs. The fixed IP address is added to the dhcpcd configuration file. I prefer to back up the original file found in the rootfs/etc directory before modifying it.

Alternatively, nano can be made to back up the original when saving a changed version of the file.

The following lines, with appropriate changes, need to be added to the configuration file.

Initial Boot

It is now time for the initial boot. All that needs to be done is to safely eject the SD card from the desktop reader, then place it in the SD slot of the Raspberry Pi. I suggest that the Raspberry Pi be connected to the local area network (LAN) with an Ethernet cable for the initial boot at least, especially if a Wi-Fi dongle is being used. Indeed if the cmdline.txt file was modified to set a fixed IP address, then a wired connection to the LAN is mandatory. You may have to wait until the LED activity stops and the red LED is steadily on to open an ssh session.

Since I was able to set up a static IP for Ethernet and Wi-Fi using the "hard but correct way", I opened a remote session with ssh at the specified address Wi-Fi address without even connecting the Ethernet cable.

In Windows the command would be the same

but the Ethernet IP address has to be used and a wired connection is necessary.

If it had not been possible to set a static IP address, then try to connect to raspberrypi.local as shown here.

This will work in a desktop computer if its operating system has so called Zeroconf technology in place. That is the case for most major Linux distributions where the Avahi service is installed. It should also work in Mac OS where Bonjour takes care of the zeroconf protocols. It will not work in a fresh install of Windows. I tried and got this.

Nevertheless, give it a try because it will work in Windows if Bonjour Print Services from Apple was installed previously to reach a networked printer.

If this is the first time you log onto a Raspberry Pi with ssh then things should go smoothly. Just enter the default password, raspberry.

If the IP address had already been used for an ssh session or if the host name raspberrypi.local had already been used then expect some complication.

Something ever more byzantine could be displayed. Assuming that everything is safe, just go along with the suggestions.

Basic Configuration

At this point you should be logged into the Raspberry Pi. If you installed the desktop of version of Raspbian and connected a monitor, keyboard and mouse to the Pi, you will be in the graphic user interface, so it will be necessary to start a terminal to follow along. This is not too hard, just find and start the LXTerminal application in the usual fashion. You could also try the 3 key combination AltCtlT which may pop up the terminal window as in Ubuntu.

The password and other basic parameters should be changed. This is best done with the configuration utility. But before, you may want to change the avahi configuration file.

With that, the "workstation" will show up in the avahi-browse --all command.

If cmdline.txt had been modified to set a static IP address for the Ethernet network interface, then this is time to set the static IP address the correct way and to remove the modification to the file. Similarly if a static IP address had not been assigned before booting, then this would be the opportune time to do it. The static address or addresses are set in the dhcpcd configuration file.

You just need to add the static addresses as shown above. This will take effect on the next reboot of the Pi. But do not reboot now, continue on to complete the basic configuration and then reboot as instructed.

If the cmdline.txt had been modified it should be returned to its original state now that dhcpcd.conf has been correctly modified.

Just remove ip=192.168.1.100 (or whatever the static IP was) at the end of the line. The sudo prefix is needed here because the copy of cmdline.txt in the ext4 partition belongs to root.

Now start the configuration utility.

This is the opening screen of the configuration.

If you logged in with Kermit over a serial connection, you will not get the pretty colours:

It does not matter, the program works the same way. Navigate with the Tab key ⇥ to move from one control to the next and with the up ↑ and down ↓ cursor keys to move within a list. Select the highlighted choice by pressing on the ↲ or Enter key. The selection can also be made by "tabbing" to the <Select> or <Select> control at the bottom and then press the Enter key. I suggest that the following changes be made.

• 5 Interfacing Options / P2 SSH:
  choose <Yes> to enable the SSH server if it has not already been done. Even if you installed the Desktop version of Rasbian and have a keyboard and monitor hooked up to the Raspberry Pi, I recommend doing this.
• 4 Localisation Options / I2 Change Timezone:
  select your continent or region and then select your city or one nearby in the same time zone. As this is a headless system, there is no real advantage to change the locale or keyboard layout.
• 7 Advanced Options / A3 Memory Split:
  if this Pi will be used as a server it makes sense to allocate as little memory as possible to the graphic processing unit. This is 16 M bytes.
• 1 Change User Password:
  do not forget it; save it in a password manager (I use KeePassXC).
• 2 Network Options / N2 Wi-fi:
  enter the wireless network name and password if Wi-Fi connectivity is desired and if it was not defined already.
• 2 Network Options / N1 Hostname:
  change the host name to avoid collisions with other Raspberry Pi that may connect to the network currently or in the future. For illustrative purposes, I set the host name to goldserver.

Reboot as suggested, and then after a while check that the Raspberry Pi publishes its host name and then reconnect with ssh. Even if a keyboard and monitor are connected to the Raspberry Pi, it is a good idea to learn how to connect via SSH to a Pi that will be used as a server.

If everything seems in order and if it had been necessary to edit the config.txt file to enable the uart because the Raspberry Pi has a bluetooth chip, then I suggest disabling the UART.

This is to regain the slight loss of performance caused by enabling the serial port. See Configuring The GPIO Serial Port On Raspbian Jessie and Stretch Including Pi 3 and 4 by Jon Watkins for details.

Change the Default User Name

For added security, it is not a bad idea to change the default user. In the past, I would simply add a new user and eventually removed the default user pi. But that was a hit-or-miss approach which usually meant hunting down a missing group membership for the new user when something new did not work. Luckily, Jim McDonnell at Unix etc. has a very good explanation of how to proceed systematically to avoid such problems. I just followed the instructions in How to Rename the Default Raspberry Pi User, executing the scripts found in that article. There were a couple of error messages.

Because the listed configuration files are not present in the Lite version of Raspbian there were no real errors, as mentioned in the article.

Because the default pi user was renamed woopi, ssh connections need to be with that account from now on.

Updating the Operating System

Before installing any major new piece of software, it is always important to update the operating system. This is something that is done more or less automatically in full blown desktop systems such as Ubuntu. On the Lite version of Raspbian, it has to be done manually.

This operation took a surprising amount of time even if the image was rather recent. Now you may be time to get a coffee, tea or other beverage.

Syslog

I also enabled the UDP module of the log server on the Raspberry Pi because a number of home automation devices send their log messages to the Pi (see Syslog Server on Raspbian).

Working Directories

I added two directories to the woopi home directory that will hold bash scripts and python scripts.

Then I added that directory to the search path by appending the following to the .profile file in the home directory.

I prefer creating a virtual Python 3 environment to contain what I'll call system Python scripts. That way it will be possible to experiment with newer libraries in the future without breaking the Python scripts the system relies upon. A couple of packages need to be installed to create virtual environments.

I have written a bash script that automates the creation of virtual environments. This will copy it to the bash script directory and make it executable.

Following a suggestion by AndyG, I added a shell function and alias in my .bashrc file.

Log out of the woopi account and log back in for these changes to take effect and then create the virtual environment.

At this point I copied the saved pythons scripts into the environment. Some of these will not work because of missing Python modules. However these are easily added if the environment had been saved from a previous installation as explained above.

Of course if you did not have a virtual Python environment in a previous installation of Raspbian, then these last two steps cannot be followed. Later I will show which Python modules were added to the Python environment.