Syslog Server on Raspbian and Tasmota Remote Logging

Remote Logging of the Orange Pi Zero Core Temperature

Two single board computers, which support the home automation system, are on at all times. An Orange Pi Zero is not doing much except running HA-Bridge and acting as an IR receiver. A Raspberry Pi, on the other hand, is rather busier, as it hosts a home automation server, an MQTT server, a VPN server, an HTTP server, an RF bridge, and occasionally it streams video. Not bad for a lowly single core Model 1 Pi. But I want it to do more, I want it to become the centralized syslog server for the home automation system. It is already receiving error log messages from all the ESP8266 devices running Tasmota around the house. I explained how that is done in a previous post: Syslog Server on Raspbian and Tasmota Remote Logging. Now I want it to serve as a centralized log receiving messages from the Orange Pi Zero.

Table of contents

1. Viewing Logs
2. Logging Messages to a Remote syslog Server
3. Email Alerts Using syslog

Viewing Logs

There is a simple command that displays log messages as they arrive in a log file such as user.log or syslog. This is how it is done in Raspbian.

Blasted! a problem with One-Wire again. Might as well use the user.log; it is much less busy. Use the two finger salute CtrlC to stop the execution of the command, and then "tail" the wanted log.

In Armbian it is necessary to request higher privileges with the sudo prefix each time.

I found it worthwhile to add the default user to the adm group.

For this experiment in remote logging, I will have three terminals opened on my desktop computer.

1. A terminal with an ssh session on the Raspberry Pi displaying the content of its user.log (or syslog) It will be distinguished by its  green background  colour.
2. A terminal with an ssh session on the Orange Pi Zero displaying the content of its user.log (or syslog). It will be shown with a  purplish background  colour.
3. A terminal with a second ssh session on the Orange Pi Zero which will be used to create and send log messages to its rsyslog.service. Its background colour will be black as usual.

Sending log messages on the Orange Pi Zero (or most Linux systems, I suppose), is fairly simple with the logger utility. The following sends the message "test local log system" to the logging system with priority err (for error of course).

Which results in the following showing up in the Orange Pi Zero user.log.

The message does not show up in the Raspberry Pi logs of course. Not yet anyway.

Instead of using logger, I prefer generating the log messages with a Python script which is what will be done in practice later on.

Logging Messages to a Remote syslog Server

All log messages from the Orange Pi Zero can be sent to the remote Raspberry Pi syslog system with the addition of a single line to the the rsyslog configuration file. By the way, I am not totally inconsistent; rsyslog is an outgrowth of the original Linux logging system syslog. The latter defined the basic protocol. There is a third project also aimed at replacing syslog called syslog-ng. I have not used the latter at all for the simple reason that rsyslog is installed by default in Debian based Linux based distributions including Armbian, Raspbian and Ubuntu.

Here is part of the configuration file /etc/rsyslog.conf. Additional rules should be added just above the default rules in the file as shown below.

As suggested, instead of modifying the /etc/ryslog.conf file, new rules can be added in text files in the /etc/rsyslog.d directory and they will be included as long as they have the .conf extension. Typical file names would be 10-forward-user.conf and 20-suspend-debug.conf. The two digit prefix being used to prioritize rules much like it is done in the udev system. I leave it up to the reader to decide which of these two methods to use.

Before looking at forwarding rules, I will mention that there are numerous ways of specifying them which I found very confusing when looking at information from different sources on the Web or event rsyslog own extensive documentation. This has come about because rsyslog is meant to be a drop-in replacement to syslog with additional bells and whistles, it accepts the syslog configuration syntax. But it introduced extensions with a new syntax which was deprecated starting with version 6.0 of the project. At that point an advanced format (a.k.a. RainerScript) was introduced. That means there may be up to three ways of specifying a rule in the configuration file, one of which is to be avoided. It is actually possible to mix the formats; there can be more than three ways to obtain the desired routing of messages.

Forwarding all log messages with the default protocol

Here is an example to forward all log messages to the remote syslog server at 192.168.0.22 using the default UDP protocol on port 514.

# # Forward all messages to the remote server # *.* @192.168.0.22:514 - or - *.* @192.168.0.22 - or - *.* action(type="omfwd" target="192.168.0.22" port="514" protocol="udp") - or - *.* action(type="omfwd" target="192.168.0.22" port="514")

Warning: zeroconf mDns hostnames such as domus1.local will not work. Once this change has been saved, rsyslog must be restarted. And then the new rule will take effect.

zero@opi:~$ sudo systemctl restart rsyslog.service zero@opi:~$ logger a first test of remote logging

Which results in the following showing up in the Orange Pi Zero and Raspberry Pi user.log files.

zero@opi:~$ tail -f /var/log/user.log

Mar 9 20:42:08 localhost zero: test local log system Mar 9 22:48:24 localhost zero: a first test of remote logging

pi@raspberry:~$ tail -f /var/log/user.log

Mar 9 22:48:24 localhost zero: a first test of remote logging

If nothing appears, check that no error was made while entering the rule. A good place to start is sudo systemctl status rsyslog.service. And, of course, the UDP input module must be loaded and enabled on the remote rsyslog daemon on the Raspberry Pi. That usually means removing the # symbol in front of the two lines containing imudp in the configuration file.

pi@raspberry~$ cat /etc/rsyslog.conf ... ################# #### MODULES #### ################# module(load="imuxsock") # provides support for local system logging module(load="imklog") # provides kernel logging support #module(load="immark") # provides --MARK-- message capability # provides UDP syslog reception module(load="imudp") input(type="imudp" port="514") ...

Don't forget to restart the service on the Raspberry Pi if this change had to be made.

Forwarding all log messages with the TCP protocol

Here is an example to forward all log messages to the remote syslog server at 192.168.0.22 using the default TCP transport protocol on port 10514.

# # Forward all messages to the remote server with TCP # *.* @@192.168.0.22:10514 - or - *.* action(type="omfwd" target="192.168.0.22" port="10514" protocol="tcp")

The TCP input module must be enabled on the remote server which means removing the # symbol in front of the two lines of /etc/rsyslog.conf containing imtcp on the Raspberry Pi.

Forwarding all messages remotely but not locally

# # Forward all messages to the remote server but not locally # *.* @192.168.0.22:514 & stop - or - *.* action(type="omfwd" target="192.168.0.22" port="514" protocol="udp") & stop - or - *.* { action(type="omfwd" target="192.168.0.22" port="514" ) stop }

Of course this could be done with the TCP protocol also. The last version could just as easily be on one line. The advanced format is free form like Pascal and C. However, the older syslog format is not and the discard command stop must be on a second line that begins with the & symbol. A lot of examples still use the tilde ~ as the discard command instead of stop but that is considered obsolete and is slated to be illegal in the future. The second version shown above is a mixture of syslog and advanced formats and probably considered bad form.

These forwarding rules send more information to the Raspberry Pi than I want because the selector *.* means that messages from all facilities (the first "*") no matter their priority (the second "*") will be sent on to the target. A facility is a subsystem that produces log messages such as the mail and cron systems. There are quite a few of those, but the one that matters here is user which is what is used by default by the logger (and the syslog Python module to be examined latter). So we could limit the messages sent to the remote server by selecting that single facility.

Forwarding all user messages to a remote server

Either version of the following rule will forward all messages from the user facility only to the remote syslog server. Note that user is the default if nothing else is specified.

# # Forward only user messages to a remote server # user.* @192.168.0.22:514 - or - user.* action(type="omfwd" target="192.168.0.22" port="514" protocol="udp")

zero@opi:~$ sudo systemctl restart rsyslog.service zero@opi:~$ logger -p user.err an error message from the user facility zero@opi:~$ logger -p ftp.notice a notice message from the ftp facility

zero@opi:~$ tail -f /var/log/user.log

Mar 9 20:42:08 localhost zero: test local log system Mar 9 22:48:24 localhost zero: a first test of remote logging Mar 9 31:12:04 localhost zero: an error message from the user facility

zero@opi:~$ tail -f /var/log/ftp.log

Mar 9 31:12:04 localhost zero: a notice message from the ftp facility

pi@raspberry:~$ tail -f /var/log/user.log

Mar 9 22:48:24 localhost zero: a first test of remote logging Mar 9 31:12:04 localhost zero: an error message from the user facility

pi@raspberry:~$ tail -f /var/log/ftp.log

Looking at the default RULES in the Orange Pi Zero rsyslog.conf file, it can be seen that the first message sent with logger will be sent to the remote server and then will be written to the syslog and user.log files in the Orange Pi Zero /var/log directory. And since the remote server has the same configuration, the log message will be appended to /var/log/syslog and /var/log/user.log on that system also.

The second message sent by logger will not be forwarded to the remote log server because only messages from the user facility are selected by the forwarding rule and the message is from the ftp facility. It will be appended to the local syslog and ftp.log files in the Orange Pi Zero /var/log directory, according to the default rules in rsyslog.conf and because none of our added rules deal with messages from that facility.

Forwarding only some user messages to a remote server

I want to be even more selective about which user messages are to be forwarded to the Raspberry Pi. So an expression-based filter comes after the selector. It tests if the message tag field is equal to "opi". Note that tag is the termed used by the logger utility, rsyslog documentation calls the field programname and when displaying the log, it shows up as the localhost field.

# # Forward messages from 'opi' only to remote server and do not log locally # user.* if $programname == 'opi' then @192.168.0.22:514 & stop - or - user.* if $programname == 'opi' then { action(type="omfwd" target="192.168.0.22" port="514" protocol="udp" ) stop }

zero@opi:~$ sudo systemctl restart rsyslog.service zero@opi:~$ logger -p user.err -t opi this message programname is opi zero@opi:~$ logger -p user.err this message prograname is not opi but zero

zero@opi:~$ tail -f /var/log/user.log

Mar 9 20:42:08 localhost zero: test local log system Mar 9 22:48:24 localhost zero: a first test of remote logging Mar 9 31:12:04 localhost zero: an error message from the user facility Mar 9 39:53:31 localhost zero: this message prograname is not opi but zero

pi@raspberry:~$ tail -f /var/log/user.log

Mar 9 22:48:24 localhost zero: a first test of remote logging Mar 9 31:12:04 localhost zero: an error message from the user facility Mar 9 39:53:31 localhost opi: this message programname is opi

The first message will be forwarded to the remote logger where it will be written to the syslog and user.log journals. It will not be added to the local log files. The second message will not be forwarded to the remote logger because it fails the expression filter since its programname is the default value used by logger which is zero and not opi. Since it is not selected, the discard command will not apply and the second message will be logged locally to the syslog and user.log.

Be careful about the tag (programname). The documentation says

Precisely, the programname is terminated by either (whichever occurs first): end of tag nonprintable character ‘:’ ‘[‘ ‘/’

Tests showed that a tag or programname containing a hyphen such as ir-rem could be used. However, there seems to be a quirk in the Python syslog module which made this particular programname unacceptable. The module apparently truncates the programname property at the '-' character. I spinned my wheels a considerable amount of time because of that problem and wished that I had used logger initially to investigate this subject instead of a Python script.

My choice for forwarding rule

I left the configuration file /etc/rsyslog.conf on the Orange Pi Zero unchanged and I created a configuration file with a single rule.

zero@opi:~$ sudo nano /etc/rsyslog.d/10-opi.conf

This is its content.

# # Forward user messages from 'opi' only to the remote server using # the TCP protocol and do not log them locally # user.* if $programname == 'opi' then { action(type="omfwd" target="192.168.0.22" port="514" protocol="tcp") stop }

One advantage of this policy is that should a number of programs need remote logging, it can easily be accomplished without creating individual forwarding rules by settig their programname property to opi.

Mail Alert Using syslog

It is possible to write a short Python script that sends an alert email and having it executed by the rsyslog service when it receives a log message with priority alert and programname equal to opi which contains the word temperature. This log message will be generated by a Python script that reads the CPU temperature at regular intervals. Details will be provided in a later post. The rsyslog service on the Orange Pi Zero or by the Raspberry Pi hosting the remote log can send the email, it does not matter which. However, the alert will be sent if the core temperature is too high on the Orange Pi Zero, so it may be better to let the Raspberry Pi send the email notification.

Here is the rsyslog rule that will execute the script.

Save it in a file named /etc/rsyslog.d/05-opi.conf. This is important on the Orange Pi Zero to ensure that it is executed before the /etc/rsyslog.d/10-opi.conf rule, which stops executions of all rules from the user facility no matter their priority. If sending emails is done on the Raspberry Pi, then this has no consequence. Notice that it will only execute the script if all four conditions are fulfilled:

1. The log message comes from the user facility
2. The log message has the alert priority
3. The log message sender is opi
4. The log message contains the word 'temperature' (case insensistive).

All these conditions should prevent a flood of emails being sent out. Here is a skeleton Python script that sends the alert message using through an external SMTP mail server (Google, Hotmail, etc) using the SSL (TLS) protocol.

Dowloadable source code: sendmail.py

It would be much better to send the error message itself in the email. By the same token, it would be better to use the omprog: Program integration Output module intead of the legacy ^program syntax in rsyslog. However this is just checking that the idea is workable, and I will not pursue the matter at this juncture because the solution based on the Domoticz server is what I am actually using. I am in the process of writing a note about that and it should be available in a few weeks or perhaps even days.

Syslog Server on Raspbian and Tasmota Remote Logging

Remote Logging of the Orange Pi Zero Core Temperature