2023-07-21
md
GNATS, a Tiny Basic ESP32 GPS Based NTP Server
Adding a Local Network Time Server in Linux-> <-First Look at the Seeed Studio XIAO ESP32C3

GNAT'S Nearly Accurate Time Server
More XIAO Fun
GNATS icon

With apologies to GNU('s not Unix) and GNU GNATS

Now that the Wi-Fi Switch for Domoticz using a XIAO ESP32C3 project is in a working state, a break was in order even if I should get on with Part 5. It was an opportunity to do something about my unfortunate contribution to a digital tragedy of the commons. All the home automation devices based on Tasmota call on outside NTP servers to get the time. Our sprinkling of voice assistants and video streaming devices do the same thing and there are usually an additional six, or more, larger devices (desktops, portable computers, NAS, smart phones, tablets) left on that probably check up on the time on a regular basis. A better Internet citizen would set up a local NTP server to service all those devices.

Table of Contents

  1. Why Build a Time Server and Why GPS?
  2. The ATGM366H-5N-31 GPS Module
  3. An NTP Server
  4. A Second Insight
  5. Other Libraries
  6. Overview of the Firmware
  7. Testing GNATS
  8. Adding a DS3231 Real Time Clock
  9. Source Code and Additional Thoughts

Why Build a Time Server and Why GPS? toc

Why buid a time server? The obvious solution is to enable an NTP server in the system router or firewall device if possible. The router from my Internet service provider (ISP) does not support this. Besides, GNATS which uses the global positioning system (GPS) will not rely on an Internet time source which is good given that storms have caused power loss for hours and even days in the past. Some would ask why care about providing accurate time to devices that rely on power when there is no power? Some power outages have been hard on our previous ISP that could not get its system on line for days after power was restored. Besides, many of our devices are battery powered and we do have an emergency generator. Its not powerful enough to run the water pump, but it can power the fridge, some lights and quite a few devices. We all have priorities. There are plenty of other reasons to run a local time server.

Why use GPS as a time source? My past experience with GPS indoors has been very unsuccessful. As it happened, I ran across that old USB GPS receiver that came with a long-lost copy of Microsoft Streets & Trips purchased years ago. A Linux daemon (gpsd — a GPS service daemon) worked with the receiver and it did coax some data from it, including the time. The next day, it would not work. Perhaps atmospheric conditions were worse, perhaps I had done something wrong when playing with the receiver, but it did not really matter because it is not a good idea to base projects on a single device that is no longer readily available. Furthermore USB GPS receivers are not recommended as time sources because of transmission delays across a USB connection.

Universal-Solder sent an e-mail about a sale which included the serial ATGM336H GPS Module for Arduino Raspberry Pi STM32 ESP32. Because Arduino libraries were easy to find and the device was affordable, I splurged on a couple receivers. That thumb nail sized receiver looked like the perfect mate to the diminutive XIAO ESP32C3 to come up with a tiny time server. So there's another reason for using GPS: it is cheap. A bare time server can be had for slightly less than $10 to $15.

There is another possibility: atomic clock receivers. I did purchase a kit, again from Universal-Solder a couple of years ago. Unfortunately, I never got around to building it. Looking at it now that the first version of GNATS is finished, it looks to be a simple kit to build, so there may very well be a follow-up post on using this time reference.

The ATGM366H-5N-31 GPS Module toc

Here is some information about the GPS module that I gleaned from some vendor sale blurbs.

Supported satellite nav systemsGNSS and BDS
InterfaceTwo wire serial and PPS (precise pulse second)
Baud9600 (default)
Output protocolNMEA 0183
Compatibiltyublox NEO-6M
Time to first fix~32 seconds
Postionning accuracy2.5 metres
Update frequency1 Hz (default) or 10 hz
Supply voltage VCC2.7 - 3.6 Volts
Average power consumption < 25 mA @ 3.3V volts
Dimensions13mm x 16mm
Antennaincluded
LibariesTinyGPS++, NeoGPS (not tested), and probably others

The baud and update frequency values imply that settings could be changed on the module. After all, it has a serial Rx connection. However the only document I could find in English, a ATGM33H-5N User Manual has no explicit information. There are references to other documents but no links. Chinese speakers may find more information from the Hangzhou ZhongKe Microelectronics web site. Consequently, I used the default values.

GPS and XIAO connections Those wanting the NMEA 0183 standard can purchase it here for a few thousand dollars. Of course, I relied on information gleaned from the web such as What Exactly Is GPS NMEA Data?. The first test of the hardware was done with a XIAO ESP32C3. Only three connections are required between the XIAO and the GPS receiver.

Then I modified the DeviceExample.ino example sketch included in the TinyGPSPlus library to use the hardware UART. This is the code with the modifications.

/* Modified version of DeviceExample.ino included in TinyGPSPlus @ https://github.com/mikalhart/TinyGPSPlus */ #include <TinyGPSPlus.h> #if defined(HDW_SERIAL_INTF) /* Modification - Modified to use the hardware serial port of the ESP32 and a 9600 baud GPS receiver if compiled for a microcontroller using the ESP32 Arduino core. Tested with XIAO32C3, XIAO32S3 and LOLIN32 lite and an ATGM336H 5N-31 GPS receiver. sigmdel */ #include <HardwareSerial.h> static const uint32_t GPSBaud = 9600; static const int RXPin = UART_RX_PIN, TXPin = UART_TX_PIN; // defined in platformio.ini // The serial connection to the GPS #define ss HDW_SERIAL_INTF // defined in platformio.ini #else /* Original - This sample sketch demonstrates the normal use of a TinyGPSPlus (TinyGPSPlus) object. It requires the use of SoftwareSerial, and assumes that you have a 4800-baud serial GPS device hooked up on pins 4(rx) and 3(tx). */ #include <SoftwareSerial.h> static const uint32_t GPSBaud = 4800; static const int RXPin = 4, TXPin = 3; // The serial connection to the GPS device SoftwareSerial ss(RXPin, TXPin); #endif // The TinyGPSPlus object TinyGPSPlus gps; void displayInfo() { Serial.print(F("Location: ")); if (gps.location.isValid()) { Serial.print(gps.location.lat(), 6); Serial.print(F(",")); Serial.print(gps.location.lng(), 6); } else { Serial.print(F("INVALID")); } Serial.print(F(" Date/Time: ")); if (gps.date.isValid()) { Serial.print(gps.date.month()); Serial.print(F("/")); Serial.print(gps.date.day()); Serial.print(F("/")); Serial.print(gps.date.year()); } else { Serial.print(F("INVALID")); } Serial.print(F(" ")); if (gps.time.isValid()) { if (gps.time.hour() < 10) Serial.print(F("0")); Serial.print(gps.time.hour()); Serial.print(F(":")); if (gps.time.minute() < 10) Serial.print(F("0")); Serial.print(gps.time.minute()); Serial.print(F(":")); if (gps.time.second() < 10) Serial.print(F("0")); Serial.print(gps.time.second()); Serial.print(F(".")); if (gps.time.centisecond() < 10) Serial.print(F("0")); Serial.print(gps.time.centisecond()); } else { Serial.print(F("INVALID")); } Serial.println(); } void setup() { #if defined(SERIAL_BAUD) Serial.begin(SERIAL_BAUD); #else Serial.begin(); #endif #if defined(HDW_SERIAL_INTF) ss.begin(GPSBaud, SERIAL_8N1, RXPin, TXPin); #else ss.begin(GPSBaud); #endif delay(2000); Serial.print("Completed setup(), starting loop()"); } void loop() { // This sketch displays information every time a new sentence is correctly encoded. while (ss.available() > 0) if (gps.encode(ss.read())) displayInfo(); if (millis() > 5000 && gps.charsProcessed() < 10) { Serial.println(F("No GPS detected: check wiring.")); while(true); } }

And here is matching PIO configuration file, platformio.ini

[platformio] default_envs = seeed_xiao_esp32c3 ;default_envs = seeed_xiao_esp32s3 [env] framework = arduino platform = espressif32 lib_deps = mikalhart/TinyGPSPlus@^1.0.3 [env:seeed_xiao_esp32c3] board = seeed_xiao_esp32c3 monitor_speed = 460800 build_flags = -DCORE_DEBUG_LEVEL=0 -DHDW_SERIAL_INTF=Serial1 -DUART_RX_PIN=D7 -DUART_TX_PIN=D6 [env:seeed_xiao_esp32s3] board = seeed_xiao_esp32s3 monitor_speed = 460800 build_flags = -DCORE_DEBUG_LEVEL=0 -DHDW_SERIAL_INTF=Serial1 -DUART_RX_PIN=D7 -DUART_TX_PIN=D6

As can be seen, I tried this with two different XIAO ESP32 modules, but all my initial development was done with the XIAO ESP32C3. I placed the GPS receiver as close to the window as the USB cable to the desktop would allow. The antenna was about 5 cm from the glass pane and about 1.2 metres from the southernmost corner of the house and perhaps 1.5 metres above the ground. After letting the program run for about an hour, the coordinates had settled to constant values.

Location: 46.2xxxxx,-64.6xxxxx Date/Time: 6/7/2023 00:16:26.00 Location: 46.2xxxxx,-64.6xxxxx Date/Time: 6/7/2023 00:16:26.00 Location: 46.2xxxxx,-64.6xxxxx Date/Time: 6/7/2023 00:16:27.00

I entered the reported longitude and latitude into the map at epsg.io and was very pleased with the result. The coordinates corresponded to the correct corner of the house based on satellite imaging. It appears that the claimed 2.5 metre accuracy is not off the mark. The program also displays the UTC time. Indeed these were available much before the first positional fix was obtained, which is of importance given the project's objective. Note the 1 second resolution. There will be more information about that later.

An NTP Server toc

Now that I had a source of accurate time signals, I needed to find a way to turn the XIAO into an NTP server. It should not have been surprised that a great number of persons have done this type of thing before. Many take this subject very seriously and most of these would not approve of having a server providing time packets over Wi-Fi. I don't have an ESP32 with an Ethernet connector. Even if technically the little XIAO ESP32C3 will be a stratum 1 NTP server, it does not have to be accurate to the microsecond. A home automation device can be turned on or off a half second late or a full two seconds early and it won't matter. So at this stage anyway, I can live with the jitters associated with a Wi-Fi connection. Time keeping on the Internet is a complex subject with diagrams such as the one below and erudite discussions about precision, frequency, jitters and so on.

Paradigme NTP client-serveur

To be honest I don't understand most of it. Nevertheless, I will try to do this first version as correctly as my limited knowledge allows. Of course, the best approach in circumstances such as these it to find an example that seems comprehensible and then try it out and hope for an aha! moment. I chose the ElektorLabs 180662 mini NTP with ESP32 project by Mathias Claussen because it had a write up and a modular approach which made it possible to start with the basics. I especially liked that it had a separate NTP server library with a very simple interface that could be used right away.

#include "Arduino.h" #include "AsyncUDP.h" class NTP_Server { public: NTP_Server( ); ~NTP_Server(); bool begin(uint16_t port , uint32_t(*fnc_getutc_time)(void) , uint32_t(*fnc_get_subsecond)(void) ); static void processUDPPacket(AsyncUDPPacket& packet); };

The server uses the asynchronous UDP library to handle incoming NTP requests and dispatch the responses with minimal interaction with the rest of the firmware. As can be seen, two callback functions are used to get the current time, so that all our main program has to do is to implement these functions. In my first version, that's exactly what I did in a very straightforward way: getUTCTime() returns the 32-bit timestamp from ESP32 built-in real-time clock (RTC). For this to work, the RTC has to be set to the correct UTC (or GMT, ZULU etc) time, but let's worry about that later. I have already justified the decision not to worry about subseconds.

// Called by the NTP server whenever a request for the time is handled // Returns the current epoch (seconds since 1970-01-01) from the ESP RTC // which is set to UTC time. uint32_t getUTCTime(void) { return time(NULL); } // Called by the NTP server whenever a request for the time is handled // Should return microseconds to add to epoch obtained by getUTCTime // but always returns 0. So the NTP server is accuracy is limited to // seconds. uint32_t getSubsecond(void) { return 0; }

That was good enough to establish that the NTP library worked which was to be expected, of course. On looking more closely at ntp_server.cpp, I spotted a few mistakes and thought that the code could be tightened a bit. Take this statement with a grain of salt, C++ is definitely not my field of expertise. It should be emphasized that GNATS is not burdened with the reuse of ... the timekeeping corefunctions [...] synchronize to different [time] sources with different priorities. For the time being, the basic server has only one time source, the GPS, which is used to correct the ESP32 real-time clock (RTC) peripheral at regular intervals. The RTC should be able to keep the time accurately enough in between these updates so that it can be used as the time source in ntp_server. The Espressif documentation Get Current Time in ESP-IDF Programming - Guide System Time shows how to get the UTC time with microsecond accuracy. So why not use that function directly without going through callback functions? My logic is that if multiple time sources were to be made available later, they should be used to synchronize the RTC at regular intervals in whatever priority will be deemed best, in a way that will be of no importance to the NTP server. Consequently the header file becomes even simpler. That was the first aha! moment: the RTC will be the time keeper, which will be "disciplined" by the GPS. That way if for whatever reason the GPS can no longer supply a valid time signal, the RTC can continue to provide a reasonably accurate time to the NTP server for a long while.

#include "Arduino.h" #include "AsyncUDP.h" class NTP_Server { public: NTP_Server( ); ~NTP_Server(); bool begin(uint16_t port = 123); private: static void processUDPPacket(AsyncUDPPacket& packet); };

Of course, the implementation has to be modified. Hopefully comparison with the original Elektor code and the generous comments make clear what was done.

#include "Arduino.h" #include "ntp_server.h" #include <lwip/def.h> #include "smalldebug.h" // The UNIX epoch starts on 1.1.1970 and the NTP epoch starts on 1.1.1900 #define NTP_TIMESTAMP_DELTA 2208988800ull // 70 years worth of seconds typedef struct{ uint8_t mode:3; // mode. Three bits. Client will pick mode 3 for client. uint8_t vn:3; // vn. Three bits. Version number of the protocol. uint8_t li:2; // li. Two bits. Leap indicator. }ntp_flags_t; typedef union { uint32_t data; uint8_t byte[4]; char c_str[4]; } refID_t; typedef struct { ntp_flags_t flags; uint8_t stratum; // Eight bits. Stratum level of the local clock. uint8_t poll; // Eight bits. Maximum interval between successive messages. int8_t precision; // Eight bits signed. Precision of the local clock. uint32_t rootDelay; // 32 bits. Total round trip delay time. uint32_t rootDispersion; // 32 bits. Max error allowed from primary clock source. refID_t refId; // 32 bits. Reference clock identifier. // Reference Timestamp: Time when the system clock was last set or // corrected, in NTP timestamp format. uint32_t refTm_s; // 32 bits. Reference time-stamp seconds. uint32_t refTm_f; // 32 bits. Reference time-stamp fraction of a second. // Origin Timestamp: Time at the client when the request departed // for the server, in NTP timestamp format. uint32_t origTm_s; // 32 bits. Origin time-stamp seconds. uint32_t origTm_f; // 32 bits. Origin time-stamp fraction of a second. // Receive Timestamp: Time at the server when the request arrived // from the client, in NTP timestamp format. uint32_t rxTm_s; // 32 bits. Received time-stamp seconds. uint32_t rxTm_f; // 32 bits. Received time-stamp fraction of a second. // Transmit Timestamp: Time at the server when the response left // for the client, in NTP timestamp format. uint32_t txTm_s; // 32 bits and the most important field the client cares about. Transmit time-stamp seconds. uint32_t txTm_f; // 32 bits. Transmit time-stamp fraction of a second. } ntp_packet_t; int8_t __calloverhead = 0; int8_t DeterminePrecision( void ){ /* Source: RFC 5905 pg 21 https://www.rfc-editor.org/rfc/rfc5905#section-7.3 Precision: 8-bit signed integer representing the precision of the system clock, in log2 seconds. For instance, a value of -18 corresponds to a precision of about one microsecond. The precision can be determined when the service first starts up as the minimum time of several iterations to read the system clock. Below the *average* time to call gettimeofday() is used. */ struct timeval tv; // time call to uint32_t start = micros(); for(uint32_t i=0;i<1024;i++) gettimeofday(&tv, NULL); uint32_t end = micros(); double runtime = ((double)(end-start) ) / ( (double) (1024000000.0) ) ; __calloverhead = log2(runtime); DBGF("DeterminedPrecision: runtime: %f, __calloverhead %d\n", runtime, __calloverhead); return __calloverhead; } AsyncUDP udp; NTP_Server::NTP_Server( ){ } NTP_Server::~NTP_Server(){ } bool NTP_Server::begin(uint16_t port){ DeterminePrecision(); if (udp.listen(port)) { udp.onPacket(NTP_Server::processUDPPacket); return true; } return false; } /* static function */ void NTP_Server::processUDPPacket(AsyncUDPPacket& packet) { uint32_t start_us = micros(); ntp_packet_t ntp_req; struct timeval tv_now; if (gettimeofday(&tv_now, NULL)) { DBG("NTP_Server unable to get time of day"); return; // error } //DBGF("NTP_Server tv_now = (%u sec, %u usec)\n", tv_now.tv_sec, tv_now.tv_usec); if (packet.length() != sizeof(ntp_packet_t)) return; // this is not what we want ! memcpy(&ntp_req, packet.data(), sizeof(ntp_packet_t)); ntp_req.flags.li = 0; // No impending leap second insertion ntp_req.flags.vn = 4; // NTP Version 4 ntp_req.flags.mode = 4; // Server ntp_req.stratum = 1; // We don't touch ntp_req.poll ntp_req.precision = __calloverhead; ntp_req.rootDelay=1; ntp_req.rootDispersion=1; strncpy(ntp_req.refId.c_str ,"GPS", sizeof(ntp_req.refId.c_str) ); // Set to NTP byte order ntp_req.rootDelay = htonl( ntp_req.rootDelay ); ntp_req.rootDispersion = htonl( ntp_req.rootDispersion ); ntp_req.refId.data = ntohl( ntp_req.refId.data ); // Set the origin Timestamp (origTm) which is the time at the client when // the request departed for the server, in NTP timestamp format. // In other words, it's the transmit time of the packet from the client. // Already in NTP byte order // // A systemd-timesyncd client will not update the system time if origTm // is not set to a "reasonable" value, even if txTm is correctly defined. ntp_req.origTm_s = ntp_req.txTm_s; ntp_req.origTm_f = ntp_req.txTm_f; // Set the receive Timestamp (rxTm) which is the time at the server // when the request arrived from the client, in NTP timestamp format. // About conversion of microseconds to 32-bit fractions of second // see https://gist.github.com/sigmdel/bea3b4065c6fdf2cc2d3c9c7fb1ddca0 ldiv_t delta = div(tv_now.tv_usec, 1000000UL); ntp_req.rxTm_s= tv_now.tv_sec + delta.quot + NTP_TIMESTAMP_DELTA; ntp_req.rxTm_f= (uint32_t) ( ((uint64_t) delta.rem << 32) / 1000000L ); // Set to NTP byte order ntp_req.rxTm_s = htonl(ntp_req.rxTm_s); ntp_req.rxTm_f = htonl(ntp_req.rxTm_f); // Set reference timestamp (refTm), which is the time when the system clock // was last set or corrected, to the received timestamp (rxTm). // - rxTm is already in NTP byte order ntp_req.refTm_s = ntp_req.rxTm_s; ntp_req.refTm_f = ntp_req.rxTm_f; // Set the transmit timestamp (txTm) which is the time at the server // when the response left for the client, in NTP timestamp format. // Using the original timestamp obtained at the beginning of the // routine plus the number of micro seconds elapsed since then. delta = div((long) (tv_now.tv_usec + micros() - start_us), 1000000L); ntp_req.txTm_s= tv_now.tv_sec + delta.quot + NTP_TIMESTAMP_DELTA; ntp_req.txTm_f= (uint32_t) ( ((uint64_t) delta.rem << 32) / 1000000L ); // set to NTP byte order ntp_req.txTm_s = htonl(ntp_req.txTm_s); ntp_req.txTm_f = htonl(ntp_req.txTm_f); packet.write((uint8_t*)&ntp_req, sizeof(ntp_packet_t)); #if (ENABLE_DBG > 0) DBGF("NTP response sent to %s:%d\n", packet.remoteIP().toString().c_str(), packet.remotePort()); ntp_req.txTm_s = htonl(ntp_req.txTm_s); ntp_req.txTm_f = htonl(ntp_req.txTm_f); DBGF("txTm_s %u sec, txTm_f %u fraction\n", ntp_req.txTm_s, ntp_req.txTm_f); #endif }

In the ntp_packet_t struct, the type of the precision code should be a signed 8-bit integer, as most times it will be a negative value. The function DeterminePrecision() in the Elektor code, was obviously meant to calculate that precision, but it was never used and its unusual divisor, which I could not fathom, always yielded a 0 value. With the obvious divisor to calculate the average time to call the gettimeofday() function the precision is -16 which was more in line with values used elsewhere. A quick test showed that using the minimum time instead of the average gives the same precision.

One of the advantages of using gettimeofday() is that the time value has a microsecond resolution while the original code used subsecond meant millisecond. Such precision was not that important for my intended use. Nevertheless, it seems like a good idea to use the increased resolution at this level, just in case I eventually implement a better time synchronization algorithm. The NTP time stamps measure fractions of a second with even greater resolution than the microsecond. I could not find a universal best way of converting microseconds to 32-bit fractions of a second, so I chose the following: ntp_fraction = (uint32_t) ( ((uint64_t) x_us << 32) / 1000000 ) where x_us is the number of microseconds. Detail in this gist.

There is no doubt room for improvement. In particular, the arbitrary assignments of values to rootDelay and rootDispersion should be investigated further. Next time I am looking for a rabbit hole in which to get lost, this might be a good topic to take on.

There is another worry. The incoming requests for NTP time packets will arrive asynchronously so that gettimeofday() will be called asynchronously. The main firmware task will update the RTC at regular intervals, so there is a potential for clashes. Hopefully, the underlying RTOS has some sort of lock mechanism to avoid contention over the resource. I am not sure. The function gettimeofday() is implemented as a wrapper around the IDF function _gettimeofday_r(). I believe that the trailing _r signals a reentrant routine which avoids one pitfall for sure. On the other hand, that does not answer my question about possible intervention between gettimeofday and settimeofday that is used to update the RTC time.

A Second Insight toc

Notwithstanding my misgivings, the modified NTP server did work and it was possible to update the RTC with time values obtained from the GPS. As often the case, the more complicated part of the code is dealing with non-normal operations. Notably, what should be done until the GPS can start acquiring time data? I kept on putting off that problem because ultimately, my goal is to add an external real-time clock, a DS3231 probably, as a backup. However I had nagging doubts about that because past experience has shown me that these RTC are not entirely dependable. I previously discussed problems with the home automation system hosted on a Raspberry Pi equipped with a malfunctioning RTC, The Domoticz Time Synchronization Problem. Then it dawned on me that what had been learned about the systemd-timesynced handling of the system clock when there is no RTC was actually applicable here.

I added an time_t mclock = 0 variable that will mimic the systemd-timesynced clock file.

/var/lib/systemd/timesync/clock
The modification time ("mtime") of this file is updated on each successful NTP synchronization or after each SaveIntervalSec= time interval, as specified in timesyncd.conf(5). At the minimum, it will be set to the systemd build date. It is used to ensure that the system clock remains roughly monotonic across reboots, in case no local RTC is available. (source)

Could not have said that any better myself. Of course there's a little problem, because a variable such as mclock will not survive reboots. Consequently, whenever a new value is assigned to mclock it is saved to non-volatile storage (NVS) which in the case of the ESP32 is flash memory. This is done with the Preferences library which is the ESP32 replacement of the Arduino EEPROM library

... #include <Preferences.h> // to save mclock to NVS ... Preferences preferences; // A timestamp that is updated at regular intervals and saved to non-volatile storage time_t mclock = 0; // Last time mclock was saved to NVS unsigned long mclocktimer = 0; // Save the current RTC time to mclock and NVS // if GPS_POLL_TIME has elapsed and if the RTC time // is greater than mclock. Called whenever a new GPS // time is obtained void savemclock(void) { if (millis() - mclocktimer >= GPS_POLL_TIME) { mclocktimer = millis(); time_t newvalid; time(&newvalid); // read current time from RTC if (newvalid > mclock) { // keep time moving along mclock = newvalid; preferences.begin("mclock", false); preferences.putULong("time", mclock); // save mclock value in NVS preferences.end(); } } } // Called in setup() to set the ESP32 RTC with the latest time // saved in NVS or the firmware compile time whichever is greater void loadmclock(void) { preferences.begin("mclock", false); mclock = preferences.getULong("time", 0); // default 0 if not already defined if (mclock < COMPILE_TIME) { mclock = COMPILE_TIME; // Unix timestamp macro set in platformio.ini preferences.putULong("time", mclock); // save mclock value in NVS } preferences.end(); if (mclock) { // should always be the case timeval tv; tv.tv_sec = mclock ; tv.tv_usec = 0; settimeofday(&tv, NULL); } }

In the course of developing this bit of code, I changed my mind about the name of the Preferences namespace. There is no way of removing a namespace, but the Espressif documentation provides a sketch to erase and reformat the NVS memory.

Other Libraries toc

It is a bit pretentious to call smalldebug.h a library. It is just defines two macros: DBG(...) and DBG(...). These are used throughout the code instead of Serial.println(...) and Serial.printf(...). The advantage of using these macros is that all the print statements will be stripped from the compiled firmware when the ENABLE_DBG macro is set to 0.

The other library used is ESP8266 and ESP32 OLED driver for SSD1306 displays by ThingPulse, Fabrice Weinberg. This library is certainly not the only available library for SDD1306 OLED displays, but I like it for a number of reasons. It has powerful graphic capabilities. The analogue clock is a good example and would be an alternate method of showing the local time. The built-in fonts support the full Latin1 character set. This means that it would be possible to write the full date in French with months such as février and août with diacritics displayed correctly. That said, the display is not a required element of this project and it can be omitted without problems. Just set the HAS_OLED build-flag to 0 in that case.

As shown in the previous section, the ESP32 real time clock will be set with the last known valid date and time. This can be the last value of mclock saved to NVS or it can be the date and time in effect when the compilation of the firmware was started on the computer hosting PlatformIO. That time is available as the built-in PlatformIO variable $UNIX_TIME which is assigned to the COMPILE_TIME macro added to environment build_flags in platformio.ini.

-DCOMPILE_TIME=$UNIX_TIME

I could not find the equivalent in the Arduino IDE. I mashed together two libraries found on GitHub, buildTime by Alex Gyver and arduino_compiledate by Mikael Sundin, to make something almost as easy to use in the Arduino IDE. It is the mdBuildTime "library". The word is in quote because it contains a single function: time_t unixbuildtime(). Here is how to use it, making sure to set the correct time zone beforehand.

#include <Preferences.h> #include "unixbuildtime.hpp" const char* timeZone = "AST4ADT,M3.2.0,M11.1.0"; ... setenv("TZ", timeZone, 1); time_t compileTime = unixbuildtime(); if (mclock < compileTime) { mclock = compileTime; ...

Here is the layout of the time server with the optional OLED screen.

GPS, OLED screen and XIAO connections

Be carefull, 0.96" OLED screen based on the SDD1306 controller are ubiquitous, but they do not all have the shown pinout. Indeed some connect over a SPI bus instead of I²C.

Overview of the Firmware toc

Program flowchart Assuming that the GPS receiver has locked onto a satellite, it sends messages over the serial link to the ESP32 at the start of every second. That data is passed on to the TinyGPSPlus library which decodes the messages. The library updates an internal time stamp, and other data such as longitude and latitude as the messages come in. Of course none of this starts until the receiver has acquired a signal which may never happen. On the other hand, the ESP RTC is continuously updated from the moment the microprocessor is powered up, but with an initial time value of 0. So there are two clocks. One knows what the actual time is with great accuracy when it functions but there is no guarantee that it will function. The other measures accurately enough the elapsed time since the last reboot of the ESP but it has no way to know when it actually started ticking. So we need to combine these two clocks by updating the RTC time as soon as the GPS provides a valid time stamp and then correcting the RTC on a regular basis with the latest GPS time. Some call these corrections discipline.

Here is a simplified flowchart of the firmware. Basically, it's just a big timing loop (like all Arduino sketches) where actions are taken at regular intervals. These intervals are defined in the platformio.ini configuration file.

Testing GNATS toc

Although not strictly necessary to test GNATS, a static IP address was assigned to the XIAO ESP32C3: 192.168.1.23. This was done with the home network DHCP server which is running on the ISP provided router. Then I created a simple Python ntpc.py by combining the work of others. There is also a more sophisticated version ntpc2.py to shows the content of the UDP packet received from the time server. It helped while improving ntp_server; more on that later.

Make sure that the script works by running it, without any argument, from a computer with Python 3 and a working connection to the Internet. It's best to ensure that the script was in a directory included in the path. In my case that was .local/bin in my home directory. After copying the script to that directory, I madeit executable and then ran it without a command line parameter.

michel@hp:~$ ls .local/bin/ntpc.py .local/bin/ntpc.py michel@hp:~$ chmod +x .local/bin/ntpc.py michel@hp:~$ ntpc.py NTP server 'pool.ntp.org' will be queried Received 48 bytes from ('173.72.40.32', 123): 1c0103ed0000000000000cd150505300e834be9872467d650000000000000000e834be9e2e5d70f1e834be9e2e7893b5 Time = Wed Jun 14 19:33:02 2023

That showed that the client does work. Testing GNATS with the script was easy.

pi@tarte:~ $ .local/bin/ntpc.py 192.168.1.23 NTP server '192.168.1.23' will be queried Received 48 bytes from ('192.168.1.23', 123): 240100f0000000010000000100535047e831f435d1e56cd60000000000000000e831f435d1e56cd6e831f435d1e8815e Time = Mon Jun 12 16:44:53 2023

Test set up Don't worry about the actual reported times, I am writing this after the fact.

The real test was to open an SSH session on a Raspberry Pi and have it use GNATS as a time server. That's a bit tricky because the Pi, which does not have a battery-backed real-time clock, automatically gets its time from well-known Internet NTP servers. To truly test GNATS as a replacement time server for the Pi, it is necessary to prevent the Pi from getting the time from a time server. The easy way of achieving this paradoxical situation is to make sure that the Pi cannot reach the Internet. This can be done by setting up a wired local network which is not linked to the Internet or to a real-time clock, but that requires an extra Wi-Fi router. Not everyone has will have a spare router on hand, so here is a procedure to perform tests in a less rigorous manner.

Connect the Raspberry Pi, GNATS and the desktop computer to the local network. The desktop can also be connected to the XIAO ESP32C3 via a USB cable to see the GNATS debug messages. Do not risk making a mess of an SD card containing a working system. I strongly suggest that this experiment be done with a new SD card on which Raspberry Pi OS is installed and updated. I ran my tests with the lastest 64 bit lite version. Open an SSH session using the correct host name of the Pi which will probably not be tarte. Actually, the Pi had a fixed IP address, 192.168.1.22, that was assigned by the DHCP server based on the Pi's MAC address. The desktop has a dynamic address assigned by the DHCP server which was at the time of the test 192.168.1.158. GNATS has a self assigned static IP address: 192.168.1.23. Finally, the router; which is also the gateway for all the devices on the LAN, is at 192.168.1.1 which is also a fixed address.

michel@hp:~$ ssh pi@tarte.local pi@tarte.local's password: ******** Linux tarte 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 ... pi@tarte:~ $ date Wed 14 Jun 20:30:07 ADT 2023

Not surprisingly, the time and date are correct, the Pi managed to get the time from a default time server on the Internet (debian.pool.ntp.org). Now let's break this by overriding this default with an address to a non-existing device on the local network.

pi@tarte:~ $ sudo nano /etc/systemd/timesyncd.conf ... [Time] #NTP= #FallbackNTP=0.debian.pool.ntp.org 1.debian.pool.ntp.org 2.debian.pool.ntp.org > FallbackNTP=192.168.1.88 ... exit (Ctrl+X), saving the file

Let's make that the Pi is not requesting the address of an NTP server from the DHCP server.

pi@tarte:~ $ cat /etc/dhcpcd.conf | grep -B 1 ntp_servers # Most distributions have NTP support. #option ntp_servers

Let's now clear the system journals as much as possible to be able to see, if necessary, what is going on after rebooting.

pi@tarte:~ $ sudo journalctl --rotate; sudo journalctl -m --vacuum-time=1s Vacuuming done, freed 0B of archived journals from /run/log/journal. Vacuuming done, freed 0B of archived journals from /var/log/journal. Deleted archived journal /var/log/journal/68c3ce45d68f40ecb7c4730a064d9f83/system@ee09835a6f8648bcb91eafc52cc3d3fb-0000000000000001-0005f06959508eae.journal (8.0M). Deleted archived journal /var/log/journal/68c3ce45d68f40ecb7c4730a064d9f83/system@0005f0695957ee93-7639cf1df51aa206.journal~ (8.0M). Deleted archived journal /var/log/journal/68c3ce45d68f40ecb7c4730a064d9f83/user-1000@7def2240baf44cbf8f5b297c18f4315e-00000000000006ae-0005fe0c27512cdb.journal (8.0M). Deleted archived journal /var/log/journal/68c3ce45d68f40ecb7c4730a064d9f83/system@0005fe0c3309af50-5f27d77197f922b9.journal~ (8.0M). Deleted archived journal /var/log/journal/68c3ce45d68f40ecb7c4730a064d9f83/user-1000@13555b8dc8df4bea8ce4670a2448a638-0000000000000299-0005fe0c2a97e7c1.journal (8.0M). Deleted archived journal /var/log/journal/68c3ce45d68f40ecb7c4730a064d9f83/system@b7bb17c8c2984dd89892509de2272f97-0000000000000001-0005fe0c330105a3.journal (8.0M). Vacuuming done, freed 48.0M of archived journals from /var/log/journal/68c3ce45d68f40ecb7c4730a064d9f83.

The output will be different, but that does not matter. Now let's reboot after erasing the time stamps that the system leaves in the file system to set the initial time of the Pi even before attempting to get the time from an NTP server.

pi@tarte:~ $ sudo rm -r /var/log/lastlog; sudo systemctl stop fake-hwclock; sudo rm /var/lib/systemd/timesync/clock; echo "2022-12-08 00:01:00" | sudo tee /etc/fake-hwclock.data; sudo touch -t 202212080001 /etc/fake-hwclock.data; sudo reboot 2022-12-08 00:01:00 Connection to tarte.local closed by remote host. Connection to tarte.local closed. michel@hp:~$ ssh pi@tarte.local pi@tarte.local's password: ******** Linux tarte 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 ... pi@tarte:~ $ date Thu 22 Dec 07:56:32 AST 2022 pi@tarte:~ $ timedatectl status Local time: Thu 2022-12-22 07:56:44 AST Universal time: Thu 2022-12-22 11:56:44 UTC RTC time: n/a Time zone: America/Moncton (AST, -0400) System clock synchronized: no NTP service: active RTC in local TZ: no pi@tarte:~ $ timedatectl show-timesync FallbackNTPServers=192.168.1.88 ServerName=192.168.1.88 ServerAddress=192.168.1.88 RootDistanceMaxUSec=5s PollIntervalMinUSec=32s PollIntervalMaxUSec=34min 8s PollIntervalUSec=2min 8s Frequency=0 pi@tarte:~ $ timedatectl timesync-status Server: 192.168.1.88 (192.168.1.88) Poll interval: 2min 8s (min: 32s; max 34min 8s) Packet count: 0

The (wrong) date and time correspond to the systemd compilation date which is used to "touch" clock which is the systemd-timesync timestamp.

pi@tarte:~ $ ls -l /lib/systemd/systemd -rwxr-xr-x 1 root root 1687072 Dec 22 07:55 /lib/systemd/systemd pi@tarte:~ $ ls -l /var/lib/systemd/timesync total 0 -rw-r--r-- 1 systemd-timesync systemd-timesync 0 Dec 22 07:55 clock

Now let's tell systemd-timesync to get its time from GNATS, check one last time that the date is wrong and then reboot.

pi@tarte:~ $ sudo nano /etc/systemd/timesyncd.conf ... [Time] #NTP= #FallbackNTP=0.debian.pool.ntp.org 1.debian.pool.ntp.org 2.debian.pool.ntp.org > NTP=192.168.1.23 FallbackNTP=192.168.1.88 ... exit (Ctrl+X), saving the file pi@tarte:~ $ date Thu 22 Dec 08:15:07 AST 2022 pi@tarte:~ $ sudo reboot

After a suitable delay, open a new SSH session and check that the time is correct and that GNATS is working as a local time server.

michel@hp:~$ ssh pi@tarte.local pi@tarte.local's password: Linux tarte 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 ... pi@tarte:~ $ date Wed 14 Jun 21:13:35 ADT 2023 pi@tarte:~ $ timedatectl status Local time: Wed 2023-06-14 21:13:55 ADT Universal time: Thu 2023-06-15 00:13:55 UTC RTC time: n/a Time zone: America/Moncton (ADT, -0300) System clock synchronized: yes NTP service: active RTC in local TZ: no pi@tarte:~ $ timedatectl show-timesync SystemNTPServers=192.168.1.23 FallbackNTPServers=192.168.1.88 ServerName=192.168.1.23 ServerAddress=192.168.1.23 RootDistanceMaxUSec=5s PollIntervalMinUSec=32s PollIntervalMaxUSec=34min 8s PollIntervalUSec=2min 8s NTPMessage={ Leap=0, Version=4, Mode=4, Stratum=1, Precision=-15, RootDelay=15us, RootDispersion=15us, Reference=, OriginateTimestamp=Wed 2023-06-14 21:13:53 ADT, ReceiveTimestamp=Wed 2023-06-14 21:13:53 ADT, TransmitTimestamp=Wed 2023-06-14 21:13:53 ADT, DestinationTimestamp=Wed 2023-06-14 21:13:53 ADT, Ignored=no PacketCount=3, Jitter=31.612ms } Frequency=-8163025 pi@tarte:~ $ timedatectl timesync-status Server: 192.168.1.23 (192.168.1.23) Poll interval: 2min 8s (min: 32s; max 34min 8s) Leap: normal Version: 4 Stratum: 1 Reference: Precision: 31us (-15) Root distance: 22us (max: 5s) Offset: -31.887ms Delay: 29.701ms Jitter: 31.612ms Packet count: 3 Frequency: -124.558ppm pi@tarte:~ $

Here is a curated list of journal entries that shows what happens during the boot process with respect to setting the system time.

pi@tarte:~ $ journalctl | grep -E 'ntp|dhcp|time' ... Dec 22 07:55:44 tarte systemd[1]: System time before build time, advancing clock. Dec 22 07:55:44 tarte systemd-journald[141]: Runtime Journal (/run/log/journal/68c3ce45d68f40ecb7c4730a064d9f83) is 2.2M, max 18.1M, 15.9M free. Dec 22 07:55:44 tarte fake-hwclock[129]: Current system time: 2022-12-22 11:55:43 Dec 22 07:55:44 tarte fake-hwclock[129]: To set system time to this saved clock anyway, use "force" Jun 14 21:18:17 tarte sudo[679]: pi : TTY=pts/0 ; PWD=/home/pi ; USER=root ; COMMAND=/usr/bin/journalctl -m --vacuum-time=1s Jun 14 21:18:35 tarte sudo[689]: pi : TTY=pts/0 ; PWD=/home/pi ; USER=root ; COMMAND=/usr/bin/rm /var/lib/systemd/timesync/clock ... Jun 14 21:18:36 tarte dhcpcd[604]: dhcpcd exited ... Jun 14 21:18:40 tarte systemd[1]: Stopped User Runtime Directory /run/user/1000. Dec 22 07:55:48 tarte systemd-timesyncd[309]: System clock time unset or jumped backwards, restoring from recorded timestamp: Wed 2023-06-14 21:18:42 ADT Jun 14 21:18:40 tarte systemd[1]: user-1000.slice: Consumed 1.638s CPU time. Jun 14 21:18:40 tarte systemd[1]: dhcpcd.service: Succeeded. ... Jun 14 21:18:43 tarte dhcpcd[396]: dev: loaded udev Jun 14 21:18:46 tarte dhcpcd[484]: wlan0: starting wpa_supplicant Jun 14 21:18:46 tarte dhcpcd-run-hooks[494]: wlan0: starting wpa_supplicant ... Jun 14 21:18:46 tarte dhcpcd[396]: eth0: offered 192.168.1.176 from 192.168.1.1 Jun 14 21:18:46 tarte dhcpcd[396]: eth0: probing address 192.168.1.176/24 Jun 14 21:18:51 tarte dhcpcd[396]: eth0: leased 192.168.1.176 for 259200 seconds Jun 14 21:18:51 tarte dhcpcd[396]: eth0: adding route to 192.168.1.0/24 Jun 14 21:18:51 tarte dhcpcd[396]: eth0: adding default route via 192.168.1.1 Jun 14 21:18:51 tarte dhcpcd[396]: forked to background, child pid 591 Jun 14 21:19:00 tarte dhcpcd[591]: eth0: no IPv6 Routers available Jun 14 21:19:34 tarte systemd-timesyncd[309]: Initial synchronization to time server 192.168.1.23:123 (192.168.1.23).

So GNATS works, but truth be told at first the Raspberry Pi would not udate the system time even if it was getting UDP time packets from GNATS. It turns out that my improvements to ntp_server were not so good; I was not setting the reference time stamp correctly. That's fixed now.

It would have been possible to test GNATS directly with the desktop computer without involving the Pi. I just did not want to risk creating any problems on my main computer just as it was best to install a new SD card on the Pi to avoid problems with the working system that was on that machine which is normally the rig used to test changes to the home automation system. Avoiding putting these complex systems in jeopardy is worth the extra work.

The real test of GNATS is actual use. In the next blog, GNATS was used to update the time on a Pi along side with real NTP servers from the Web. Unfortunately, the results were not very good. According to chrony, GNATS was classed as a falseticker because its time was too divergent from the majority of other sources. This result make me hesitate between calling the project GNAT's Not Accurate Time Server and GNAT's Nearly Accurate Time Server.

A proper timeserver should take advantage of the PPS (precise pulse per second) signal of the ATGM366. This pulse should trigger an interrupt that starts a true microsecond timer at the beginning of each second. Also a proper time server should not be running over Wi-Fi which suffers from much greater variability in the time to transmit packets than an Ethernet connection would.

Adding a DS3231 Real Time Clock toc

Minimal support for a battery-backed real time clock can be added with relative ease. As I had DS3231 based clock modules, which were used with the Raspberry Pi, it was an obvious choice. Since it is an I²C device, connecting it to the GNATS was straightforward.

schematic with OLED and RTC

startup flowchart Because I use the easily found CR2032 as the backup library instead of a rechargeable LIR2032, the schematic shows that the current limiting resistor of the battery charge circuit has been removed. There is plenty of documentation about this on the web: ZS-042 DS3231 RTC module, [DS3231: CR2032 vs LIR2032] Warning! Is your module killing the battery?. Perhaps the biggest problem is choosing from among the numerous Arduino libraries for the DS3231 real time clock that are compatible with the ESP32.

Let's be clear, the battery backup RTC is not being used as an alternate time source should the GPS receiver not be able to connect to enough satellites to obtain time signals. Instead the RTC is a supplementary mechanism to improve the initial setting of the ESP real time clock until the GPS receiver acquires time signals. The flowchart on the right shows how this is done. In the end, the ESP RTC will be set to whichever is the latest: the time saved in NVS, the time read from the battery-backed DS3231, the firmware compile time.

Whenever the ESP RTC is updated from the GPS time (in function savemclock()), that time stamp is also saved to NVS memory and used to reset the DS3231.These are the only changes made to GNATS when adding the RTC. The ElektorLabs NTP project makes better use of the hardware RTC. Time can be updated from 1) an RTC, 2) an NTP server (or servers I assume) and 3) a GPS receiver. When a time stamp is obtained from a source with a higher priority than ever encountered, it becomes the current master source and it's time stamp is used to update the time sources with a lower priority. If 15 minutes elapses without obtaining a time stamp from the current master, it is demoted and replaced with the source lower down the priority chain. That is an interesting mechanism since it means that the hardware real time clock can be used to update the ESP RTC when the GPS receiver is no longer able to get time values.

Source Code and Additional Thoughts toc

GNATS is available on GitHub. While the repository is publicly available, it should not be viewed as anything more than a pedagogical tool. It should not be used as the primary source of accurate time. However, I believe that it is still a valuable addition to my home automation system as a backup time source when access to better clocks is lost. In that role, I have no doubt that the single core ESP32-C3 found on the XIAO ESP32C3 will be up to the task. The follow-on post, Using a Local Network Time Server provides some insight into how GNATS can be added to a local network.

As mentioned above, better use of the DS3231 hardware clock could be made as a backup for the GPS and it is certainly something that I would add if GNATS were to be added to my home automation system. However that may not be the case. I will soon be adding a hardware firewall running either pfSense or OPNsense. It seems that both these can use a serial GPS receiver as a time source, so that it would no longer be necessary to create a full NTP server.

One of the very nice characteristics of the XIAO line of products from SeeedStudio is that they all provide UART, I²C and SPI serial interfaces on the same pins. While developing GNATS on a XIAO ESP32C3, I received a XIAO ESP32S3 with its two core ESP32-S3 that should be even more powerful than the original two core ESP32. I have used it instead of the XIAO ESP32C3 with success. All that had to be done was to modify the board definition in the platformio.ini configuration file. That is already available in the file supplied in the Github repository. I should point out that I had to move to the latest version of the Espressif 32 platform (version 6.3.1) released on May 26th because support for the XIAO ESP32S3 has been included only recently.

Adding a Local Network Time Server in Linux-> <-First Look at the Seeed Studio XIAO ESP32C3