NTP Servers with a Stable Oscillator

CESNET technical report number 7/2007
also available in PDF, PostScript, and XML formats.

Vladimír Smotlacha
31. 10. 2007

1 Abstract

This technical report describes design and construction of a stable and efficient NTP server disciplined by a 1 PPS signal.

Keywords: NTP server, PPS capture card

2 Introduction

Several network time services have been defined and implemented. Currently, only NTP (Network Time Protocol) [Mil92] is widely used. It constitutes a hierarchical structure of time servers where every primary NTP server requires an external source of time. When the clock stability and time accuracy is the main goal, the design of NTP server is a rather alchemistic task combining hardware and software approaches.

3 Consideration

CESNET operates primary (Stratum-1) NTP servers controlled by a GPS receiver since 2000. Later, we have installed another GPS receiver whose signal is now distributed to many servers and workstations requiring accurate time ([Smo04]). Our NTP servers were upgraded several time in last six years and we reached significant improvement of time accuracy. We observed that the best clock stability and performance had desktops with Pentium III processor due to low interrupt latency and its variation. Until now, we were not able to find a modern, rack-mountable computer suitable for next generation of NTP servers [Smo04]. All tested platforms had problem to achieve an acceptable clock stability due to rapid changes of the ambient temperature and significant variation of the interrupt latency degrading stability of the external time signal. This report describes a design and an implementation of our new NTP servers.

Our goal was to develop generic NTP server with following parameters:

best available internal clock stability,
effective processing of 1 PPS (pulse per second) signal,
high performance,
authentication support,
rack-mountable 1U case,
low power consumption,
acceptable cost,
optional front panel display showing server time and status.

4 Hardware

4.1 Mainboard

We tried to find a suitable mini-ITX format mainboard in order to avoid large heating mainboards.

We decided to use the VIA EPIA-EK, which is equipped with all standard interfaces including two Ethernet ports and four serial ports. The performance of the mainboard equipped by 1 GHz CPU is good enough for heavy loaded NTP server.

4.2 Oscillator

All i386 family boards have 14.318 MHz quartz oscillator from which both the main clock frequency (1.193 MHz) and the CPU clock (1 GHz in our case) are derived. Although the NTP package controls the system clock by a digital, kernel implemented PLL (Phase-Locked Loop), the short stability of the clock is done by the quartz oscillator stability. We can significantly improve the clock stability by replacement of the original crystal by another one with lower temperature dependence or by an ovenized oscillator (OCX). We have designed and manufactured an oscillator module (Figure) which is directly connected to the mainboard instead of the removed crystal. As the oscillator module contains a programmable frequency multiplier/divider, it can be controlled by OCX block having nearly any frequency, for instance 10 MHz. The oscillator module has the same size and position of mounting holes as the 3.5" hard disk and therefore it can be easily installed in a free disk slot. Detailed description of the OCX module and its parameters is out of scope of this report and will be done in a separate document.

Figure 1: OCX module

4.3 PPS capture card

The 1 PPS signal is traditionally connected to the DCD pin of a serial port. An edge of the signal invokes an interrupt and the interrupt routine assigns timestamp to this event. Unfortunately, the interrupt latency is not constant and its statistical distribution depends on the CPU load [Smo01]. An elimination of the interrupt latency requires some hardware support - there exist processors equipped by the programmable timers (e.g., AMD Elan SC520) but utilizing a specialized PCI card is more generic and reproducible solution.

Figure 2: PPS capture card

We have designed such an adapter - its brief schema is shown in Figure. The adapter contains an oscillator with frequency f, a counter and a control logic. The active edge of the 1 PPS signal starts counting of on-board oscillator pulses and causes a CPU interrupt. In the interrupt routine, the timestamp T₀ is evaluated and the counter register is read - its value n corresponds to the time elapsed since the last 1 PPS pulse. Knowing the actual time T₀ and the counter value n, the exact timestamp T of incoming signal is calculated:

T = T₀ - n/f

While the typical interrupt latency of a serial port ranges from 8 to 50 microseconds, the PPS capture card allows to decrease it down to 50 nanoseconds with 20MHz oscillator. Further reduction of uncertainty is not possible as counter reading requires transfer of at least 3 bytes on the PCI bus.

The designed PPS capture card has been implemented in cooperation with Tedia s.r.o. company as a customized FPGA program for their PCI cards PCD7004 and PCT7424. Device driver has been written by us and is available under the GNU GPL license.

4.4 Case

There are available many desktop mini-ITX cases but only a few of them are rack-mountable. The case EM-161LB, made by the company EMKO Case, a.s., perfectly fits for our NTP server. This case is 1U high and allows installation of two 3.5" disks and one PCI card. The position of the second disk is occupied by the OCX module. The case with all components is shown in Figure.

Figure 3: NTP server internals

4.5 LCD Display

Design of our NTP server integrates an optional 2x16 characters LCD display with HD44780 controller. The display is driven by the parallel port driver.

The first row shows name of the server, the flag indicating an active NTP process and the exact UTC time. The second row changes every 3 seconds and displays in a loop following information:

Stratum level of the NTP process,
reference clock identification - either an id of external clock in case of Stratum-1 or the IP address of the master NTP server otherwise,
time offset in microseconds,
clock stability in 10^-9 (i.e., ns / s),
server load - average number of NTP queries per second or per minute.

Figure 4: Status display

5 Software

5.1 Operating System

We run all our NTP servers under Linux. We use solely kernel version 2.4.x with the nanokernel patch, which implements internal clock with nanosecond resolution and supports an improved digital PLL and 1 PPS signal processing. Our long time experience shows that the nanokernel is stable and provides better clock stability than the standard kernel. Unfortunately, no such nanosecond clock is available in 2.6.x kernels.

5.2 NTP package

We use the latest NTP stable version (currently 4.2.4) that we have compiled from source code with NANO feature switched on.

5.3 Capture Card Driver

As the PPS capture card is implemented by a customized FPGA program which has been developed upon our request, we had to write our own Linux device driver. The driver uploads the card firmware, provides an access to card registers and implements the PPS API [Mog00] as a character device from which 1 PPS signal timestamps can be read. The driver is implemented as a kernel module.

5.4 LCD Display Driver

The LCD display is controlled by the LCD-Linux package. We wrote a set of scripts and C programs that collect status information and write it to the display. All these values except the server load are provided by NTP utilities, the server load is obtained from iptables counter. Refresh time of the status is one minute.

Our goal was to keep the displayed clock as accurate as possible - the time information is updated within a few milliseconds after beginning of every UTC second and can be used as a reliable and very accurate human-readable clock.

6 Server authentication

Authentication of the NTP server is an essential feature which protects the service provider from compromising by a forged system. The NTP protocol supports both symmetric and public key cryptography but only the second one is scalable enough to provide the service for large number of (possibly anonymous) clients. Several identity schemes have been implemented [Mills] - the Schnorr schema is the only one convenient for our situation. Unfortunately, the implementation of public key cryptography in NTP distribution is rather obscure, for instance it requires passphrase crypted certificates where the passphrase must be either shared or the service provider must offer a special utility to generate the specific client key for each submitted passphrase.

NTP client require special non-trivial configuration in order to verify the server authenticity. The procedure consists of following steps:

Create directory /etc/ntp and go there.
Choose the client passphrase, however it will not be secret as it must be given to the service provider. For testing purpose, it is good enough to use shared passphrase 'heslo'.
Generate the client RSA host key and the certificate:
```
ntp-keygen -H -m 1024 -p heslo
```
Ask service provider for the server public key encrypted by the selected passphrase (e.g., 'heslo') and store it into /ntp/key. Assume that the key file name is <key> and the server is <server url> (e.g., tik_key and tik.cesnet.cz).
Include following rows into the configuration file /etc/ntp.conf (first two lines must occur before any server specification):
```
keysdir /etc/ntp
crypto pw heslo ident iff
server tik.cesnet.cz autokey
[ generic: server <server_url> autokey ]
```

Create the symbolic link

ln -s tik_key ntpkey_iff_tik.cesnet.cz
[ generic: ln -s <key> ntpkey_iff_<server_url> ]

Restart the ntpd daemon.
Check the server status:
```
ntpq -c pe
```
and the authentication status:
```
ntpq -c ass
```

Figure 5: Frequency offset - crystal

Figure 6: Frequency offset - OCX

7 Conclusions

We have designed the new, highly accurate and stable NTP server in the rack mountable case. Main improvements is achieved by the OCX oscillator and the PPS capture card integration. Until now, we have built three prototypes of NTP servers, one of them is in routine operation since May 2007 as the primary NTP server tik.cesnet.cz (ntp.cesnet.cz). This server is referenced on the list of public stratum1 NTP servers. It has many thousands clients and replies more than 300 queries every second.

The comparison of former NTP server and server with an OCX oscillator is shown in Figure and Figure. We see how daily periodic changes of temperature in the computer room influence the clock stability.

References

[Mil92]	Mills D. L.: Network Time Protocol Specification. RFC-1305, IETF, March 1992.
[Mog00]	Mogul J., Mills D. L., Brittenson J., Stone J., Windl U.: Pulse-per-second API for Unix-like operating systems. RFC-2783, IETF, March 2000.
[Mills]	Mills D. L.: Autokey identity schemes. Available online.
[Smo01]	Smotlacha V.: Measurement of Time Servers. Technical report 18/2001, Praha: CESNET, 2001.
[Smo04]	V. Smotlacha: Experience with Precise Timekeeping in End-hosts. Technical report 18/2004, Praha: CESNET, 2004.