Synchronization of WWW proxy cache configurations

TEN-155 CZ technical report number 2/2000
also available in PDF, PostScript, and XML formats.

Pavel Satrapa
September 5, 2000

Introduction

Introduction of distributed WWW proxy caching system containing more than ten caches placed in various cities of the Czech Republic leaded to strong requirement of some smart mechanism ensuring synchonization of their configurations. It is assumed that the core caches will serve as parents for many local caches in various institutions or departments. If such a cooperation is to be put into operation, the change in configuration of core caches should be as simple as possible.

Main requirements also are:

configuration of whole system in one place
keep homogenity of all individual configurations

Basic principle

The whole configuration system is based on a central database of proxy caches. It contains all members of the cooperation - central proxy caches and the cooperating (local) caches too. There are three categories of caches:

core (backbone): The task of the system is to create configuration files for all the core caches.
partner: Caches in other institutions, which offer enough storage space and connection speed to make sense to cooperate with them bidirectionaly - objects are supplied and requested as well.
client: There the cooperation works in one way only. Client cache uses the core caches but offers nothing.

After every database modification the manager must run the make program. It creates configuration files for all the core caches containing information from the database, and subsequently transfers them to the caches.

The new configuration files are not activated on-line. I suppose, that this step is not so urgent to create some sophisticated synchronization mechanisms. It is sufficient to call a control program in regular intervals (in current configuration once per night). It checks if a new configuration was uploaded. If so, it creates the new configuration file and submits it to the squid.

Figure 1: Scheme of the configuration system

Data file

Data about cache servers are stored in a central file tencache.cfg. It is a text file in fixed format. Every field is put on a separate line. It has the following syntax:

name<tab>value

Collections of rows describing individual proxy caches are separated by empty rows. Following field types are defined:

name: Domain name of the proxy cache.
IP: IP address of the proxy cache.
type: Cache type (category). Acceptable values are: core, partner and client.
cont: Contact - name of the manager of the cache.
mail: E-mail of the cache manager.
phone: Phone number of the manager.
desc: Notes, additional information and so on.

Field name must be the first one for every proxy cache in data file. Only the first three fields are obligatory - name, IP and type. Example of a configuration file:

name    cache.ten34.ces.net
IP      195.178.64.58
type    core
cont    Vladimir Smotlacha
mail    vs@cesnet.cz

name    cache-lb.ten34.ces.net
IP      147.230.16.100
type    core
cont    Pavel Satrapa
mail    Pavel.Satrapa@vslib.cz

Generated configurations

It does not make sense to centraly create and distribute the whole configuration of every core cache. The core caches are serving their local academic communities, which means there are many specific configuration commands. For example the requests from local networks or another local caches are allowed, memory or disk space sizes differs and so on.

Consequently the configuration is divided into two parts - the automatically generated one and the local one, which is maintained manually by the cache administrator. The automatically generated part contains only the configuration commands describing the global relations. Exactly it consists of the following constructs:

set of cache_peer commands for core and partner caches (where it is possible to send requests)
definition of access list tencore containing all the core caches
definition of access list tenother containing remaining caches (partners and clients) allowed to send requests to the core caches

These configuration files are distributed to the corresponding caches using the scp command.

Activities on the configured caches

The automatically created configuration file part is stored on the proxy cache into the file automat.cfg placed in dedicated directory. This directory must contain also a file local.cfg with the local part of configuration. Commands defining adequate access rights for the tencore and tenother access lists are part of local.cfg.

Activation of configuration ensures the program makecfg. It is started in regular intervals using the standard cron daemon. Besides this it can be run manually by the administrator. Its working consists of these steps:

it joins automat.cfg and local.cfg into one file
it compares contents of the created file with the current squid.conf
if some differences are found, makecfg stores a backup of squid.conf into backup directory, replace it by the new one and using killall sends a HUP signal to the squid to load the new configuration

Configuration file /usr/local/squid/etc/squid.conf is fully replaced by the described mechanism. Consequently it does not make sense to make any changes in this file. The administrator has to change the local.cfg file and run the makecfg progam, if he wants to modify the cache configuration.