IBP Deployment Tests and Integration with DiDaS Project Lukáš Hejtmánek, Petr Holub 2003-11-23

Abstract

In this report we describe testing setup of Internet Backplane Protocol (IBP) and its integration with Distributed Data Storage project (DiDaS). We give a short overview of IBP followed by description of developer interfaces for IBP as well as end user tools for manipulating files stored in IBP infrastructure. We have created a library for developers that offers easy interface to IBP resembling traditional UN*X file operation calls and we have also implanted IBP capabilities into several end user software tools using this library. RAID arrays benchmarks and preliminary IBP benchmarks are presented as well.

Introduction

In late 2002 CESNET Development Fund decided to support a project for building distributed data storage called DiDaS as an extension of Grid activities of MetaCenter project . The DiDaS project aims to build large distributed storage capacity based on probably the most extensive project in this field: Global Distributed Network Storage developed at University of Tennessee, Knoxville , . Furthermore DiDaS project addresses support several pilot user communities that will use the storage infrastructure.

One of these pilot groups is a project developing Distributed Encoding Environment for multimedia processing,which will use both DiDaS storage capacity and MetaCenter computing capacity (esp. Linux PC clusters that are available at large now). Contemporary video processing tools suffer from low processing power and low disk capacity available on single computer. Building large multiprocessor systems with shared memory and large disk capacity attached locally is fairly expensive while video transcoding can be run using more cost-efficient PC cluster environment very well. PC clusters can be distributed across various sites (which holds true for MetaCenter PC clusters) and thus distributed storage capacity is very natural solution for storing video that is to be transcoded, as well as for storing intermediate temporary files resulting from transcoding process and final encoded video. Coordination between distributed storage and scheduling systems on some level is very promising as it allows either to optimize location of files with respect to available computing capacity, or vice versa to optimize location of computational processing with respect to where the data are available in distributed storage.

As a part of the DiDaS project we have developed a library called libxio that allows easy incorporation of IBP into existing UN*X applications. Resulting from collaboration between DiDaS and DEE projects we have enhanced a video processing tool and video rendering tool with capabilities of using IBP storage. In this report we also present an overview of IBP and libxio library from both developer's and end-user's point of view to give other groups and projects an opportunity to extend their tools to adopt IBP capabilities. We conclude with some important issues we have encountered while using these prototypes.

Network storage stack overview

Network storage stack that uses IBP consists of three layers: the IBP layer, L-Bone and Ex-Node layer, and LoRS layer. An overview of this stack is shown in table. An application built using this stack is thus interacting with LoRS layer only.

ApplicationLoRSL-Bone + Ex-NodeIBPNetwork storage stack overview

IBP

Similar to IP protocol for network connections, the IBP offers unreliable network block storage for data. The IBP uses soft consistency model with time-limited allocation thus operating in best effort mode. Basic atomic unit of IBP is a byte array providing abstraction independent on physical device the data are stored on. An IBP depot (server) is the basic building block of IBP infrastructure offering disc capacity. The IBP defines three classes of requests described below: allocation, read/write and management.

Allocation

When an IBP client needs to store data to some depot it allocates some amount of capacity first. Client may specify several attributes for the allocation:

IBP depot owner may specify capabilities of the depot so it might offer e.g. only volatile and time-limited allocations of size less then 10GB.

Read/Write

The IBP protocol has APIs for reading data from the server(s) and writing data to the server(s). Additionally it offers APIs for requesting data transfer directly between two IBP depots without involving the client that initiates the transfer in actual data transfer. This functionality allows to avoid unnecessary data transfers through network end nodes and to optimize transfers by utilizing fast backbone links that are more likely to be available between two IBP depots than between IBP depot and client.

Management

Allocation parameters and attributes may be changed and tuned during existence of allocation: the client can modify time-limit or increase size of allocation. However such request may be refused by IBP servers when no depots with requested parameters are available. The client is also able to receive information on current allocation status and both time and space constraints.

L-Bone and Ex-Node

L-Bone

L-Bone layer creates a compounding layer over particular IBP depots that is able to offer locations to clients meeting their requests to maximum extent. L-Bone servers form a directory layer based on LDAP mechanism. An example might be a depot that both meets client's requirements and is the geographically closest one (or the closest one from network point of view). Each created IBP depot must be registered to L-Bone server to be available for L-Bone server brokering.

Ex-Node

The Ex-Node is a construct remotely resembling UN*X I-node concept. It comprises particular IBP allocations and forms representation of a whole file. Individual allocations inside one Ex-Node neither need to be of same size nor need to form continuous block. The allocations inside one Ex-node may even be overlapping. The Ex-Node uses XML representation when saved (serialized) to local file. The Ex-Node keeps track of end-to-end services: each block may contain MD5 checksum and may be encrypted using DES, AES, or XOR algorithms. The blocks may also be compressed using UN*X compress algorithm available in libz.

LoRS

The layer of logistical tools called LoRS is built on the top of L-Bone and Ex-Node layers. It offers interfaces for locating set of IBP depots, APIs for reading, writing and overwriting of the data in the depots with given number of copies and number of concurrent TCP streams to be used. This layer also provides functions for reading and writing Ex-Node from/to the local file using XML representation as mentioned above.

Prototype implementation of IBP for DiDaS project

There is an initial experimental implementation of IBP infrastructure for Distributed Data Storage project (DiDaS) designed for preliminary tests and benchmarks of IBP infrastructure as well as for experiments with incorporation of IBP functionality into some pilot applications. Production setup will be optimized based on experiences gained with this prototype.

For preliminary tests and benchmarks we have two servers with internal disc arrays. One server has SCSI disc array and the other one has parallel ATA disc array. We are planing to expand IBP depots to seven servers distributed across academic high speed network CESNET2. We want to use both servers with external RAID arrays and servers with internal RAID arrays to compare both kinds of storage. We will offer about 6TB to 10TB of network storage interconnected with 1Gbps - 2Gbps network links.

In our experimental setup the server with SCSI disc array hosts LDAP server, L-Bone server and IBP depot and also provides web interface to the L-Bone server while other the one hosts IBP depot only.

End-user how-to for IBP in DiDaS

Up to now we have modified two applications based on requests of end user community. transcode program has been patched for the DEE project to work with IBP in computational cluster infrastructure so that the transcode program can load and store files from/to IBP depots. The second application is a media player Mplayer that is able to play the content directly from IBP depot. Besides these two applications there is a number of command line utilities available for the manipulation with files in the IBP depots.

If user wants to access file in the IBP depot then URI in form lors://host:port/local_path/file?bs=number&duration=number&copies=number&threads=number&timeout=number is used. User can also use the short form URI lors:///local_path/file. When user wants to read an IBP file, the local_path/file specifies local file where is stored Ex-Node with XML specification. When writing an IBP file the local_path/file specifies local file where to store Ex-Node with XML specification of IBP depots used. The paremeters in the full form of URI mean as follows:

There are environment variables that can be used instead of URI parameters for the LoRS layer.

At least LBONE_SERVER or host must be set.

transcode

transcode program can be used in almost traditional way. If user wants to store the file to the IBP depot then URI in form lors:///local_path/file is used instead of local file name. The environment variables should be set unless user specifies URI in its full form. The same form of URI can be used for loading the file from the IBP depot. Transcode can read and write ordinary local files as well. The prefix lors:// is required for accessing IBP, otherwise local file is used.

For example:

LBONE_SERVER=udomiel.ics.muni.cz; transcode -i lors:///video.dv.xnd -P1 -N 0x1 -y raw -o lors:///temp1-remux.avi.xnd -E 44100,16,2 -J resample

Transcode will use source file identified by serialized description of IBP Ex-Node stored in video.dv.xnd file on local disc in the current directory. The L-Bone server location is taken from environment variable (referring to udomiel.ics.muni.cz) and resulting serialized Ex-Node XML description will be stored in the current directory as temp1-remux.avi.xnd

Mplayer

Mplayer has been modified to use the same URI semantics as transcode. However we have encountered serious problem with reading latency from IBP when reading data and playing video in single thread (what is the way Mplayer internally works). More detailed description together with benchmarks can be found in second appendix.

Command line utilities

There is number of general purpose command line utilities available, which can be used for manipulation with files stored in the IBP infrastructure. Some of the parameters can be specified in ~/.xndrc file to avoid repeating specification of the same information when using command-line tools.

lors_upload

The lors_upload utility is used for storing files into the IBP depots. Basic usage is as follows:

lors_upload -f -H host -n my_file

The file my_file will be stored into the IBP depot returned by L-Bone server specified using host parameter. The -f option says that output Ex-Node file will be called my_file.xnd. The -n switch turns off any end-to-end services that can otherwise be DES (-e), AES (-a) or XOR (-x) encryption, compression (-z), or MD5 check sum (-l).

There are other options available:

lors_download

The lors_download utility is used for downloading files from the IBP depots. Basic usage is as follows:

lors_download -o my_file exnode.xnd

The data described by exnode.xnd will be stored to local file called my_file. If -o is omitted, the data is sent to stdout.

Other options are available:

-b and -t options have the same meaning as with lors_upload.

lors_trim

The lors_trim utility is used to decrease reference count for specified Ex-Node. If reference count is decreased to zero, the allocation is removed from the IBP depot. Basic usage is as follows:

lors_trim -o new_exnode.xnd -d exnode.xnd

This command means that the original exnode.xnd can be deleted while new_exnode.xnd is created. If -f switch is used then the original Ex-Node file exnode.xnd will be overwritten by updated Ex-Node.

lors_augment

The lors_augment utility can be used for increasing reference count for any particular Ex-Node. -c switch can be employed to determine how many copies are requested. We can use the same options as with lors_upload. If -f switch is used then the original Ex-Node file exnode.xnd will be overwritten by updated Ex-Node.

lors_modify

The lors_modify utility may be used to remove the specified capabilities from an Ex-Node. Following capabilities can be removed: read (-r), write (-w), or manage (-m). Note: there is no utility that adds capabilities back to Ex-Node.

lors_ls

The lors_ls utility lists allocation blocks for given Ex-Node(s). The listing includes offsets, lengths, and locations. Basic usage is

lors_ls exnode.xnd.

Developer how-to for IBP in DiDaS

We have developed a abstraction library called libxio that is now available for the developers. It provides standard UN*X I/O interface that allows to access local files as well as with files represented by URI lors:///local_path/exnode. The functions xio_open, xio_close, xio_read, xio_write, xio_ftruncate, xio_lseek, xio_stat, xio_fstat, and xio_lstat are available. These functions have the same semantics as standard UN*X I/O functions except for IBP files can not be opened in O_RDWR mode at the moment.

#include <xio.h>

int xio_open(const char *pathname, int flags);
int xio_close(int fd);
ssize_t xio_read(int fd, void *buf, size_t count);
ssize_t xio_write(int fd, const void *buf, size_t count);
int xio_ftruncate(int fd, off_t length);
off_t xio_lseek(int fildes, off_t offset, int whence);
int xio_stat(const char *file_name, struct stat *buf);
int xio_fstat(int filedes, struct stat *buf);
int xio_lstat(const char *file_name, struct stat *buf);

For accessing IBP infrastructure the libxio library uses lors library that provides interface described in subsequent subsections.

Creating a new file

lorsGetDepotPool(LorsDepotPool *dp, char *host, int port, IBP_depot *dpt, 
                 int maxdepot, char *location_hint, unsigned long storage_size,
                 int storage_type, time_t duration, int max_threads,
                 int timeout, int opts)

This function returns a list of appropriate IBP depots (dp) meeting given criteria: location_hint [city=Brno], storage_size (in megabytes), storage_type [IBP_SOFT|IBP_HARD] (volatility), duration (expiry time in seconds).

lorsExnodeCreate(LorsExnode **ex)

This function returns a newly created Ex-Node.

Opening a file for reading

lorsFileDeserialize(LorsExnode **ex, char *filename, char *schema)

This function opens an Ex-Node XML file called filename and creates a corresponding Ex-Node structure in memory. The schema parameter is currently unused.

lorsUpdateDepotPool(LorsExnode *ex, LorsDepotPool **dp, char *lbone_server, 
                    int lbone_server_port, char *location_hint, int nthreads, 
                    int timeout, int opts)

This function returns a list of appropriate IBP depots for given Ex-Node. L-Bone server may have NULL value in which case no location hinting will be used and IBP depots are selected randomly.

Writing to a file in IBP

lorsQuery(LorsExnode *ex, LorsSet **set, longlong offset, longlong size, 
          int opt)

This function returns a set (set) of allocations containing data block beginning at offset and having size length. LORS_QUERY_REMOVE for opt parameter is used.

jrb_empty(set->mapping_map)

This function returns true if and only if the set does not contain any data block. This means that data will be written to empty (unallocated) block.

Writing to an empty block

lorsSetInit(LorsSet **set, ulong_t data_blocksize, int copies, int opts)

This function returns a newly created empty set of blocks.

lorsSetStore(LorsSet *set, LorsDepotPool *dp, char *buffer, longlong offset,
             longlong length, LorsConditionStruct *lc, int nthreads, 
             int timeout, int opts)

This function stores set into the IBP depots. The parameter LorsConditionStruct is always NULL at this time. It is recommended to use value LORS_RETRY_UNTIL_TIMEOUT for the opts parameter.

Overwriting an existing block

First we need to assign the number of copies and block size to the particular set:

set->copies = number
set->data_blocksize = size

lorsSetUpdate(LorsSet *set, LorsDepotPool *dp, char *buffer, longlong offset,
              longlong length, int nthreads, int timeout, int opts)

This function overwrites existing blocks with new data. All parameters have already been described above. More precisely this function splits the existing allocation into pieces and decreases reference count of overwritten blocks.

lorsAppendSet(LorsExnode *ex, LorsSet *set)

This function adds set to Ex-Node that describes file we work with.

lorsSetFree(LorsSet *set, int opt)

This function frees set that is no longer necessary (e.g. when it has already been added to Ex-Node).

lorsFileSerialize(LorsExnode *ex, char *filename, int read_only, int opts)

This function saves Ex-Node into the local file. Setting parameter read_only to 1 means that we want to save read-only capability while write and manage capabilities will not be saved. The parameter opts must be always set to 0.

Reading from a file in IBP

We select a set from Ex-Node using lorsQuery() function. This set covers requested data blocks.

lorsSetLoad(LorsSet *set, char *buffer, longlong offset, long length, 
            ulong_t block_size, LorsConditionStruct *lc, int nthreads, 
            int timeout, int opts)

This function reads requested data up to length. On success, the number of read bytes is returned. Zero indicates end of file while negative value means error.

Closing a file

Ex-Node data should be saved using lorsFileSerialize() function before we close a file. We can free structures used by calling lorsExnodeFree(LorsExnode *ex) and lorsFreeDepotPool(LorsDepotPool *dp)functions.

Truncating a file

Again we start off by creating a set using lorsQuery() function. This set contains truncated blocks.

lorsSetTrim(LorsSet *set, longlong offset, longlong length, int nthreads, 
            int timeout, int opts)

This function decreases reference count for blocks beginning at offset having length of length. The parameter opts has to have the value LORS_TRIM_ALL.

Getting size of a file

The actual size of data that described by Ex-Node is kept in LorsExnode->logical_length variable.

Conclusions

Our first version of libxio stores data to the IBP depot with each single xio_write call. This results in the important latency issue that does not harm a lot while video transcoding but it also results in very large Ex-Node XML description. Each call adds approximately 400B to Ex-Node. We are using a write buffer of 10MB size at the moment. The buffer reduces latency and rapidly reduces the Ex-Node XML file at the cost of unnecessary memcpy call. However we believe that extra hit to extremely fast memory is a lot better then extra hit to relatively slower network.

Our preliminary tests have shown IBP storage as very suitable for DEE project. We don't not use any location hinting so far but as we are planing to spread IBP depots over Czech academic network, the location hinting and some form of integration with scheduling system will be necessary. As integration with scheduling system currently in use - PBS - seems to be rather tough task and new generation of scheduling systems with required capabilities seems to come rather shortly (e.g. from DataGrid Project ), we plan to do the integration of PBS and IBP on application layer so that application will give hints to PBS where the jobs should be scheduled to.

The experiments with Mplayer have been somewhat worse since the read latency has very negative impact on Mplayer. We can use read buffer as well but this only helps on fast network with low latency. The only chance is to start another thread that will try to read as big blocks as possible from IBP depots. The authors of IBP technology promise a new generation of IBP that will meet real-time requirements in better way. We will go on with modifying our library to use read thread until the new version of IBP is be developed.

Appendix A Software and Hardware RAID arrays benchmarks

Internal discs array was equipped with single Intel Pentium 4 Xeon 2.4GHz processor and 1GB RAM. On this computer was installed operating system Linux with vanilla kernel 2.4.22 with patch for XFS filesystem. Disc arrays were benchmarked with program that does sequential reading and sequential writing test to XFS partition.

We used parallel ATA Western Digital 250GB 7200RPM discs with 8MB cache, serial ATA Western Digital 250GB 7200RPM discs with 8MB cache and SCSI Seagate Cheetah 73GB 10000RPM discs.

We did performance test on single disc, four discs in software RAID 0 array, four discs in software RAID 5 array, four discs in hardware RAID 0 array and four discs in hardware RAID 5 array. In the case of hardware array we increased shared memory size to 1GB, vm.min-readahead to 128, and vm.max-readahead to 256 in kernel.

IDE RAID arrays

In this case, each disc was switched to master mode. The Adaptec AAR 2400A card has four parallel ATA interfaces. The 3ware Escalade 8506-8 card has eight serial ATA interfaces. While benchmarking these arrays we have found that Linux driver for Adaptec AAR 2400A is unstable as of kernel version 2.4.22 and should not be used for any production system. Following setup was used for the IDE RAID arrays benchmarks:

PATASingle discHW RAID 0HW RAID 5SW RAID 0SW RAID 5read51.823MB/sec84.359MB/sec47.902MB/sec95.111MB/sec51.902MB/secwrite50.785MB/sec78.330MB/sec15.349MB/sec42.989MB/sec48.616MB/secParallel ATA disc array benchmark

Parallel ATA disc array benchmark.

SATASingle discHW RAID 0HW RAID 5SW RAID 0SW RAID 5read45.273MB/sec146.437MB/sec58.186MB/sec182.129MB/sec80.294MB/secwrite52.663MB/sec119.643MB/sec24.719MB/sec115.402MB/sec67.601MB/secSerial ATA disc array benchmark

Serial ATA disc array benchmark.

SCSI RAID arrays

In this case, four discs were attached to one bus. Adaptec AIC card used posses only one internal SCSI interface. Following setup was used for the SCSI RAID arrays benchmarks:

SCSISingle DiscHW RAID 0 ASRHW RAID 5 ASRSW RAID 0 ASRSW RAID 5 ASRSW RAID 0 AICSW RAID 5 AICread66.846MB/sec125.564MB/sec124.893MB/sec102.670MB/sec104.319MB/sec258.230MB/sec200.974MB/secwrite61.191MB/sec114.030MB/sec42.864MB/sec56.046MB/sec36.956MB/sec169.391MB/sec89.744MB/secSCSI disc array benchmark

SCSI disc array benchmark.

Appendix B IBP read latency benchmarks

We have done several experiments with playing content directly from IBP depots. LoCI laboratory uses command line utilities together with Mplayer for playing content from IBP depots. For example:

lors_download my_file.xnd | mplayer -

We patched Mplayer so that it can handle Ex-Nodes on its own. While Mplayer runs in single thread mode (and author of Mplayer insists on the fact that this is the way to go) we must look after read latency from IBP depot. We have set up an experiment to show how the downloading speed and latency depends on the chunk size. We have done this experiment both on local network and the network whose connection is a little bit slower. We have used one-stream transfers only as multiple streams require multiple threads.

Chunk sizeDownloading speedSingle read call duration1B0.16KB/sec0.006sec2B1.09KB/sec0.002sec4B2.22KB/sec0.002sec8B4.44KB/sec0.002sec16B8.85KB/sec0.002sec32B17.73KB/sec0.002sec64B35.11KB/sec0.002sec128B70.30KB/sec0.002sec256B139.35KB/sec0.002sec512B270.71KB/sec0.002sec1KB486.38KB/sec0.002sec2KB973.24KB/sec0.002sec4KB1770.69KB/sec0.002sec8KB3049.94KB/sec0.003sec16KB4668.81KB/sec0.003sec32KB6526.62KB/sec0.005sec64KB8096.14KB/sec0.008sec128KB9201.35KB/sec0.014sec256KB10032.53KB/sec0.026sec521KB10451.11KB/sec0.049secLoRS speeds on local 100Mbps (FE) network

LoRS speed on local 100Mbps (FE) network.

LoRS latency on local 100Mbps (FE) network.

Chunk sizeDownloading speedSingle read call duration1B0.05KB/sec0.018sec2B0.11KB/sec0.019sec4B0.21KB/sec0.018sec8B0.41KB/sec0.019sec16B0.80KB/sec0.019sec32B1.70KB/sec0.018sec64B3.35KB/sec0.019sec128B6.65KB/sec0.019sec256B12.78KB/sec0.020sec512B26.50KB/sec0.019sec1KB49.75KB/sec0.020sec2KB95.08KB/sec0.021sec4KB183.11KB/sec0.022sec8KB317.41KB/sec0.025sec16KB538.05KB/sec0.030sec32KB812.89KB/sec0.039sec64KB1145.00KB/sec0.056sec128KB1436.87KB/sec0.089sec256KB1591.58KB/sec0.160sec521KB1572.75KB/sec0.326secLoRS speeds on foreigner 100Mbps (FE) network

LoRS speeds on foreigner 100Mbps (FE) network.

LoRS latency on foreigner 100Mbps (FE) network.

First we thought that LoRS layer creates that latency thus we did the same experiment with IBP layer directly.

Chunk sizeDownloading speedSingle read call duration1B0.16KB/sec0.006sec2B0.29KB/sec0.006sec4B0.62KB/sec0.006sec8B1.22KB/sec0.006sec16B2.43KB/sec0.006sec32B5.04KB/sec0.006sec64B9.82KB/sec0.006sec128B19.32KB/sec0.006sec256B38.15KB/sec0.006sec512B69.12KB/sec0.007sec1KB155.51KB/sec0.006sec2KB278.38KB/sec0.007sec4KB478.45KB/sec0.008sec8KB642.54KB/sec0.012sec16KB848.10KB/sec0.018sec32KB1049.23B/sec0.030sec64KB1241.19KB/sec0.052sec128KB1633.99KB/sec0.078sec256KB1637.22KB/sec0.156sec521KB1715.69KB/sec0.298secIBP speeds on foreigner 100Mbps (FE) network

IBP speeds on foreigner 100Mbps (FE) network.

IBP latency on foreigner 100Mbps (FE) network.

We compared IBP transfer speed with single stream ftp transfer speed using bbftp on the slower network.

Chunk sizeDownloading speedSingle read call duration1B0KB/sec0.005sec2B0KB/sec0.008sec4B0KB/sec0.008sec8B1KB/sec0.011sec16B0KB/sec0.047sec32B3KB/sec0.008sec64B1KB/sec0.048sec128B3KB/sec0.037sec256B28KB/sec0.009sec512B40KB/sec0.012sec1KB84KB/sec0.012sec2KB105KB/sec0.019sec4KB500KB/sec0.008sec8KB331KB/sec0.024sec16KB553KB/sec0.029sec32KB915B/sec0.035sec64KB945KB/sec0.068sec128KB1330KB/sec0.096sec256KB1460KB/sec0.176sec521KB1610KB/sec0.317secbbftp speeds on foreigner 100Mbps (FE) network

bbftp speeds on foreigner 100Mbps (FE) network.

bbftp latency on foreigner 100Mbps (FE) network.

MetaCenter Project, http://meta.cesnet.cz/ Beck M., Moore T., Planck J. S. An End-to-End Approach to Globally Scalable Network Storage. SIGCOMM 2002. http://loci.cs.utk.edu/ibp/files/pdf/SIGCOMM02p1783-beck.pdf Internet Backplane Protocol, http://loci.cs.utk.edu/ibp/ Logistical Computing and Internetworking Lab, http://loci.cs.utk.edu Linux Video Stream Processing Tool. http://www.theorie.physik.uni-goettingen.de/%7Eostreich/transcode/ and http://zebra.fh-weingarten.de/%7Etranscode/ Distributed Encoding Environment project, http://sitola.fi.muni.cz/projekty/strizna/strizna.html. Linux Media Player Tool. http://www.mplayerhq.hu Portable Batch System: OpenPBS (http://www.openpbs.org/) and PBSPro (http://www.pbspro.com/) EU DataGrid Project (http://www.eu-datagrid.org/) Disc I/O Benchmark Tool. http://www.iozone.org bbFTP -- Large files transfer protocol. http://doc.in2p3.fr/bbftp/