CESNET2 Network Deployment

CESNET technical report number 22/2006
also available in PDF, PostScript, and XML formats.

Václav Novák, Petr Adamec
5.1.2007

1   Abstract

This report describes the progress in CESNET2 NREN and results achieved in 2006. It includes optical DWDM and IP/MPLS network layers deployment and describes the current network status and network services as well.

Keywords: DWDM, ROADM, cross-border fibre, CzechLight amplifier

2   CESNET2 optical topology

The CESNET2 backbone network is based on utilization of leased dark optical fibers. The current CESNET2 optical topology is shown in Figure. The original concepts was primarily built with the optical EDFA amplifiers working with single-channel "gray" solution as the point-to-point circuits between the routers. From the beginning of 2004 we started to experiment with the DWDM (Dense Wavelength Division Multiplexing). The first pilot project was open on the optical line Prague - Brno. Later, after 1 year experimental traffic and getting more experiences with this technology we decided to rebuild the whole CESNET backbone. The DWDM technology allows us to run separate networks independently on the same fiber and to use higher capacity for bandwidth-intensive research applications. When in 2005, the DWDM technology made a big progress and the software-reconfigurable ROADM (Reconfigurable Optical Add Drop Multiplexer) was available, the dynamic reconfiguration of wavelength paths (lambdas), we continued with the improving our DWDM. Using this new elements in our system was important further step in increasing flexibility of backbone. With this CESNET can relatively simple make topological changes and to create multiple paths on its backbones for different applications. The IP network provided as fundamental service for all users became one of such application. It has one independent channel which is preserved from overloading causing by bandwidth-intensive applications. With the DWDM, the CESNET network has got new powerfull option to established new dedicated lines (lambda) on demand and to introduce system of provisioning services. Provision of lambda services is important for many types of applications which are not satisfied with basic shared packet-switched services provided by IP networks and would benefit from end-to-end optical circuits provided directly by the underlying optical infrastructure.

The main DWDM core optical transport network is based on Cisco ONS15454 MSTP system with the 2-way ROADMs. The main DWDM ring interconnects PoPs Prague - Brno - Olomouc - Hradec Kralove - Praha with the transmission capacity of up 32 optical channels in C-band at the speed up to 10 Gbps. In 2007 new optical spans were added, Prague - Pilsen and Olomouc - Ostrava - Cieszyn.

[Figure]

Figure 1: CESNET2 optical topology and technology (large image)

New spans are connected at the optical level to the main DWDM ring, so it is possible to provide wavelength paths between the arbitrary DWDM nodes. In this phase of DWDM deployment we are using a workaround solution with the optical patch cables (any wavelength to add/drop from span must be connected manually to ROADM node of the main ring). We plane to test and deploy three-way ROADM (based on wavelength cross-connects) or optical switches to achieve flexible wavelength provisioning within the whole DWDM system.

The full-featured DWDM system based on Cisco technology is complemented by static DWDM lines with the CLA (CzechLight Amplifiers) optical EDFA amplifiers and 4 - 8 channels DWDM Mux/Demux solution. It is very cost effective solution used on minor optical lines of our network. All elements of this "Czechlight" solution were developed within the Cesnet optical research activities. The elements are based on building kit composed from commercially available elements and modules. The CLA are designed for 10 Gbps wavelengths (includes Fibre Bragg Gratings). The solution applied the NIL (Nothing-in-line approach), so there is no inline amplification needed in the lines. Using this own developed technology, the CESNET could easily expand their activity into many new sites. With this technology were designed and deployed two CBF (Cross-Border Fiber) to Slovakia and to the Austria. The new line Brno - Bratislava brings direct connection between the CESNET2 network and Slovakia SANET and the line Brno - Vienna made direct link to ACONET (Austrian NREN). These connection allows make different type of networking experiments on different levels. We also extended the line Prague - Usti n. Labem using CLA solution and establish there a private wavelength connection between the hospitals data centers. All these lines are constructed as multiple lambda. It allows to use it independently for CESNET2 IP traffic and for particular applications, experiments or as a backup links.

3   CESNET IP/MPLS topology

The IP/MPLS CESNET network layer follows the optical transmission topology (see Figure). In the DWDM nodes of the main optical ring (Prague, Brno, Olomouc and Hradec Kralove) we run backbone network IP/MPLS routers as the P node elements within the MPLS network topology. In other PoPs are located access routers as the PE routers which provides the all functionality and services of backbone network (MPLS, EoMPLS, QoS, IPv4/IPv6 unicast, IPv4 multicast routing and NetFlow statistics). Both the P and PE routers are Cisco OSR 7609's with the SUP720-3BXL processors and 1GE and 10GE line cards.

The small PoPs without the MPLS functionality runs L2/L3 switches (Catalyst 3750G) as CE devices from MPLS perspective. The full network services capability is achieved by parent PE routers. There are trunks between PE and CE devices and VLANs configured. For the EoMPLS (Ethernet over MPLS) L2 services extension to the CE PoPs we use EoMPLS tunnels mapping into local VLANs.

[Figure]

Figure 2: CESNET2 IP/MPLS topology and external connections (large image)

The detailed IP/MPLS topology is shown in Figure. There are MPLS P routers in the core (marked as red) and PE routers (marked as blue). We run OSPFv2 as the IGP routing protocol in the MPLS core and iBGP between the PE routers with the route-reflectors on the internet peering routers R84, R85 and R98. The same route-reflectors are used for iMBGP (interior Multicast BGP) to exchange multicast routing information for IPv4 multicast and for IPv6 unicast routing as well. The IPv6 unicast is distributed over MPLS, so we run hybrid unicast IPv4/IPv6 using PE/6PE technology (dual-stack mode). The IPv4 multicast topology in non-congruent with the IPv4 unicast one (unicast packets are MPLS-switched whereas multicast is transported as pure IP).

The extension of the DWDM optical transport network to new CESNET2 sites invoked upgrading of several IP/MPLS PoPs (Pilsen and Ostrava) to 10GE. The next extension of DWDM span Olomouc - Ostrava to Cieszyn provided upgrade of peering connection with the Poland NREN PIONIER to 10GE as well. PIONIER is using this upgrade and new static CLA DWDM 10GE connection from Brno to Vienna for peering with the VIX (Vienna Internet eXchange). Related traffic travels CESNET2 backbone network using the EoMPLS tunnel.

4   Network security

Since 2005 we devoted special attentions to the network security. The CoPP (Control Plane Policing) was deployed to reduce the possibility of intrusions and helps to protect the backbone routers against to DoS attacks. The CoPP recently increased both the routers and network stability and reliability. As the next phase of the security projects we focused on the sophisticated systems for the network-wide data collection, analysis and anomally detection to protect the CESNET2 backbone network against security threats that endanger network perfomance. This includes rapidly propagating worms, distributed denial of service (DdoS) attacks and many others.

The infrastructure security system must be able to proactively detect and mitigate network-wide anomalies caused by threats. There is also expected to use methods which can automatically drop undesirable traffic at the edge of network based on the source or destination address.

The most used method is access-filtering on the edge routers. The access filters are commonly configured on many ports of many routers within the backbone network and it is very complicated to maintain it the reaction to detected anomalies and attacks. The large access-filters can negatively impact the router perfomance as well.

As the perspective solution we tested an implemented RTBH (Remote-triggered black hole). The RTBH filtering is a technique that uses routing protocol updates to manipulate routing tables at the network edge or anywhere else in the network and provides the ability to drop undesirable traffic before in enters backbone network. Source and destination-based RTBH filtering are both techniques that filter undesirable traffic by forwarding it to the Null0 interface. Null0, is a pseudo-interface that is always up and can never forward or receive traffic. Forwarding packets to Null0 is a common way to filter packets to a specific destination.

[Figure]

Figure 3: CESNET2 RTBH implementation (large image)

A trigger is a special device that is installed exclusively for the purpose of triggering a black hole. The trigger must have an iBGP peering relationships with all the edge routers (see Figure). In the CESNET2 network environment we conf igured the dedicated iBGP peers with the all edge routers instead to use route-reflectors (RR1, RR2 and RR3), so the RTBH router functionality is independent on the RR`s. We use BGP communities, the trigger sets the BGP community for a route and sends it to the edge routers using iBGP. The edge routers use a route map to match this community and set attributes locally, such as next hop and other routing metrics.

The decision-making process is pushed out to the edge of network as the flexible solution that can be used to selectively drop traffic. This allows for several levels of flexibility based on predefined BGP communities. The blackholing could be activated at the selected edge routers, at the local internet exchange point, at all international gateways, any eBGP routers. With the selective RTBH we can for instance activate RTBH only at international gateways (see routers R84 and R85 in Figure), which would in turn preserve the local connectivity via national exchange points. The disadvantage of RTBH is that all traffic to the target destination is dropped at the selected PE`s. The main goal is flexible blocking attacker traffic at the network edge from the RTBH router only (using static routes configuration) and simple maintance and control. The RTBH technique is supported by some systems for detection and mitigation network-wide anomalies, so it is possible to automatically block the attackers by control of RTBH trigger router in future.

For proactive detection and mitigation of network-wide anomalies we tested two systems in the CESNET2 environment, the Cisco CS-MARS (Cisco Security Monitoring and Response system and Arbor Networks PeakFlow SP.

The CS-MARS system operates at distinct and separate levels based on how much information is provided about the network reporting devices. At its most basic level, CS-MARS functions as a syslog server. As you add information about reporting devices, CS-MARS begins to sessionize the raw data, and after you configure additional reporting devices (e.g. NetFlow exports) and enable the more verbose reporting features, it presents a much more comprehensive view of your network, from which you can quickly drill-down to more specific information. The major drawbacks we encountered during the tests of CS-MARS are summarized as follows:

On the other way, the CS-MARS capabilities presents very useful real-time information about the current attacks/threats coming into the network, what are the devices under attacks and where the attacks are coming from. It supports strong sophisticated correlations and efficient data reduction make CS-MARS a very good choice to help track and mitigate most of the real world network security problems (including viruses spread etc.). CS-MARS is relatively easy to configure and manage with good tools to mask false positives in an easy way. When routers are configured to begin sending NetFlow v5 exports to CS-MARS, it starts baselining the network so it knows what `standard traffic' looks like. Thus CS-MARS can use the NetFlow data to know what type of traffic is an anomaly and it can send appropriate alerts to administrators so they can use the data to track down the infected system immediately. We considered this NetFlow traffic anomaly rule valuable, but unfortunately due to the lack of NetFlow v9 support we were limited to non-backbone exporting routers only at the final period of testing.

At the end of this year we tested the Arbor Peakflow SP device for security incidents monitoring and evaluation, real-time traffic checking and traffic anomalities evaluation. The device has been kindly provided by Arbor Networks. It allowed to monitor the information regarding running attacks originated from or targeted to our network. We ware able to concurrently monitor the anomalities in backbone traffic. And last but not least it allowed to monitor wel arranged and useful information about routing to individual autonomous systems.

We tested Arbor Peakflow SP in version 3.5.1. Netflow was sent from border routers R84, R85, R96 and R98. We discovered a problem in netflow v9 export from Cisco 7600 routers which does not send correct TCP flags. Hhereat we added single Cisco 7200 router to our test.

Testing clearly demonstrated advantages of this solution - particular in very good detection of DoS and DDoS attacks. For example one of the attacks targeted to one of our computers consumed large portion of our international connectivity (hundreds of megabytes) for indispensable time. The system is well developed regarding the lucidity. It allows direct output of access lists or rate limiters for most commonly used routers to stop the attacks immediately. Farther it is able to stop automatically the anomalities using the RTBH (Remote-triggered black hole), which we unfortunately did not test - we plan to do this test in the future. May be the only drawback is the configuration, which is complicated and not easy to take in. Grasping the

philosophy of this product is a bit hard. But the high-quality support simplifies the overcomming of these disadvantages. We suppose that the Arbor PeakFlow SP is perspective for deployment in the CESNET2 environment. We will continue with testing of new versions with the possible implementation in the future.

5   E2E services

The new type end-to-end (formerly E2E) guaranteed services is being developed within the GÉANT2+ research network activities. There are end-to-end service provisioning activities which are divided into three main areas: provisioning, performance measurement and direct support to deal with perfomance issues.

There are new technologies introduced like the Layer-2 ethernet services and optical transmission systems.

The NRENs are collaborating with the GÉANT2+ to distribute E2E services across the national backbone networks to the end user (research activities and advanced users). The CESNET2 researchers collaborates in the investigation of stitching scenarios including technical issues and necessary coordination works between the individual domains (communication channels between domains).

GÉANT2+ Prague PoP topology and CESNET2 connections to GÉANT2+ is shown in Figure. CESNET2 is currently connected by two 10GE LAN PHY interfaces, one for IP network and one for E2E services distributions. .

[Figure]

Figure 4: GN2 Prague PoP and CESNET2 connections (large image)

The GÉANT2+ Ethernet/SDH switches for E2E services supports EVPL (Ethernet Virtual Private Lines) on 10GE cards, which allows the mapping of ethernet VLANs into VCGs (Virtual Concatenation Group). The VCG members are transported as individual VCs (Virtual Concatenation, VCAT) across the GÉANT2+ SONET/SDH network layer.

CESNET2 NREN deployed L2/L3 modular switch with the support of pluggable optics and 10GE and 1GE interfaces. It provides E2E service aggregation based on VLANs funcionality. There are 802.1Q trunks configured with GÉANT2+ Ethernet/SDH switch.

There are following methods possible for E2E services distributions to the end users within the CESNET2 network:

Ethernet services support is limited to point-to-point connections only. The multi-point ethernet services such as VPLS are currently not supported for the HW limitations of the backbone routers. As the core facing 10GE interfaces in the PE routers we run X6704 type of cards which do`nt support VPLS. It is possible to upgrade to new 10GE interfaces based on SIP-600 modules, but this solution is very expensive. For this reason we don`t propose the VPLS deployment for the next IP/MPLS network development. More promissing is the support of the native ethernet services within the DWDM system on the new 10GbE Xponder cards which supports ethernet multiplexing within the DWDM using the VLANs.

The most critical part of E2E services delivery is the extension of these services from the nearest CESNET2 PoP to the end users. The most common solution is CWDM or DWDM Mux/Demux implementation on the local loops (to allow two or more independent optical channels for standard IP conectivity and E2E services). Very important is the end user support to help with the termination of E2E services in its network environment and with the overall E2E service design to meet its requirements. It includes E2E circuit termination (for example wavelength conversion to "gray" optics or UTP), IP addressing plane (the most applications use IP level) and operational support.

In the collaboration with GÉANT2+ network and German NREN DFN we deployed the E2E ethernet circuit at the capacity of 1 Gbps from Physical Institute in Prague (FZU) to Karlsruhe (LHC-OPN TIER2), which enabled the dedicated fast access of our physicists to the data of LHC (Large Hadron Collider) research project in the CERN. The load tests successfully finished (see statistics in Figure).

[Figure]

Figure 5: 1 Gbps E2E ethernet circuit FZU - Karlsruhe test load

Within the GN2 JRA3 research activity we collaborate with the GÉANT2+ network and Internet2 on the E2E trial Lousiana - Brno. The stitching scenarios cover many administrative domains and various technology to put together into one E2E channel (see Figure) :

[Figure]

Figure 6: Louisiana - Brno trial test topology (large image)

The main goal of this trial is to investigate possible stitching levels like the communication channels between the domains, adaptation layer within the domain and technology questions (determination of interfaces parameters and others). The trial results will be used by BoD (Bandwidth on Demand) E2E paths provisioning system, which is developed within the JRA3 to automate E2E provisioning.

6    CESNET2 future development

The DWDM optical transmission layer is the platform for the IP/MPLS network layer development as well the E2E services delivery.

In the 2007 we expect to achieve full SW wavelength provisioning in the DWDM core based on ONS 15454, expand the DWDM infrastructure to new PoPs and paths, set up native ethernet services (point-to-point and point-to-multipoint) within the DWDM system, test and verify "alien" wavelength transport at 10 Gbps speeds, continue with DWDM performance monitoring development and last but not least support pilot projects on E2E services for specific research activities.

In co-operation with the CESNET optical research group we will continue with static CLA DWDM development, testing and deployment in the CESNET2 network environment. The static CLA DWDM we use as very cost effective solution for the small PoPs connections to the main DWDM core and for the pilot research projects support, such as private wavelength connections between the hospital data centers. We expect the deployment of static CLA DWDM to build private optical hospital network (also called POSN).

The expected optical topology in 2007 is shown in Figure.

At the IP/MPLS network layer we expect to continue with the PoPs 10GE upgrade, topology optimization and provide large scale OS upgrades. We would like to finish the IPv6 multicast design and deploy it. We will concentrate on quality, reliability and security of the IP/MPLS network using new features of network protocols, MPLS Traffic Engineering Enhancement and continuous traffic monitoring. We also expect to test and optionally deploy some suitable system for network-wide anomalies detection.

[Figure]

Figure 7: CESNET2 expected optical topology in 2007 (large image)

The future development of CESNET2 optical and IP/MPLS layers will address E2E services provisioning. We will coordinate our effort in this area with the GN2 project, GÉANT2+ network and other NRENs. We will concentrate on:

References

[MSTP] Cisco Systems Inc.: Cisco ONS 15454 Multiservice Transport Platform (MSTP). Available online.
[NSC06] Novák V., Slavíček K., Cihlář J., Forghieri A.: Design and Deployment of CESNET2 DWDM Core Network. In: Proc. CESNET Conference 2006, Praha: CESNET, 2006, p. 43-53. ISBN 80-239-6533-6
[KRV05] Karásek M., Radil J., Vojtěch J.: Optical amplifiers in CzechLight and CESNET2. Customer Empowered Fibre Networks workshop, Praha, May 2005
[NS06] Novák V., Slavíček K.: Design and Deployment of Phase 4 of the CESNET2 DWDM Optical Transport Core Network. Technical report 15/2006, Praha: CESNET, 2006.
[NV06] Novák V., Vojtěch J.: Deployment of a DWDM System with CLA Optical Amplifiers in the CESNET2 Network. Technical report 25/2006, Praha: CESNET, 2006.
další weby:fond rozvojemetacentrumCzechLightpřenosyvideoservereduroameduID.cz