CESNET2 IP/MPLS Backbone Network Design and Deployment in 2008

CESNET technical report 20/2008
PDF format

Václav Novák, Petr Adamec, Pavel Šmrha, Josef Verich

Received 5.12.2008

Abstract

This paper describes recent changes and innovations of the CESNET2 IP/MPLS network layer in 2008. The article is divided into the following logical parts: Current Network Design, Planned Network Design and CRS-1/16 Implementation, Prague PoP Splitting, CESNET2 Network Resilience and Availability. The Current Network Design part deals with the actual IP/MPLS layer and the underlying DWDM layer topologies based on the Cisco 7600 and newly introduced Cisco CRS-1 core routers as well as the Cisco ONS 15454 MSTP DWDM platform. The Planned Network Design and CRS-1/16 Implementation part covers the deployment of the new advanced “carrier-class” Cisco CRS-1 core router using the Secure Domain Router (SDR) technology in the Praha PoP. The Praha PoP Splitting part describes the first phase of the planned physical splitting of the yet single Praha PoP into two geographically distributed PoPs Praha I. and Praha II. to ensure even higher network core resilience through further redundancy. The CESNET2 Network Resilience and Availability part covers several techniques for achieving high availability of network services such as introduction of new backup circuits as well as advanced protocol and configuration features including BFD (Bidirectional Forwarding Detection), NSF (Non-Stop Forwarding) with the SSO (Stateful SwitchOver) to achieve very rapid (subsecond) convergence in case of link or module failures.

Keywords: CESNET2, optical network, DWDM, MPLS, CzechLight

1  Current CESNET2 network design

CESNET Association operates a national DWDM and IP/MPLS network (CESNET2) covering multiple regions in the Czech Republic. The current CESNET2 network is built around a DWDM core providing 10 Gbps channels. It is based on Cisco ONS15454 MSTP system with the ROADM technology. The main DWDM ring interconnects PoPs Praha – Brno – Olomouc – Hradec Králové – Praha with a transmission capacity of up to 32 optical channels in C-band at the speed up to 10 Gbps. The ONS 15454 MSTP based DWDM system is complemented by another system based on family of open photonic devices called CzechLight (CL) that were developed by CESNET [3]. The key motivation for CL DWDM systems is a provision of cost efficient solution for the delivery/provision of gigabit (1-10 Gbps) connectivity to small PoPs and creation of backup connectivity for any PoPs based on pure optical circuits. A NIL (Nothing-in-line) approach, i.e. optical line design without the inline components (e.g. amplifiers and chromatic dispersion compensators) was used.

The current CESNET2 optical topology is shown in Figure 1.

[Image]

Figure 1. Current CESNET2 optical topology

The IP/MPLS CESNET2 network layer follows the optical transmission topology. In the DWDM nodes of the main optical ring (Praha, Brno, Olomouc and Hradec Králové). CESNET operates backbone IP/MPLS routers as the P node elements within the MPLS network topology. In major PoPs, there reside the access routers functioning as PE routers which provide all the functionality and services of the backbone network (MPLS, EoMPLS, QoS, IPv4/IPv6 unicast, IPv4 multicast routing and NetFlow statistics). Both the P and PE routers are Cisco OSR 7609’s with the SUP720-3BXL processors and 1 GE and 10 GE line cards.

The small PoPs without the MPLS functionality run L2/L3 switches (Catalyst 3750G) functioning as/representing CE devices from the MPLS perspective. The parent PE routers achieve the full network services capability. There are trunks between PE and CE devices and VLANs configured. For the EoMPLS (Ethernet over MPLS) L2 services extension to the CE PoPs, CESNET2 uses EoMPLS tunnels mapping into local VLANs.

The detailed IP/MPLS topology is shown in Figure 2. There are MPLS P routers in the core (marked as red) and PE routers (marked as blue). CESNET runs OSPFv2/OSPFv3 as an IGP routing protocol in the MPLS core and iBGP between the PE routers with the route reflectors on the Internet peering routers R84, R85 and R98. The same route reflectors are used for iMBGP (interior Multiprotocol BGP) to exchange routing information for IPv4 multicast as well as for IPv6 unicast.

CESNET2 supports hybrid unicast IPv4/IPv6 using PE/6PE technology (dual-stack mode). The IPv4 multicast topologies are non congruent (unicast IPv4/IPv6 packets are MPLS-switched whereas multicast packets are transported at IPv4/IPv6 layer).

[Image]

Figure 2. CESNET2 IP/MPLS topology at the beginning of 2008

The deployment of the core P and PE routers, Cisco 7609, as the main routing platform started in 2002. For current hardware and software features virtually all components have been gradually replaced by more recent and efficient ones (chassis, CPU, interface cards). In current configuration, routers are typically equipped with 10 GE interface cards with the throughput up to 40 Gbps/slot. With the demands on the properties and functions of the network and the increasing volume of traffic a number of problems arose, especially with the main Internet peering routers, such as high CPU load and thus longer BGP convergence, TCAM table overflow in the case of NetFlow v9 data export, the lack of TCP flags in NetFlow v9 records (which is a hardware limitation of the SUP720-3BXL or RSP720-3CXL CPUs). All these problems can negatively influence the backbone network performance and stability in the future. We have the following network research plans: to test and deploy new technologies, such as 40 and 100 Gbps, IPoDWDM principles, integration with the DWDM network, and other new advanced features. Therefore, we started the selection process of new and more powerful advanced routers for the main core of the network, which will enable further smooth development of CESNET2. The basic requirements were as follows:

In the first phase of the migration process we planed new router (s) installation in the Praha PoP. Current Internet peering and core routers R84 and R107 will be replaced (see Figure 2). The final solution could be based on two physical routers or one router with logical router support to configure the required PE and P functionality. The next phases will include the new router deployment in Brno PoP and 40 Gbps DWDM line Praha-Brno installation (over the current DWDM system). The expected final result with the planned physical Praha PoP splitting is depicted in Figure 3.

[Image]

Figure 3. New core routers planned implementation

Both new PE and P routers must be connected by at least 20 Gbps capacity. Considering the solution based on the logical routers (i.e. within one physical router), we require that control plane and dedicated HW and SW resources for each PE an P router must be separated.

The public tender winner was the solution based on Cisco CRS-1/16 router and logical PE and P routers configuration. Summary of the key features and characteristics of quoted CRS-1/16 configuration are:

More information about this hardware and its features is available on the manufacturer's web pages.

In 2008 we also focused to CESNET2 IP/MPLS network features and services improvement. The main areas of our interest were the network resilience and availability, IPv6 multicast and new QoS design and deployment.

The initial phase of QoS implementation in the CESNET2 IP/MPLS network was completed in 2005. The second network QoS deployment phase in the CESNET2 MPLS backbone was accomplished in 2008. It covered mainly service class restructuring, adding IPv6 support and unification of QoS configuration for new hardware platforms or modules (e.g. CRS-1 and Cisco 7600 ES20 modules). The basic goal of Quality of Service (QoS) is to provide service differentiation among packets in the network. Such a service differentiation is noticeable during periods of network congestion (i.e. in case of contention for resources) and results in different levels of network performance. The technical approach was to implement DiffServ within the MPLS-based CESNET2 backbone using the E-LSP point-to-cloud (destination unaware) QoS model for 6 classes of traffic aggregates with different QoS characteristics. To ensure the DSCP transparency for transported IPv4/IPv6 user traffic the so called “Short-Pipe” MPLS DiffServ tunneling mode was deployed. The QoS implementation for IPv4/IPv6 traffic currently uses 6 classes of service: Real-Time (EF-PHB), Network Control (AF-PHB), Video (AF-PHB), Critical Traffic (AF-PHB), Best Effort (default PHB), and Less than Best Effort (AF-PHB). The QoS admission control has been applied by means of policing at all ingress of the CESNET2 MPLS DiffServ and is available for academic user traffic. The detailed QoS design/redesign is described in detail in the technical report [4].

2  Planned Network Design and CRS-1/16 Implementation

In order to increase the capacity and stability in the current CESNET2 network we implemented the CRS-1/16 routing platform in the core Praha PoP. Performance upgrade was the main driver for implementing CRS-1 providing 40 Gbps/slot capacity allowing CESNET2 to deploy 40Gbps interfaces (OC768/STM256). The second objective was to simplify the architecture of Praha I PoP and reduce operational costs (with fewer devices to manage). This could be achieved by consolidating the edge and peering functionality on a single CRS-1/16-Slot node and implementing Secure Domain Routers.

[Image]

Figure 4. CRS-1/16 in Praha PoP topology

The migration process was divided into the following phases:

The CRS-1/16 “carrier-class” routing platform has specific requirements for the installation and environment compared to traditional routers. The chassis size is DxWxH 60x100x213cm, weight approx. 700 kg (with card included), max. power 11 kW, cooling about 32000 BTU/hr. Installed HW configuration included newest MSC card with lower power requirements, so the real CRS-1/16 power consumption is just about 5.5 kW. After powering up, the system boots for about 30 min (diagnostic tests, IOS-XR loading into all installed cards), so the very stable UPS is necessary. This is the basic design concept of CRS-1/16, because all the maintenance works (HW and SW upgrades) could?can be done in service.

The physical installation included OOB? equipment for all main and backup RPs (Route Processor) and DRPs (Distributed Route Processor).

The CRS-1 power system is fully redundant and consists of two AC power shelves, each with three AC rectifiers, and alarm module in shelf. The chassis power distribution system divides the chassis into physical ‘load-zones’. Whilst the power system is designed to provide resilience in the event of a failure, a double failure may result in critical linecards loosing power and hence in the node being cut off from the rest of the network. With respect to the physical network design and topology, it is very important to plan slot allocation for high reliability. Three factors have been considered:

The final allocation of PE-SDR, P-SDR and LC (Line Card) and power zone selection is in Figure 5.

[Image]

Figure 5. CRS-1/16 Chassis: Power Load Zones and LC/SDR/Slot allocation

The chassis is populated from the outside to the center starting with slot 0, 1, 6, 7 in upper PLIM Card Cage assigned to PE-SDR and 8, 9, 10, 13, 14 and 15 in lower PLIM Card Cage assigned to P-SDR. Slot 3 and 4 in upper PLIM Card Cage is reserved for PE-SDR DRPs and will not be used for linecards. As noted above, if one of the shelves or one of the rectifiers were to fail, there is sufficient capacity to ensure that components within each of the load-zones will continue to operate. However, to provide redundancy for a double failure of respective A and B rectifiers, the linecards will be spread across the different power zones, taking into account physical link redundancy and consistent slot placement policies across all chassis. The slot placement will ensure that at least one DRP and at least one SIP800 providing 10GE connections to redundant core sites will be operational under a "double rectifier failure" condition.

The IOS to IOS-XR Configuration Conversion process covered migration of R84 and R107 IOS configurations, adjusting the configurations according to IOS-XR specifics (Flexible NetFlow CLI, BGP RPL) and some new features in IOS-XR (LDP NSR, MPP) implementation. In the migration process specific CRS-1/16 features have been tested and verified before the implementation (i.e. IPv4/IPv6 unicast/multicast, BGP/OSPFv2/OSPFv3 protocols, NetFlow v9, MPLS and overall network compatibility and stability, and many others) No recent problems has been founded, so the migration process successfully finished without any impact on the CESNET2 network stability and operations. We experienced the following issues:

The CRS-1/16 installation and migration process successfully passed the NFRU tests, which verified compatibility with the IP/MPLS and DWDM ONS15454 network layers, routing system management and monitoring, and hardware and network resiliency features. All NFRU tests passed on the routing system connected to the live network under normal traffic load.

The resiliency tests covered following topics:

The status of CRS-1/16 and the entire CESNET2 network was monitored and checked. Packet losses were tested by iperf SW on the two Linux PCs connected by 10 GE to the both CRS-1/16 logical routers (PE-node and P-node). The test results demonstrated and confirmed the high availability features of the router.

3  Praha PoP Splitting

From the perspective of the CESNET2 network topology, the network core is resilient and fully redundant (see Figure 6). There are two PE Internet peering routers (R118 and R115) where main and backup connections to upstream ISP, to the NIX.CZ and GN2 network are connected. Peering routers are double connected to core P routers (R119 and R105), where the circuits from the CESNET2 PoPs terminate. Various optimization techniques are implemented for network protocols such as MPLS LDP, OSPF and BGP to accomplish very quick convergence.

[Image]

Figure 6. Current CESNET2 IP/MPLS topology

The main core routers are now located in the Praha PoP in the same computer room. The node is not resilient against rare outages like to power problems, cooling unit problems, etc. The same risk we experience with the key network servers (DNS, mail servers) which are resilient but again in the same computer room. Current Praha PoP encounters many limitations with the available space, power and cooling capacity.

Last year we started the preparation works to find the available housing in the Praha area to split this main CESNET2 PoP into two geographically separated ones. The repeated tender in 2008 was successful, so we started the PoP splitting planning and preparation. We expect to move PE Internet peering router R115 and P router of network core R105 to new housing location (Praha II. PoP). Key network servers and some backbone optical circuits termination from current PoP (Praha I.) will be also moved there. The PoP migration must be performed smoothly and is divided into several following phases:

  1. PE Internet peering router and P router installation in Praha II.

    • Other routers will be used in parallel with the current R115 and R105. As the PE new R114 (OSR7609-S) will be used with the more efficient RSP-720 processor. The P router R107 will be moved from Praha I. PoP (now is available after the migration to CRS-1/16).
    • Planned to finish by mid-December 2008.

  2. DWDM circuit Praha I. - Praha II.

    • DWDM technology will be used on the optical circuit between the PoPs. The main goal is to operate several data circuits on one optic fiber for various PoP interconnects (see Figure 7) to maintain the logical network resiliency and to separate core MPLS and temporary 802.1Q data circuits. 32 channels DWDM Mux/Demux and DWDM pluggable optics in the routers will be used.
    • Planned to finish by mid-December 2008.

  3. ONS 15454 ROADM node installation in Praha II. and its integration into main core DWDM ring

    • The most data circuits between the CESNET2 PoPs are provided by main DWDM optical transmission system based on ONS15454 platform. For a flexible move pf these circuits from Praha I. to Praha II. there is a need to install ROADM node in Praha II. PoP and to integrate it with the main DWDM ring.
    • Optical circuit Praha I. - Praha II. For DWDM integration it will be geographically different from the first one.
    • ONS15454 ROADM network simulation has been performed. New DCU-100 units and Praha-Hradec Králové link redesign will be needed for the correct DWDM operations.
    • Planned to finish in February 2009.

  4. Selected optical circuit and services migration from Praha I. to Praha II.

    • The circuits and services are provided by third party (Internet connectivity, optical lines lease).
    • It must be provided step by step with the provider, CESNET and housing center cooperation.
    • Planned to finish in April 2009.

  5. Smooth server migration

    • The key network segments will be extended to Praha II. PoP by dedicated data circuits and 802.1Q.
    • Beginning planned for January 2009.
[Image]

Figure 7. Praha PoP splitting - Phase 1

4  CESNET2 Network Resilience and Availability

CESNET2 network services availability and resiliency is the key motivation for the backbone network deployment. In order to accomplish high availability CESNET2 network as a whole, we have to consider all the network layers and technologies. It begins with the dark fiber topology (especially geographically diverse last miles to the PoPs), continues with the DWDM optical transmission technology and network, and ends with the IP/MPLS network. Redundancy and high availability are important features of the active network elements (DWDM nodes, routers, switches and others) and device service coverage with the short disturbance clearance time.

Deployment of new CL DWDM system on the optical lines Plzeň – Cheb – Most – Ústí nad Labem – Děčín, České Budějovice – Jindřichův Hradec – Jihlava – Brno and Letohrad – Opava (see Figure 1) allowed the backup circuits for these CESNET2 PoPs. All the PoPs above are now dual-home connected with the new PE routers C7606-S in Jihlava and Most PoPs. There are also express 10 Gbps backup optical circuits České Budějovice – Brno and Plzeň – Ústí n. Labem Labem included in the main network core.

The replacement of the main PE Internet peering and P core routers by new high availability and high performance routing system CRS-1/16 recently increased the availability of network services and overall network stability. There are many advanced features implemented in the modular IOS-XR which guarantee high availability:

The rapid convergence of the running protocols used in the backbone network is a key for network availability and resilience guarantee . In CESNET2 IP/MPLS network there are MPLS LDP, OSPF and BGP protocols, which have to rapidly converge in case of line or router failure, to reroute the forwarded traffic.

In 2008 we tested and implemented rapid/subsecond convergence of MPLS core backbone network CESNET2 (typically below one second) for the eventual failure of the line/router through rapid detection of disorders by using BFD (Bidirectional Forwarding Detection) protocol.

The BFD is a detection protocol that is designed to provide fast forwarding path failure detection times for all media types, encapsulations, topologies, and routing protocols. In addition to fast forwarding path failure detection, BFD provides a consistent failure detection method. Using BFD to detect forwarding path failures at a uniform rate, rather than the variable rates for different routing protocol hello mechanisms, network profiling and planning is easier, and reconvergence time, is consistent and predictable.

We configured BFD with the parameters of 3 x 0.1 sec using OSPF configuration / LDP Session Protection and optimization of the internal parameters of OSPF time increment, while the use of synchronization protocols OSPF, and LDP to avoid the transition phenomenon route to the "black hole" (blackholing avoidance) during the period of convergence of the two protocols.

It also increased the availability of the network (and its resilience to failures) with the implementation of NSF (Non-Stop Forwarding) with the SSO (Stateful switchover) for the Cisco OSR 7609 routers with redundant supervisors for the LDP, OSPF and BGP protocols. Using NSF with SSO we get the ability to route packets when primary RP crashed and a router is being switched to the secondary one (RP failover) while keeping routing state information (graceful restart).

This feature minimize (or completely eliminates) the forwarded packet loss (routing is typically interrupted for a period of about 0 - 3 sec on the affected Cisco 7600 router) to strongly enhance the stability of the network, since the original routing state information is replaced after the successful resynchronization with the NSF neighbors.

5  Future plans

In the next period we plan to finish the Praha II. PoP migration to improve the CESNET2 network availability even further. This includes a new ONS15454 ROADM node installation and integration into the main core DWDM ring and moving selected data circuits to this new PoP. The CESNET2 network topology must be adapted to the new PoP to ensure better network services availability.

The CRS-1/16 routing platform implementation in the PoP Brno will be our main goal in the near future. This is a fundamental prerequisite for the future testing and implementation of 40 Gbps DWDM link between Praha and Brno PoP. Finally, the new CRS-1/16 will replace the currently overloaded peering router R98 (OSR7609). We also plan to start the Brno PoP splitting process (both at the IP/MPLS and ONS15454 DWDM network layers).

The CL DWDM system is going to be delivered on the fiber optic lines Pardubice – Hradec Králové (as the extension of the current line Praha – Pardubice) and Brno – Zlín – Olomouc. The planned optical channels capacity is 10 Gbps and the IP/MPLS Zlín PoP will migrate to 10 Gbps. We further plan to upgrade the single-fiber lines Ostrava – Opava and Ostrava – Karviná to CL DWDM technology (1 Gbps channels only).

References

[1]  NOVÁK, V.; ADAMEC, P. CESNET2 Network Deployment. In LHOTKA, L.; SATRAPA, P. (ed.). Networking Studies. Praha: CESNET, 2007, p. 3–14. ISBN 978-80-239-9285-4. Available online.
[2]  NOVÁK, V.; SLAVÍČEK, K. Design and Deployment of Phase 4 of the CESNET2 DWDM Optical Transport Core Network. In LHOTKA, L.; SATRAPA, P. (ed.). Networking Studies. Praha: CESNET, 2007, p. 15–25. ISBN 978-80-239-9285-4.
[3]  NOVÁK, V.; Josef Vojtěch: Deployment of a DWDM System with CLA Optical Amplifiers in the CESNET2 Network. In LHOTKA, L.; SATRAPA, P. (ed.). Networking Studies. Praha: CESNET, 2007, p. 27–37. ISBN 978-80-239-9285-4.
[4]  ŠMRHA, P.; VERICH, J. QoS Design and Implementation in the CESNET2 MPLS-based Backbone. Technical report 17/2008, Praha: CESNET, 2008.
další weby:fond rozvojemetacentrumCzechLightpřenosyvideoservereduroameduID.cz