Mark Foster, Hugh LaMaster

January 1999

Currently, the NREN network is a two-level hierarchy, connecting routers via a full mesh of ATM switch PVCs. Although the PVCs are configured as a full mesh, the router ATM interfaces use Cisco’s point-to-multipoint feature. This feature permits a simplified router configuration since it effectively treats the ATM infrastructure as a pseudobroadcast medium: outbound traffic is replicated on each PVC of the interface. This note describes conversion of the router ATM interfaces to use point-to-point connections, and the motivation for doing so.

Why consider wrecking something that works? What we have doesn't fully "work".

For distribution of routing updates, the replication of traffic at the ATM layer hasn’t posed a significant problem. Currently, routing updates are multicast throughout the network. This process is implemented by Cisco’s multicast extensions to IP over ATM as "broadcast to all VC’s in a subinterface", including non-router workstation hosts that are logically part of the subnet. Although this mechanism sounds "wasteful", the link-state routing protocols generate a trivial amount of traffic relative to high-speed link bandwidth, so the wasted bandwidth is very small if that is the only use of multicast.

However, as NREN has deployed native multicast as a facility for distributing real-time data, including "Mbone"-style videoconferences, the amount of "gratuitous" traffic has grown substantially. Usually only a small number of other routers (1-3) actually need to receive the multicast data stream, based on the set of recipients for the stream, but the data are replicated and sent out every PVC in the subinterface regardless of this set of receivers. As a consequence, the aggregate multicast traffic can consume a significant portion of the available capacity on the ARC-Sprint connection. Just the ongoing Mbone multicast traffic (2-4 Mbps) consumes 15-30 Mbps, when replicated to each of the NREN backbone routers.

Traffic from NREN-ARC to other NREN routers via Sprint is currently limited by the available bandwidth into the Sprint ATM cloud: a single OC-3 port. Already, during periods of high multicast traffic, the ATM interface has experienced 100 Mbps rates for extended periods, saturating the link, and the router CPU. The router CPU saturation is a result of another feature of multipoint subinterfaces: multicast traffic forwarding is 100% process-switched, and does not take advantage of Cisco hardware-accelerated fast/express switching. It has been observed on a 7513 that about 1 Mbps of IP traffic over the ATM translates to about 1% CPU utilization. Thus, even if NREN had OC-12 service from Sprint, total multicast traffic would still be limited to about 100 Mbps using the existing routers.

Upcoming applications that will use NREN multicasting infrastructure have significant bandwidth requirements (sourcing 15-40 Mbps; perhaps more). If we attempt to push this much multicast traffic into the NREN backbone with the current configuration, we will quickly exhaust the OC-3 link capacity (even without consideration for bandwidth sharing with NISN on these OC-3 links). In fact, 40 Mbps would require almost three times the bandwidth we have available.

To contemplate supporting large bandwidth multicasting applications, we need to deploy a routing architecture that will replicate multicast traffic only where it is required—that is, only to routers where there are downstream receivers of the traffic. In a LAN environment, Cisco provides several options that make use of ATM signalling to establish point to multipoint PVCs precisely at the location of traffic divergence. In a WAN environment, the effectiveness of these options would depend on the ability to signal Sprint’s core ATM switches. This capability is not currently available from Sprint. Consequently, the core needs to be treated as a non-broadcast medium. In other words, the links between the routers must be configured as a mesh of explicit point-to-point connections.

In addition, Cisco has created the Multicast Source Discovery Protocol (MSDP), which will allow the backbone to be operated in Protocol Independent Multicast (PIM) Sparse-Mode instead of PIM Dense-Mode. Using PIM-SM eliminates flood-and-prune traffic and router state required by PIM-DM.

By converting the core to a point-to-point mesh, and by deploying PIM-SM with MSDP, the bandwidth requirements for multicasting on any particular backbone link will be dramatically reduced. We expect to see multicast capacity scale according to the scope of the receivers of the traffic, rather than according to the extent of the entire NREN network.

The following sections describe a suggested approach for implementing such changes. The process assumes a complete shutdown of all core traffic during the transition, with reliable out-of-band access for reconfiguration services. An alternative approach would involve building a second mesh of PVC’s throughout the network, to simultaneously support both the point-to-point links and the point-to-multipoint links.

Convert to point-to-point full mesh:

3.1. Define /30 address assignments mappings for each link

3.2. Document PVC mappings

3.3. Snapshot existing configurations, upload to tftpboot server

3.4. Modify uploaded snapshots to reflect new addressing, p2p links

3.5. Dowload modified configs & reboot with new configs

3.6. Verify OSPF functionality, connectivity for all p2p links

3.7. Verify BGP/MBGP and PIM Sparse-Dense functionality, external connectivity

Deploy MSDP, PIM-SM:

3.8. Designate Rendezvous Point (RP) for multicast group membership coordination

3.9. Enable MSDP at all MBGP border routers; run either MSDP peering, or, proxy for dense-mode neighbors (primarily at Chicago).

3.10. Verify PIM-SM multicast connectivity

Assuming we break 198.10.64/24 into a bunch of /30’s for the point to point mesh, the following general and specific examples show how the routers should be configured.

ospf general configuration

ospf specific configuration – ARC

bgp general configuration

pim specific configuration - ARC

!

special cases

The primary goal of the schedule is to compress the total reconfiguration time into two or three days, with a very high probability of success in the short period. Test procedures will be used to altert us to key difficulties that may arise during the transition.

During testing, a handful of problems were identified. These problems and their solutions or workarounds are highlighted here. Some of the problems relate specifically to the renumbering work, and some are more generic. When possible, a distinction(*) has been made for the renumbering-specific difficulties.

• Starting BGP with an active ATM interface may crash the router

• ATM multipoint subinterface removal requires reload (*)

• Cannot have p2mp subint’s with distributed route caching or distributed cef

• ami (ascii) version of cdb is inadequate

• Default routes are needed (*)

(*) We found maintaining out-of-band access cricital to performing the readdressing while reusing the existing PVC mesh. Th e only option for part of the work on the Fore switches is via the console port (dialin). Suitable access should be able to be maintained via ethernet or FDDI ISOLAN interfaces on the routers, providing that suitable routing tables are in place (the most obvious would be a static route back to a specific management workstation).