NASA Research & Education Network / Recom Technologies

NASA Ames Research Center

September 1999

(submitted to IEEE Comsoc for publication consideration)

NASA’s Research and Education Network is in the process of configuring and deploying monitoring and measurement platforms for performance analysis. In particular, we wish to contrast IP flows that are given preferential treatment over other competing IP flows. The mechanisms we plan to use incorporate passive optical splitters and low-cost PC-based packet capture. This article describes some of the motivations behind the measurements and the key elements planned for our measurement systems.

With the deployment of network services that give preferential treatment to certain traffic comes the need to monitor and measure the effectiveness of this treatment. One project that is expected to need such monitoring is NASA’s Information Power Grid (IPG) [1]. Some of the IPG applications will use capacity that is specifically designated for flows generated by those applications, so that desired performance levels can be achieved. By deploying a monitoring and measurement platform at strategic locations in the network, the capacity available for the preferred flows relative to other background flows can be demonstrated. These measurements can be done by passively monitoring all traffic on a link, then generating a summarization of the amount of traffic for specific types (or classes) of flows. The overall goal is to confirm the availability of the reserved resources with a mechanism that is completely independent (and out-of-band) of the applications.

While the IPG project is a good candidate for such monitoring, we anticipate a number of other application environments will be able to benefit from our measurement system. Previous and ongoing work has demonstrated the general benefit of similar passive monitoring solutions. Both the MCI/NSF very-high-performance Backbone Network Service (vBNS) project [2] and NSF-sponsored National Laboratory for Applied Network Research (NLANR) in concert with the Cooperative Association for Internet Data Analysis (CAIDA) [3] have made significant contributions toward measuring and understanding emerging high performance networks. By distinguishing traffic based on the Differentiated Services (DS) field marking, we expect our system to provide valuable feedback for a variety of Quality of Service (QoS) deployments. One deployment that plans to incorporate mechanisms using the DS field is the Internet2 QBone Project [4]. We anticipate that the success of the QBone work will encourage similar coordinated use of DS field marking for other QoS projects.

This marking involves setting the value of the DS field. The DS field is part of the IP header, and in the context of IPv4, is a portion of the TOS octet [5]. The values are used to define what type of special treatment, if any, the packet is to receive from routers along a path. In both the IPG and QBone efforts, there will be one value used for preferential treatment, and another for best effort treatment. RFC 2474 [6] provides a complete description of how the DS field is specified and how such packets are treated on a per-hop basis.

In addition to actually monitoring and summarizing the different traffic flows, a handful of other features will be provided. This system will include an intuitive (graphical) display in near real-time. By doing so, the behavior experienced by a given application can more readily be correlated with the observed network performance. If a change is made in one of the resource parameters, the application (and any competing network traffic) should notice the change, and the measurement system should indicate the change in close temporal proximity. The ability to perform retrospective analysis will be helpful for some applications, so the system will also have mechanisms for recording and digesting long-duration measurements (hours and days). The storage implications of such recording may be high, thus coalescing some data points (reducing granularity of the measurements) may be required.

The particular elements we expect to include in the measurements are:

• Total bandwidth available (overall link speed)
• Preferred traffic bandwidth values (maximum reserved, minimum reserved)
• Best Effort traffic - bandwidth used
• Preferential treatment flows – bandwidth used
• Latency (round trip time)

There are a number of projects focusing on characterizing network traffic and network performance. Highlights of some of these projects are given below. The ones mentioned here have some elements and/or ideas that we expect to incorporate into the measurement activities described in this article. See Table 1. for reference URL’s.

CoralReef is a CAIDA effort. It evolved from an MCI/vBNS project that monitors the OC-3 and OC-12 activity of vBNS network. This work was based on measurement and analysis tools assessing active flows for Ethernet and FDDI, and was developed at San Diego Supercomputing Center under the NLANR project (now NLANR Measurement & Operations Analysis Team).

CoralReef provides a rich set of functionality and interfaces. CoralReef is a network traffic collection, summarization, and analysis system. A PC running FreeBSD is used to passively collect ATM cells, and provide flow statistics or packet traces of the traffic. The packet capture, filtering, and preliminary storage is currently intended to all be done on the same PC host. The CoralReef project has ongoing development and enhancement activities. Most of these enhancements are intended to provide a more modular system and to incorporate visual data analysis capabilities.

Surveyor is a measurement infrastructure that is being deployed at participating sites around the world. Based on standards work being done in the Internet Engineering Task Force (IETF) IP Performance Metrics Working Group, Surveyor measures the performance of the Internet paths among participants. The project aims to create the infrastructure and tools that will improve the ability to understand and effectively engineer the Internet infrastructure. The project is sponsored by Advanced Network & Services and participating organizations.

The NLANR active measurement project (AMP) is undertaking site to site measurement across the vBNS. This work is intended to compliment the measurements taken by MCI within the vBNS infrastructure. Currently round trip times, topology and loss are being measured.

Surveyor and AMP are both active measurement systems. Being "active" means they send packets in-band, and make measurements based on timing of the transmitted packets. Surveyor provides a full mesh of one-way delays between all the Surveyor probes. AMP provides a full mesh of one-way delay, timing, and loss characteristics between all the AMP probes. The storage of the continuously collected data with these systems can pose an problem as the scale of the mesh and frequency of sampling increase. However, the collected data provide very precise current and historical measurements for judging the evolving capabilities of the networks being measured.

The Waikato Applied Network Dynamics research group is a multifaceted project at the University of Waikato, New Zealand. WAND is performing passive traffic measurements on ATM networks, does traffic modeling with a simulator, and performs measurements of IP packet delay and loss on the Internet using the GPS system for timing to and from various international sites. These measurement activities are similar to the ones previously discussed. They have also developed an inexpensive PCI "DAG" card that can be integrated into ATM and Packet-over-SONET measurement platforms. Some of their ongoing effort is targeted toward validation and improvement of the measurements, and extending the speed and types of interfaces supported on the DAG cards.

Written by Nevil Brownlee at the University of Auckland, New Zealand, the Network Traffic Meter (NeTraMet) is a system which collects data about network traffic on a specific link. Similar to the CoralReef software, this runs on FreeBSD (and a wide number of other UNIX systems), and is able to collect information on designated flows using both local and distributed elements. NeTraMet is being integrated into the OCXMON platforms, and it has been used to produce a network usage accounting package called ipmeter. NeTraMet also includes support for retrieval and summarization of Cisco Netflow information. The NeTraMet software is at the core of our monitoring system and is discussed in greater detail in the NeTraMet Summary below.

Table 1. References to Related Projects

NASA’s Information Power Grid project is devised as a way to integrate distributed large-scale computing resources, making the collection (or "grid") of resources more readily available to researchers and their applications. The resources are comprised of computational systems, data repositories, scientific and instrument systems, and human collaborators. To solve some of the complex problems faced by these collaborators, the relevant resources need to be integrated to focus on a single problem. This integration involves significant coordination, development, and deployment of middleware, system administration, and application development environments among the cooperating facilities. While the primary motivation for the IPG effort is support of NASA’s aerospace research and development, the overall goal is to support scientific and engineering work in many other fields.

A significant component of the IPG project will be the network that provides bandwidth and performance capabilities to interconnect the distributed facilities and the applications that use them. Some of these applications will have the ability to reserve portions of bandwidth and use traffic classification to achieve the performance needed. While there will be some performance instrumentation incorporated into the services available to the applications, an independent, out-of-band mechanism is needed to better validate the efficacy of some of the reservation and priority mechanisms.

NASA’s component of the Next Generation Internet program is the NASA Research and Education Network (NREN). NREN currently provides high speed interconnections between many of the NASA research facilities, including those that are part of the IPG project. Both NREN and the IPG project are supported through NASA’s High Performance Computing and Communications program. Thus, NREN is actively involved in the IPG project, and will be configuring key parts of the initial network supporting IPG activities.

The monitoring and analysis of the IPG network traffic can be done at various points throughout NREN using passive monitoring devices that have been designed to interface with some of the IPG services. These devices incorporate NeTraMet software running on FreeBSD PC’s.

NeTraMet is an implementation of the IETF Realtime Traffic Flow Measurement (RTFM) Working Group’s measurement architecture. The architecture specifies what information a traffic flow measurement system needs to collect and describes the requirements and tradeoffs for an implementation. A detailed description is given in RFC 2063, "Traffic Flow Measurement: Architecture". In essence, there are three components to this architecture:

• Meters
• Meter readers
• Managers

Meters are lightweight entities (a process in a switch, router, or small host) that measure specific attributes of traffic (e.g., numbers of bytes) flowing on a particular network segment. The meters provide a way of concisely aggregating usage information for given flows. In this context, a flow is a sequence of packets distinguished by source, destination, and start/stop times. In particular, IP flows are ultimately defined by rules that specify the attributes to be used to coalesce collections of packets. Examples of such rules are given below.

Meter Readers reliably retrieve usage data from meters, making it available for analysis on a network management system.

Managers are used to configure the meters: they load into the meters rule sets that are used to define which flows to measure. The Managers also control the Meter Readers, indicating which meters to collect data from, and at what frequency to do the collection.

Currently, NeTraMet combines the Meter Reader and Manager functionality into a single entity. The Meter remains a distinct process. Communication between the Manager and the Meter is via the Simple Network Management Protocol (SNMP) [7,8]. In the current implementation, particular attention has been given to making the communication very efficient, and the retrieval of bulk data low overhead. The system also implements the Traffic Flow Meter MIB, as described in RFC 2064.

The architecture, and the NeTraMet implementation of it, are flexible and modular. Use of the same mechanisms for network-level and application-level measurements offers the potential for generic analysis applications which can effectively correlate various layers of traffic and usage. Furthermore, with the measurement data available via SNMP requests, existing SNMP-capable tools can be exploited.

In this section, we detail the capabilities and overall architecture planned for the NREN measurement and monitoring system.

The monitoring system will have the following general capabilities:

• Passive traffic observation with high-speed (10Mbps or faster) out-of-band access for data retrieval. The passive capability can be achieved using optical splitters at strategic points in a fiber-optic circuit. Passive monitoring avoids impacting or altering the characteristics of the applications being monitored.
• Dynamically alterable criteria being used to classify and summarize traffic flows. By being dynamic, the traffic classification can be more readily adjusted to the existing traffic.
• The ability to retrieve collected data from multiple locations simultaneously. For any given measurement, we expect to have at least two (and often three or four) monitoring systems active. Presenting a reasonably accurate display of the distributed measurements requires concurrent on-demand retrieval.
• Near real-time graphical display of the collected data.
• Archive of collected data for possible post-processing or future analysis.
• Reliable, remotely manageable hardware that can easily be reconfigured for different layer 2 interfaces (e.g., ATM or Packet over Sonet). It would also be beneficial to be able to hot swap mass storage devices in the field, to better support major operating system/software updates and cope with mass storage device failures.

In addition, there are capabilities needed which are specific to the IPG project:

• Communication with the IPG Resource Manager for determining the reserved bandwidth values (maximum, minimum), and the DS field values designated for the preferred traffic flows.

An example deployment of a monitoring platform is detailed in Figure 1. We insert an optical splitter at a point where the desired traffic can be monitored. The splitter provides separate access to signals traveling in opposite directions, while also transmitting the signal on the primary path. The signals split off are sent to a monitoring PC that has two ATM interfaces. Each interface is used for only one direction of traffic. For example, ATM-1 handles traffic flowing left to right, and ATM-2 handles traffic flowing right to left.

Figure 1. Example monitoring platform deployment.

The PC runs a NeTraMet meter process [9] that collects and processes IP packets according to rules it has been provided. This meter process can send and receive data and configuration information (rules) via SNMP. The NeTraMet manager process runs on a separate Unix platform, and is used to control the actions of each meter process, downloading rules into the meters, and retrieving information from the meters. This organization is partially diagrammed in Figure 2 and detailed in RFC 2063 [10].

Figure 2. Traffic measurement elements.

In addition to incorporating the Realtime Traffic Flow Measurement features, Figure 2 also shows elements needed for the NREN deployment. The storage, processing, and display of data provide a way to do analysis of the traffic flows being measured. An intuitive graphical display will allow the users of the system to more effectively control what flows are being measured, and how the measurements are displayed.

Some development and integration work is required to provide the desired functionality of the monitoring platforms. This effort focuses on early deployment and demonstration of the basic elements followed by increasing functionality and utility. Changes are needed to the NeTraMet code, and rules specifying what we want NeTraMet meters to capture must be specified. Graphical representations for text-based input and measurement results will be deployed, and a mechanism for integration into the IPG infrastructure will be created.

NeTraMet (version 4.3), as distributed, is unable to recognize the encoding used for reassembled IP packets from the FreeBSD Midway ATM driver (i.e. Chuck Craynor’s BSDATM driver). This driver encodes IP over ATM using the methods detailed in RFC 1483 [11]. We have adapted NeTraMet so it can recognize this encoding for IP packets.

The next step is to construct NeTraMet rule sets that are appropriate for performing the traffic summarization already discussed. The rule sets are composed of lines of the form:

attribute & mask = value : action, index

where the attribute, in general, specifies a field in the IP header, the mask is used to discriminate the significant portion of the attribute being examined, and the value is compared with the masked result. If the value matches, then a specified action is performed. The index is a parameter for the action, and is often a label that is the target of a branch or loop. This description is a simplification of genuine rulesets. The following examples should provide a somewhat more complete picture.

A simple case for simply counting all the IP traffic (grouped in blocks where 24-bits of the address are the subnet):

The Simple Ruleset Language (SRL) developed by Nevil Brownlee provides another way to specify NeTraMet rules. The srl compiler included in NeTraMet translates SRL programs into rulesets that are suitable for use with the Manager and Meter entities. We’ve found the SRL programs a little easier to read, particularly when the complexity of the specification increases.

In this example, the DS field is referenced via the NeTraMet attribute DSCodePoint.

DSCodePoint is a single attribute for a given flow that can be used in two ways:

(a) Split up traffic of a flow into its various DSCodePoints, i.e. break it into sub-flows for each distinct DSCodePoint value.

(b) Look at total traffic only, splitting it into DSCodePoints.

The following is an example SRL program that does (a) for a subnet 198.10.0.0:

Note that in both examples, there is a format statement; this statement defines which attributes are to be included in each output record, and how they are to be separated; additional white space can be used to improve (human) readability.

Once the system is able to effectively count the separate flows, we will use scripts that process this data to produce graphical elements (i.e., GIF images) that can easily be integrated into web pages. A prototype of such a graphic is given in Figure 3. In this example, the best effort traffic can only exceed the utilization of the reserved flow traffic when the maximum reservation is reduced, thereby restricting the resources used by the reserved flow traffic. Once this limit is returned to normal, the best effort traffic restriction is resumed.

Figure 3. Display of Traffic Flows

A useful improvement to text-based rule set generation would be to provide a web form that could be used to define the variable details in the rule sets (e.g., DS field values, source and destination addresses, etc.). The output of this form would then be integrated into a new rule set that would be subsequently transmitted to each affected meter process.

Programmatic interfaces will be defined and implemented to provide a two-way communication between the measurement system and the resource management system used by IPG applications. At the most basic, this mechanism will allow the measurement system to retrieve the current bandwidth settings, and then communicate the flow measurements back to the resource management system. In the intial design, this won’t be intended to provide closed-loop feedback sufficient to immediately effect changes, but instead will provide advisory information to the application execution environment.

Enhancements to the measurement system will be needed as the uses expand. These uses are likely to include higher speeds, IP packet-over-sonet, and paths with asymmetric delays.

End-to-end latency measurements will be performed using NTP-synchronized clocks. To more accurately measure this latency, addition of off-the-shelf GPS interfaces could be used to establish more accurate timing (< 1 m sec). The Surveyor, AMP, WAND, and similar projects all use forms of GPS-based clocking to provide very effective one-way synchronization and measurements.

The University of Waikato, Auckland, New Zealand, has been developing inexpensive PCI cards that are microprogrammable, and thus, can very flexibly support higher signalling speeds and alternative framing mechanisms. To accommodate OC12 and above, and to accommodate POS framing, these "DAG" cards are likely to be needed for the measurement platforms we deploy.

Multicast support has not, at this point, been planned for integration into the monitoring and measurement package described in this article. Nothing in the overall design of this system precludes support for such an addition. However, extensive work to NeTraMet, or addition of another traffic summarization utility would be necessary to get more than very rudimentary information. In particular, monitoring of RTP/RTCP traffic will require looking not just at the IP header, but also at the UDP payload, since that is where the RTP header is located. Digging more deeply into each packet will impose more computation and buffering overhead, particularly for the higher speed interfaces. If some of the NeTraMet meter functionality is moved into the Waikato DAG cards, and this functionality incorporates multicast capabilities, monitoring of high data rate multicast flows may be more readily achieved.

We’ve given a short overview of the motivations for measuring IP flows that receive preferential treatment and we summarized the mechanisms we plan to use to make these measurements. We expect to have prototypes in place by the time this is published, but there is clearly a lot of work to do to accomplish the overall goals. Measurement and characterization of IP traffic has received considerable attention during the last few years; some projects, such as this one, are now focusing on issues related to Quality of Service. We anticipate there will be continued development in these areas, particularly as more organizations deploy services that handle some types of IP traffic differently than others.

[1] W. Johnston, et.al., Information Power Grid Implementation Plan (draft), NASA Ames, (1999).

[2] J. Apisdorf, et.al., OC3MON: Flexible, Affordable, High Performance Statistics Collection, INET97 Conference, (1997).

[5] W. R. Stevens, TCP/IP Illustrated, Vol. 1, Addison Wesley, (1994).

[6] K. Nichols, et.al., RFC 2474: Definition of the Differentiated Services Field (DS Field) in the IPv4 and IPv6 Headers, (1998).

[7] M. Rose, The Simple Book, 2^nd Ed., Prentice Hall, (1994).

[8] N. Brownlee, RFC 2064: Traffic Flow Measurement: Meter MIB, (1997).

[9] N. Brownlee, NeTraMet & NeMaC Ref. Manual, Univ of Auckland, Auckland, NZ, (1997).

[10] N. Brownlee, et.al., RFC 2063: Traffic Flow Measurement: Architecture, (1997).

[11] J. Heinanen , RFC 1483: Multiprotocol Encapsulation over ATM Adaptation Layer 5, (1993).