The Passive Measurement and Analysis activity is making significant progress. Requests for participation, especially in conjunction with the Internet2 community, have been distributed and new hardware based on DAG3 technology is on order.

The Active Measurement Program is steadily growing; we had 89 machines deployed at the close of the quarter, with several more anticipated after the end of the quarter. Various reports and sites have been added to the AMP web page. We have increased our ability to catch potential hardware failures and remotely diagnose them.

BGP analysis has continued. Visuals depicting data collected from the IRTs have been created using Cichlid. Cichlid itself has undergone several positive changes, most of which make the package more flexible.

A web page supporting the creation of CD-ROMs has been developed, two more papers have been published, and three staff members have been hired.

During this quarter, we initiated closer ties to the Internet2 community by collaboratively working towards deploying 25 more passive measurement machines within the foreseeable future. A request for participation was mailed out by Matthew J Zekauskas to the Internet2 community. The announcement is available at http://moat.nlanr.net/Passive/Announcement/.

Given that the Active Measurement Program is called AMP, we will use PMA (passive measurements and analysis) for the passive activities.

The optical line cards in the new machines will be based on DAG3 technology of the University of Waikato in New Zealand. The team at Waikato, headed by Ian Graham, is collaborating with us on this project; it used a prototype machine at SDSC to further develop the code. A new student added during the quarter, David Cheney, works with passive measurements and analysis, focusing on the new DAG3-based machines.

We will give priority to sites wanting to actively collaborate by analyzing collected data, and to sites that aggregate connectivity, such as Abilene to GigaPOP connections.

We have 50 DAG3 cards on order, two per machine, which are supposed to arrive in batches of ten, and two following batches of 20 cards each. There will be an acceptance period of several weeks for the first batch of cards.

The passive monitors require two network connections. One is to the high-performance network on which the measurements will take place. This is an optical connection, single-mode or multi-mode fiber, OC3 or OC12, and ATM or Packet over Sonet. It is purely passive and requires no IP address. The other network connection is an Ethernet connection for control and the return of results. We prefer a 100baseT interface, but a 10baseT interface is acceptable. This interface must have a static IP address assigned for its location as well as a static default route. Encrypted secure-shell connections are the only way to access the monitors.

The passive monitors will be PC-type machines in a rack-mountable chassis.

With the exception of the OC3- and OC12-monitoring cards (DAG3), the PCs utilize off-the-shelf hardware. The DAG3 cards take up PCI slots.

A pair of DAG3 cards will be used to interface to the optical fiber pair via optical splitters:

The splitters take off a fraction of light to allow for non-invasive measurements. A depiction of a complete system takes the PC, the DAG cards, and the splitters into account:

The machines only run network servers essential to operation as a measurement machine. Specifically, inetd and related services are disabled, and access is possible solely via ssh. The ssh is access-controlled to allow only specific hosts to connect.

Daily messages are now being generated, which approximate the maximum throughputs seen in traces we collect:

   Date: Thu, 30 Sep 1999 23:55:06 -0700
   From: Daily Max Throughput Summary 
   To: tputsum@nlanr.net
   Subject: Experimental Daily Max Throughput Summary from packet header traces
   
   ==== =============== ======= =============== ========= =========
   rank Mbps Throughput  site   applic. triplet   Bytes   duration 
   ==== =============== ======= =============== ========= =========
      1       37.603110     ANL      6:20:11179 176499988 37.550083
      2       18.410446   12SDC   17:1114:56544    389496  0.169250
      3       16.859415   12SDC   17:1122:56544    293318  0.139183
      4       15.753099   12SDC   17:1122:56544    272505  0.138388
      5       15.136738   12SDC   17:1122:56544    258965  0.136867
      6        7.583245   12NCS    6:22895:1057    539820  0.569487
      7        6.711261   12SDC      6:22710:20  74035708 88.252507
      8        6.691066   12SDC      6:19524:20  72980688 87.257459
      9        6.631921   12SDC      6:21251:20  73121852 88.205934
     10        6.477111   12SDC      6:22042:20  72259120 89.248572
     11        6.144272   12SDC      6:27237:20  67984408 88.517441
     12        6.119089   12SDC      6:24327:20  68439660 89.476921
     13        6.015576     NCA      6:13056:22  68051064 90.499814
     14        5.754050   12SDC      6:18496:20  64050980 89.051678
     15        5.680204     TXS       6:3549:25    217429  0.306227
     16        5.417602     FXW          47:0:0  59821387 88.336320
     17        5.414523     FXW          47:0:0  59786730 88.335344
     18        5.135515   12SDC      6:13056:22  56842936 88.548748
     19        5.082175     FXW          47:0:0  55974617 88.111264
     20        5.081024     FXW          47:0:0  55959459 88.107360

     . . . .

     99        2.397137     MRT     6:4935:1310   1236159  4.125451
    100        2.396470   12SDC   17:4549:55524  26693147 89.108184

Those summaries are available at http://moat.nlanr.net/TopThroughput/, with the support files per site by date being obtainable at http://moat.nlanr.net/Datacube/Structure/Project/TopThroughput.dates.html.

By the end of this quarter, 89 AMP machines (AMPlets) were deployed and operational, with at least six more to be deployed soon. Occasional difficulties arose with power supplies and fans in deployed machines. As a result, we began utilizing system boards with voltage, temperature, and fan sensors. Procuring these boards has imposed a significant delay, however, the benefit for future management of remote machines was deemed to outweigh the current difficulties.

• Analysis

  A series of new reports has been added to the AMP web pages that summarize the routes that are in use between sites. (See http://amp.nlanr.net/active/cgi-bin/rstatus.cgi?amp-alaska and http://amp.nlanr.net/active/cgi-bin/count-rstatus.cgi?amp-alaska for examples of these reports). Among other things, these reports show 2400 vBNS only, 1900 Abilene and 1100 combined Abilene/vBNS routes between our monitors.

  We continue to collaborate with sites and the network providers. The second of these reports in particular, the router counts, was inspired by a discussion with Kevin Thompson from the vBNS team. Upon its first presentation to the vBNS team, it enabled a routing fault to be discovered.

  Infrastructure for providing general information about the monitors has been improved by adding links for general site information and current issues that affect data collection. This is important because, with around 100 monitors, there are always some that are down because of factors outside of our control (such events this quarter have included router problems, machine room upgrades, and a fire!).

• Hardware Faults

  During this quarter, a number of monitors experienced hardware faults. Three types of faults have occurred:

  1. Power supply failures
  2. Cabinet fan failures
  3. Hard disk failures

  We are reducing the chance of the first two types of failure in new monitors by making some small changes to the monitors' hardware specifications. The rate of failure of hard disks seems to be within industry expectations. We are endearing to catch more of these before machines leave San Diego by increasing the operation-prior-to-shipment period.

  We have increased our ability to remotely diagnose problems and to catch potential problems by changing to a motherboard that includes voltage, temperature, and fan-speed monitors.

• Operational Changes

  There have been many small improvements to the operational infrastructure of AMP. Many of these have involved modifications to the transfer of data from the monitors to the central site, even in the presence of various kinds of network and equipment failure.

  A major change has been the addition of a second central machine to provide redundancy and the replication of all data transfer, storage, and analysis.

  A number of new status- and consistency-checking web pages and procedures have been added. The presence of a full-time system manager has been a major step forward with providing quality data.

Neil Cotofana continued working with BGP routing table statistics. The University of Oregon's ongoing RouteViews project as well as the the BGP peering session between moat.nlanr.net and a router on the vBNS provided data. Perl scripts analyzed the data and subsequently created 2-D graphs, while Cichlid developed 3-D visualizations. The main emphasis has been on the size of the Internet Routing Table and the distribution of prefix lengths within the routing tables of different ISPs. It is interesting to note from these graphs that the /24 prefixes, which make up the largest group of prefixes in the IRT (approximately 30,000), offer reachability to a relatively small number of hosts. Stills from the Cichlid graphs of these visualizations can be found at http://moat.nlanr.net/~neil/cichlid.html and 2-D graphs, which are more quantitative, can be found at http://moat.nlanr.net/~neil/bars.html

Neil also experimented with graphs depicting the redundancy of /24 prefixes as seen across the routing tables of the core routers participating in the RouteViews project. These graphs can be found at http://moat/~neil/24.html

Work continued on scripts analyzing data from the gated.log file. The file, which logs all transactions between moat and the vBNS router, is publicly available at http://moat.nlanr.net/BGP/VBNS/

During this quarter, Jeff Brown made significant progress in developing Cichlid. Several changes were implemented internally to make the package more flexible, and a new version was released. Work-in-progress snapshots are periodically posted at http://moat.nlanr.net/Software/Cichlid/snapshots/snapshots.html

Cichlid was converted over to an object-oriented C-programming model, particularly for the server API. This took a lot of work, and loses compatibility with old servers (i.e. from last year), but is worth it since subsequent changes are easily hidden behind the object interface.

With the new object-based architecture, it became possible to add things like multi-stack bar charts in a day's work of coding, with only a single minor change to the API (1->0 indexing). Jeff added partial-update support -- a potentially daunting set of changes -- with two day's of coding and debugging, and no changes to the API. With the new architecture, as new features are added, the "user" code gets to take advantage of them automatically, with little or no modification.

The new vertex/edge graphs introduced in 2Q99 have been improved a bit with some speed optimizations. Portability concerns have been confined to two modules ("lib/sysdep" and "lib/socket") so that all porting work is confined to those two modules. The "pack" library introduced in 2Q99 has grown up and proven itself invaluable in dealing with more complex things like partial updates.

Using the "sysdep" library introduced in 2Q99, the calls needed to support Win32 compilation have been merged into the source tree, so there is no longer a separate windows version.

Feedback and labelling have been re-introduced to the new architecture, and label colors were made dynamic. A library to support "named" colors -- "Red" and "ForestGreen" instead of {255,0,0} and {34,139,34} -- was added. Feedback windows were made re-drawable.

The collection of sample servers was extended to include "text" servers -- servers that read a mini command language from stdin and perform object manipulation based on the commands -- this allows things like Perl scripts to be piped to these servers and produce data.

Stereo rendering support was added as well, with the CrystalEyes goggles as well as other tools.

Images continue to be available at http://moat.nlanr.net/Software/Cichlid/gallery/gallery.html and http://moat.nlanr.net/~jabrown/cichlid-tests/.

Todd Hansen had been collaborating with Jeff Brown to keep the Windows 95/98/NT port of Cichlid current.

To support the burning of CD-ROMs for such things as distributing sets of measurement data, Jeff Brown has created a web site at http://moat.nlanr.net/CDBurning/ complete with software to make the process intuitive and easy.

Todd Hansen created a new utility which attempts to tune the Windows TCP stack. He recently finished a new version 1.1.1 which supports all of the 32-bit windows operating systems and has some new features to make it easier to use. It autodetects the current OS version and curtails the UI for that particular OS. More info on TCPtune is available from: http://moat.nlanr.net/Software/TCPtune/.

Two papers are now available:

"The NLANR Network Analysis Infrastructure"
by A.J. McGregor, H-W Braun, and J.A. Brown
Submitted to IEEE Communications Magazine special issue on Network Traffic Measurements and Experiments

"Passive Monitoring of Internet Traffic at Supercomputing'98"
by Brynjar Viken,
Eunice'99, August 1999

Proceedings of the 1 July 1999 "Challenges and Opportunities for Measurement and Analysis in a High Performance Computing Environment" workshop as well as the 29-30 June 1999 "Measurement and Analysis Collaborations Workshop" ) are available at http://moat.nlanr.net/Workshops/. Several printed copies have been made and are available upon request.

During this quarter, Bud Hale was added as our full-time system administrator. David Cheney , a UCSD undergraduate student, began focusing on passive measurements and analysis. In October, Meiling Guentzel took a temporary job to help with technical writing until the full-time position is filled.

We also created a staff directory to foster a better community understanding about who works on the NLANR measurement and analysis project and what primary function belongs to each team member.

Several more AMP and PMA machines should be deployed by the end of the next quarter.

More research on collected data is necessary, with an emphasis on the publication of results. We would like to create a regular newsletter that focuses on measurements and analysis. It would feature our own work as well as the work of those who collaborate with us and/or utilize our infrastructure.

We hope to get the full-time manager and technical-writer positions filled.

We would like to maximize the automation of problem warnings, as we need to spend more time on our own research.

There is a need to provide for site status reports. Plans are underway to provide for such reports online to detail causes for data outages and corrective actions planned.