National Laboratory for Applied Network Research
Measurement and Operations Analysis Team
(NLANR/MOAT)
Quarterly Report for 2nd Q 2000
April, 2000 through June, 2000
Table of Contents
1. Summary
2. Measurement Infrastructure and Network Analysis Activities
- Infrastructure Management
- Passive Measurement and Analysis (PMA)
- Active Measurement Project (AMP)
- Network Routing
3. Networked Data and Result Presentation
- Cichlid 3-D visualization system
- Documentation, databases, and publications
- Papers, Presentations, and Conference/Meeting Participation
4. Collaborations, Students, and Additional Activities
5. New Environments for Measurement and Analysis (wireless activities)
6. Goals for Next Quarter
The network analysis infrastructure (NAI) continues to be developed and improved in many ways. Ongoing growth of the NAI brought the number of Active Measurement Project (AMP) machines to 116 and the number of Passive Measurement and Analysis (PMA) machines to 21. The reliability and performance of the hardware is excellent: outages (down-time) averaged over time for all sites is significantly less than 1 percent.
With a large number of machines deployed all across the United States, developing methods to assist in their management is highly important. Automation of repetitive tasks will make the management and maintenance of an increased diversity of systems possible. This needed automation has begun with the design of a system manager to automatically monitor, maintain, and synchronize the configuration and software version of each Active Measurement Project (AMP) monitor.
At the beginning of the quarter, the new passive monitoring (DAG3) cards became available; passive systems based on these cards have become the core of our new Passive Measurement and Analysis (PMA) technology. A light weight throughput test is important for the Active Measurement Project (AMP) because we want to be able to take frequent throughput measurements over a large number of paths, and assess the variations over time. Work on the development of a lightweight throughput test began with the creation of a test network.
Documentation of our activities and various areas of measurement (active measurement data, passive measurement, and network routing) continues. The vehicle for accessing the approximately 2GB of passive data generated daily, the Data Cube, was redesigned this quarter, resulting in significant improvements in accessibility. An automated system for processing and analysis of the collected BGP data was created. A redesign of the Web interface pages to NLANR/MOAT and NLANR was begun at the end of the quarter.
The majority of the effort regarding the wireless network was devoted to making modifications in order to be able to make better measurements of its performance properties. An additional connection was added to the wireless network.
2. Measurement Infrastructure and Network Analysis Activities
Infrastructure Management
During this quarter, requests from sites were received which added eight new machines to the AMP constellation; the total number of AMP monitor sites now stands at 116. Requests for passive (PMA) monitors were also filled bringing the number of sites in the passive infrastructure to 21. Our new passive measurement cards based on DAG3 technology by the WAND group at the University of Waikato in New Zealand became available. Systems based on those cards became the core of our new PMA technology. The last six passive machines deployed used the DAG3 OC3/OC12 adapter cards (one was a replacement of an older monitor using a Fore adapter).
Methods were developed and refined to update entries and to maintain site data verifications and contacts on the new NLANR/MOAT Site Information and Status database. During this period the database was fully proven and has become a valuable tool in maintaining the network analysis infrastructure (NAI) in an efficient manner.
We have designed, and are currently implementing, an automated remote monitor management system, (a system manager internally known as "RoboTony"). This software system is being designed to automatically monitor and maintain all remote site machines at the correct configuration and software version. It will automatically synchronize the configuration of each of the remote monitor systems, thereby reducing the system management tasks that must be performed manually. This will make the maintenance of an increased diversity and number of systems possible. We anticipate that it will be operational by the end of next quarter.
The reliability and performance of the network analysis infrastructure (NAI) hardware continues to be measured and evaluated. With 137 operating sites, outages (down-time) averaged over time, is excellent: significantly less than 1 percent.
The term vBNS that occurred in some of our URLs and page headings was changed to read HPC. This long overdue change had been delayed due the tricky nature of the task. The term (vBNS) came directly from the directory structure and this structure needed to be changed, however, without interference to the ongoing measurements (we did not want to shut them down to correct this).
A piece of code was developed to monitor how many parallel processes (executing traceroute commands) are going and limits the number of traceroutes that get started if there are network faults. (The traces take longer to do if there is a network fault, in this case, they overlap more.) We have tested this on a dedicated server and installed it on the AMP machine located at San Diego State University, which has been having some problems.
Additional infrastructure activities during the quarter:
- a new Cichlid fuzzball server (with maximum values) was added to our server; also, the Cichlid server start page was updated. (The Cichlid 3-D Visualization system is a general purpose visualization tool that transforms raw data into compelling images and animations. It was developed by Jeff Brown, one of our student researchers.) (http://moat.nlanr.net/AMP/active/cichlid.html)
- updated the fping script on the monitors so that it only sends one packet and records a loss if it is lost. We still need to investigate suggestions that fping appears to give numbers different to ping.
- installed a test program on the monitors that runs from cron once an hour and checks the logs for the signature of the cronbug and sends notification (to Tony) if it is detected.
- upgraded the status mail program on amp so that status reports include a list of monitors that have previously been reported as down.
- added a new file to watt:~actmon/src/pinger called "shutdown-sites." This file lists monitors that have data, but we no longer expect to be active. Currently, it contains amp-sc99 and amp-kiwi. It will be used to make the reports more meaningful.
Passive Measurement and Analysis (PMA)
The primary focus for the quarter was the new DAG3 based PMA cards and the preparation and testing involved in deploying them. A large amount of work was done by student researcher, David Cheney, on the data collection infrastructure of the passive measurement data, which included a redesign of the Data Cube (more detail can be found in section 3 - Networked Data and Result Presentation). A FAQ page to assist in accessing and using the passive data has been created. The page is currently in a beginning stage, with the plan to expand and evolve it over time (http://moat.nlanr.net/PMA/).
Throughput method computation comparison research was continued by our new researcher manager, Ronn Ritke, whose dissertation covered this subject and used NLANR/MOAT data.
Active Measurement Project (AMP)
Discussions have begun (Tony McGregor, Hans-Werner Braun, and Todd Hansen) on how to handle throughput and RTT/Loss tests at the same time, and in particular, how to handle throughput tests interfering with the RTT tests.
An important aspect of our AMP strategic goals is a strong emphasis on throughput tests that give an indication of achievable end-end performance. Jamie Curtis (Waikato graduate student) is focusing his
work on our existing throughput measurements, and how to modify them to consistent, low impact, tests throughout the measurement mesh. He is hoping to provide interesting results for the users of the AMP project, as well as further knowledge about throughput tests, and the variables that effect them. He has begun the creation of a test network to support his activities.
Following the creation of a high performance wireless network, we will use separate interconnected measurement meshes across two networking systems, one being the existing HPC mesh, the other the wireless infrastructure that we are creating, and then interconnect the two meshes at one or more strategic points. Also in the planning stages are what tests should be run over the wireless infrastructure.
Network Routing
Student researcher Neil Cotofana returned and worked on automated processing and analysis infrastructure for the network routing data. He began creation of an automated system for processing and analysis of the collected BGP data. The system is designed to allow the addition of new statistics over time. Currently, there are two main sections that offer different statistics: 1) the daily page, which graphs parameters such as the number of add/withdrawn messages and the number of prefixes contained in each for the preceding day's dataset; and 2) the long term page which graphs summaries of the daily data over longer periods of time. These graphs are useful for quick visual summary of the behavior of the connection for a certain period of time, or as a basis to pursue more elaborate analysis.
3. Networked Data and Result Presentation
Cichlid 3-D visualization system
A number of improvements were made to Cichlid this quarter. Jeff Brown (student researcher, Cichlid developer) added support for an abstract "display list manage," which has the potential to speed things up greatly on machines with "real" graphics hardware. It has not been exploited much yet, although these changes also led to substantial speedups on the flat screen SGI O2 (which we use for demonstrations). The addition of new features, and other changes, revealed a number of bugs that were fixed.
Other significant improvements to Cichlid this quarter, included:
Documentation, databases, and publications
We make all data, that we reasonably can, available on a regular basis, through our Web site. Part of the evolution and expansion of the site involves making this data more easily accessible (and usable) by a wider range of users. These goals are being pursued by a first step of creating database referencing for our publicly available data. The lengthy process of creating and designing the databases, then testing the performance and accessibility has begun for each of our publicly available measurement data types. Each database is being dynamically designed to include current data. Documentation of each of the areas that are the focus of our measurement and network analysis activities is also beginning. Some of the steps taken this quarter:
A general redesign of the Web interface pages for NLANR/MOAT and NLANR was started at the end of the quarter (moat.nlanr.net and www.nlanr.net), and will continue.
A new Active Measurement Project (AMP) poster has been created and is currently being displayed and distributed at meetings and conferences. We had two rollable, laminated versions of the large size poster made for ease and durability in shipment and travel. Substantial work has been completed on a new general NLANR poster (covering the activities of each of the three groups).
Plans are underway for the next issue of the "Network Analysis Times" (to be released the end of July). NA Times printed copies were distributed at Pam2000 (April, New Zealand), NLANR Techs Meeting (May, Minneapolis), CENIC 2000 (May, San Diego), CAIDA's IHEC workshop (June, SDSC), and the CIC-2000 conference (June, Las Vegas).
A new issue of the Web-based publication "NLANR Packets" was issued (May 12, 2000); printed copies of the summary are distributed at conferences and meetings.
Articles regarding NLANR/MOAT activities appeared this quarter on the NPACI/SDSC Web-based publication, "Online" including: a reprint of the March 2000 Cichlid article (http://www.npaci.edu/online/v4.7/news.html);
weather article: http://www.npaci.edu/online/nlanr_weather.html;
wireless article: http://www.npaci.edu/online/v4.6/HWB_wireless.html;
IRCache bakeoff: http://www.npaci.edu/online/v4.5/bakeoff.html
Papers, Presentations, and Conference/Meeting Participation
Papers published:
Brown, J., McGregor, A., and Braun, H-W., "Network performance visualization: insight through animation," PAM2000 Passive and Active Measurement Workshop, Hamilton, New Zealand, pp. 33-41, Apr. 2000.
Curtis, J., Cleary, J., McGregor, A., and Pearson, M., "Measurement of voice over IP traffic," PAM2000: Passive and Active Measurement Workshop, Hamilton, New Zealand, pp. 43-59, Apr. 2000.
Hansen, T., Otero, J., McGregor, A., and Braun, H-W., "Active measurement data analysis techniques," Proceedings of the International Conference on Communications in Computing (CIC'2000), Las Vegas, Nevada, p. 105, Jun. 26 - 29, 2000.
McGregor, A., and Braun, H-W., "Balancing cost and utility in active monitoring: The AMP example," Accepted for publication and presentation at the Inet2000 conference, Japan, Jul. 2000.
McGregor, A., Braun, H-W., and Brown, J., "The NLANR Network Analysis Infrastructure," IEEE Communications Magazine, Vol. 38, No. 5, pp. 122-128, May 2000.
Ritke, R., "An internet toolbox," Dr. Dobb's Journal, Jun. 2000. Reprinted in Information Week Magazine, May 29 2000.
NLANR/MOAT was represented at the following Conferences/Meetings during the quarter:
Tony McGregor attended the Pam2000 (April, New Zealand) conference and presented:
Brown, J., McGregor, A., and Braun, H-W., "Network performance visualization: insight through animation," PAM2000 Passive and Active Measurement Workshop, Hamilton, New Zealand, pp. 33-41, Apr. 2000.
Todd Hansen attended the CIC'2000 conference and presented:
Hansen, T., Otero, J., McGregor, A., and Braun, H-W., "Active measurement data analysis techniques," Proceedings of the International Conference on Communications in Computing (CIC'2000), Las Vegas, Nevada, p. 105, Jun. 26 - 29, 2000.
Ronn Ritke attended the NLANR Techs Meeting (May, Minneapolis) and gave a presentation on MOAT's activities. Bud Hale attended CENIC 2000 (May, San Diego); Tony McGregor attended CAIDA's IHEC workshop (June, SDSC). For both of the meetings held in San Diego, especially CAIDA's workshop held at SDSC, several members of our research group met with various attendees to touch base with colleagues, discuss possible collaborations, share information, and meet researchers in the field whom they had not yet met.
4. Collaborations, Students, and Additional Activities
Current collaborations:
Often collaborators are developed through the Web site and the available data. While networking and contact development is currently being done by all members of our team on a regular basis, primarily as a result of their enthusiasm and interest in the field of network research, specific contacts are sought for collaboration. Outreach efforts have been started with researchers at UC Los Angeles, UC Irvine and CSU Long Beach with a view to future research and collaborations.
Students
Three of our students graduated in this quarter: Jose Otero (April), Todd Hansen (June) and Jeff Brown (June). All three are remaining with NLANR/MOAT to some degree, and will continue and expand upon work that they have begun. Jose (half-time) and Todd (full-time) have become regular staff members, as of this quarter. Jose is expected to enter graduate school (Physics). Jeff will be interning at Bell Labs (Lucent Technology) this summer and will return in the fall to attend graduate school (Computer Science). He will continue to develop the Cichlid 3-D visualization system, but it is unknown how much time he will have available.
We added a student, Justin Fields, a UCSD freshman. In addition to learning a great deal, he rewrote the cgi script for graphing throughput for the wireless network, and worked on using Cichlid for graphing seismic data and on the AMP status page.
There are several new students working on NLANR/MOAT activities at Waikato University (New Zealand), under the supervision of Tony McGregor:
Jamie Curtis, joined the project as a PhD student. He is looking at throughput tests, in particular he is trying to see if it's possible to develop a lightweight throughput test. He started on the 1st of May. His initial thesis proposal is available at: http://wand.cs.waikato.ac.nz/~jpc2/initial_prop/.
Matthew Luckie, is doing the core of a FreeBSD IPMP implementation, as part of Tony's undergraduate networks class.
Matthias Dallmeier, also a member of the undergraduate networks class, is working on correlations between paths passing through a common network component in an attempt to isolate the source of network problems.
Pieter van Dijk, an undergraduate project student is working on the event detection program.
5. New Environments for Measurement and Analysis (wireless activities)
The focus regarding the wireless activities has been on the wireless setup, including making modifications so we can do better measurements of performance properties. In addition, a system was
designed and implemented which reports weather data at both end points for our initial wireless network between SDSC and Ramona. This included finding appropriate weather instrumentation, writing software to pull data from these instruments and then posting them to a Web page for user retrieval. This data will be used to search for correlations between wireless performance and various weather phenomena. The weather information is available at: http://moat.nlanr.net/Weather/.
Since the last quarter we have upgraded the OTC wireless equipment to Lucent radios, as we experienced significant performance limiting data buffering problems with the initial installation. However, given the lesser power output of the Lucent hardware, it became necessary to introduce more amplifiers to make the connectivity work reliably. In addition, we connected a Scripps Institution of Oceanography (SIO) microbiologist to the wireless network, who was previously connected only via a dial-up modem. This required upgrading the radio on Mt. Woodson to point-to-multipoint equipment, while leaving the Mt. Woodson to UCSD connection as point-to-point. We still expect another amplifier soon to improve the more long distance link between Mt. Woodson and UCSD.
We made significant progress in our abilities to collect routine data from the radios, as well as measurements about the wireless network performance in general. More results will be available on our Web site soon.
6. Goals for Next Quarter
- Passive measurement machine deployment
Make the new version of the DAG3 software, fully operational (complete the system software upgrades on the remote machines)
Install additional disk arrays, which are needed for the traces (eventually determine course of action re backing up the traces, as we are starting to have a lot more data with the new machines.
- Active Measurement Project
We anticipate that the automated system manager (for remote monitor management) will be completed, tested and fully operational by the end of the quarter.
The throughput work will be built upon over the next quarter, by scaling the current limited throughput testing to a full system over all AMP machines. Also, the first steps towards a light weight throughput test will be carried out, assisted by the understanding generated by the current investigation.
- Network Routing
The next step consists of adding more statistics, evolving the way the graphs are presented and possibly correlating the behavior of the BGP session with phenomena in other areas (such as those handled by the AMP project).
- Wireless
We are still awaiting an amplifier/filter to increase the signal to noise ratio so that we can reduce the packet loss to the new location (SIO microbiologist's site). Also, we still expect another amplifier (soon) to improve the more long distance link between Mt. Woodson and UCSD. We expect to install other long distance links into the mountains, for an initial installation with SIO earthquake sensors and San Diego State University (SDSU) astronomers.
- Meetings and Conferences
Tony McGregor will attend and present at the July Inet2000 conference in Japan. Ronn Ritke will attend and present at the August NLANR workshop in Toronto Canada; he will also attend the August NASA Gigabit network meeting.
- Documentation, papers, databases, and publications
Publication of two accepted papers
Completion and production of the new (general) NLANR poster
The next issue of the Network Analysis Times
The next issue of NLANR Packets
Additions and improvements to each of the referenced databases
Continuation of the general redesign of the Web interface pages for NLANR/MOAT and NLANR
Potentially, some new collaborative efforts