Continuing development of the network analysis infrastructure (NAI) progressed well this quarter. Six passive monitors with the new DAG3 interface adapters were deployed; improvements to the firmware were made and all six are now fully operational. Significant improvement in the performance of the passive monitors is expected. Joerg Micheel (one of our collaborators) who has been working with us regarding the DAG cards and long header traces, has been brought on board to take the lead for the Passive Measurement and Analysis project (PMA); he starts early next quarter.
The Active Measurement Project (AMP) system manager created last quarter to monitor and maintain all remote sites to ensure uniformity of configuration and software across the mesh is currently being deployed in stages (as testing and debugging continues). The investigation of throughput tests for AMP was problematic, due to the network connection and operating system chosen for the testing; progress was made in understanding the parameters necessary for successful testing. The algorithm for the event detection system has been created. IPMP has been implemented for FreeBSD. Both are in the testing phase.
The reliability and performance of the hardware continues to be excellent, with the rate of outages still significantly less than 1%. To prepare for changes in availability, as well as improvements in devices, investigation of alternate microprocessors and system boards was begun. Testing was performed to evaluate appropriateness (and compatibility) as replacements for future deployments.
In our network routing activities, improvements were made to the appearance of the on demand graphs and a new script was written to graph statistics related to the prefixes that are most often repeatedly added, or withdrawn, from the Internet Routing Table. Both have been found to be useful in illustrating network instability by graphing the fluctuation of routes over time.
Additions have been made to our ongoing efforts of making all data that we have available for use by the HPC research and engineering community easier to access and use. The Data Cube (passive measurement data) and corresponding FAQ are independent and stable now, requiring few changes and little maintenance. Additional query systems and FAQs are under development. All of these projects promote feedback from users in order to improve.
New issues of the Network Analysis Times and NLANR Packets were produced. A new NLANR general-purpose poster and single-sheet brochure, briefly covering all three NLANR site’s activities, were completed and distributed. The NLANR/MOAT main page was redesigned, the content redistributed, and a navigation bar was added, all of which make the site easier to navigate.
We continued close collaborations with several researchers, as well as adding new ones. We are working with the network systems group at SDSC in the creation of a data compute engine for AMP data (for intensive computing). Outreach efforts have continued with researchers at UC Los Angeles, UC Irvine and CSU Long Beach with a view to future collaborations.
Progress continued to be made on the beginning of the wireless backbone, as additional wireless links were created and successfully tested from SDSC to sites in the eastern San Diego County. The experience and information gathered from these varying and contrasting sites enables continued development of the first phase of the HPWREN.
Cichlid visualizations were created of the wireless network and are nearly completed for the seismic data. It is anticipated that seeing the seismic data in 3-D will improve and clarify what activity is occurring in the data stream (and thereby improve understanding of seismic activity).
During the past quarter the AMP infrastructure has held constant at 116 total Active Measurement Project (AMP) monitor sites. Support and maintenance of the network analysis infrastructure (NAI) continues; anomaly diagnostic methods are also continuing to develop. This has resulted in a continued downward trend in remote site monitor down time. Overall site monitor downtime for the AMP monitors is still just a fraction of one percent. The NLANR Site Information and Site Status database has proven to be a valuable tool during this period. This database is being maintained and updated regularly.
As the state of microprocessors and system boards continues to develop, there are changes in the availability of devices. Preparation for these changes in availability requires testing and research into the performance of new devices coming on the market. It also offers the opportunity to take advantage of new features and device improvements. Therefore, we have begun to test new microprocessors and system boards with a view to evaluating their appropriateness (and compatibility) as replacements for AMP monitors in the future.
These tests have shown that it is desirable to keep informed of developments in the FreeBSD operating system used on the AMP monitors. (These tests have involved several of our AMP staff, as well as two of our collaborators, New Zealand and DREN.) There is ongoing testing of the network interface throughput capabilities of the released versions of the OS; issues discovered during this quarter are still to be resolved.
The system manager created last quarter (“RoboTony” - a method to synchronize all of the systems with a single directory hierarchy, designed to automatically monitor and maintain all remote site machines with the correct configuration and software version) is beginning to be deployed (in stages as testing and debugging continues).
Six passive monitors with the new DAG3 OCx interface adapters (received from Waikato University last quarter) were shipped to passive monitor sites this quarter, giving us the opportunity to gain experience with these adapters. Knowledge gained from this experience led to updates and improvements to the firmware image to be loaded to the Xilinx chip when the monitor boots and the DAG3 cards initialize. All six of the recently deployed monitors are now operating with the new image. Additional deployments of passive monitors with these adapters are planned for the beginning of next quarter. Significant improvement in the performance of the passive monitors is expected. Previously, discussions were held reviewing and planning for passive monitor deployments at Internet2 sites (several of which already have passive machines in place). Now that the DAG3 cards are fully POS network capable, new deployments on I2 can again be reviewed and planned.
Joerg Micheel, one of our collaborators at the University of Waikato, who has been working with us regarding the DAG cards and long header traces, has been brought on board to take the lead for the passive measurement and analysis project (PMA). He starts early next quarter and will work closely with MOAT staff, in particular David Cheney who has been working on various aspects of the PMA project (the Data Cube, the Data Format FAQ page and the statistical query system - more information on these projects is available in section 3 - Networked Data and Result Presentation). (Joerg’s role will be similar to that of Tony McGregor’s with regard to the AMP project.)
Other highlights include work to increase the disk space on the nai/moat server (new disk arrays were installed, configured and are currently in use) and the development of a statistical query system for the PMA data (for more information on this query system, please see section 3 - Networked Data and Result Presentation).
In the last quarter the Active Measurement Project (AMP) has made a number of steps forward. We are currently at 116 monitors and are expecting to deploy more in the future. As we approach our primary target of having a monitor at each HPC connected campus, the rate of deployment has slowed and is not expected to be as high as in previous quarters.
The investigation of throughput tests for AMP began with a limited test, involving a few machines, running tests between SDSC and the Pittsburgh Supercomputer Center [PSC]. The purpose of the test design was to investigate any differences relative to network performance between the FreeBSD 3.0 and 3.3 kernels. Problems were found with both the network connection at PSC and the machine set-up at SDSC which ended up dominating the results obtained. Unfortunately, this limited the usefulness of these results for comparison of the kernel versions. Planning for the second round of tests is well underway; to avoid these problems, the design is being built upon the lessons learned in this trial.
We have begun work with the network systems group at SDSC (SDSC Enterprise Network Services [ENS]) to set up an AMP data compute engine that we will use for more intensive analysis. Basically, this repository will be a special purpose server that contains a replica of AMP data. The benefit of this system is that it will allow us to do heavy processing of the data without impacting the performance of our data collectors and Web servers. ENS plans to access this data for analysis and correlation with other data derived from an ongoing study to characterize application behavior between SDSC and NPACI partner sites.
The algorithm for the event detection system has been created. It is working and currently undergoing testing. A rough draft of the report is available at: http://www.cs.waikato.ac.nz/~pv3/.
IPMP has been implemented for FreeBSD. IPMP is a measurement protocol that should provide more measurement capabilities. This implementation will allow us to test out the protocol.
The Waikato Applied Network Dynamics (WAND) group at the University of Waikato (longtime collaborators of ours) currently has one AMP monitor (amp-kiwi) and has just deployed another one at the University of Auckland (as of early October). These two machines will act as a small peer network; they hope to obtain funding to expand this to a dozen or so machines sometime in the future. If deployed, this mesh could be useful to us in a number of ways, including as a test-bed for outside network peers with AMP networks, which we are planning. And, in general, it will foster international collaboration on both the creation and measurement instrumentation aspect and later, in analyzing the results.
Continuing development of the infrastructure created to analyze and graph data on demand from a BGP peering session includes improvements that have resulted in cleaner, easier to read graphs. (The set-up uses logs generated by the GateD daemon.) A script has been added to graph statistics related to the prefixes that are most often repeatedly added, or withdrawn, from the Internet Routing Table. Thus far, these scripts and graphs have been useful in illustrating network instability by graphing the fluctuation of routes over time. Plans are underway for continued improvements.
The graph above depicts incoming BGP routing messages ('Add' and 'Withdraw') to a specific router over a 24 hour period. These messages inform the router regarding changes in the availability of routes to different networks. From graphs like these, an idea can be drawn about fluctuations within the router's routing table and in local network connectivity.
Cichlid developer, Jeff Brown spent the summer at Lucent Technologies (formerly Bell Labs). Upon his return in September, he entered the UCSD Computer Science and Engineering department's Ph.D. program. As time permits, he will continue to evolve the Cichlid 3-D visualization system. During this time (including the summer) he is/has been available to consult regarding Cichlid projects.
Cichlid visualization of seismic data
The vertices (balls) in this
image represent seismic sensor sites (located throughout the world). This
image is a
screenshot of an animated visualization, in which the
vertices
vibrate when there is activity at that site. The seismic data is collected by geophysicist Frank Vernon of
Scripps Institute of Oceanography (SIO).
Justin Fields (new student researcher hired last quarter) worked on a project using Cichlid to visualize seismic data (project nearly completed). Please refer to the sample Cichlid visualization above. It is anticipated that 3-D data visualizations of seismic activity will be used to clarify what is occurring in the data stream and therefore improve the understanding of seismic activity.
Justin also created a Cichlid visualization for the wireless network (details and image are available in section 5 - New Environments ... wireless activities).
As part of our continuing goal of making all data that we reasonably can not only available to the HPC research and engineering community, but also easier to access and use, we have made additional improvements to the data sets.
A statistical query system is currently under development for the passive measurement (PMA) data, with which researchers can make queries about certain statistics as a method for finding data sets which will most closely fit their needs. As we receive feedback from various collaborators, additional options will be added to this system. A preliminary version is available: http://moat.nlanr.net/PMA/StatQuery.html.
The design and early stages of the Data Format FAQ page were completed; the FAQ page will now expand and evolve over time in response to user requests. The Data Cube (passive measurement data) and corresponding FAQ are independent and stable now, requiring few changes and little maintenance. All are available at http://moat.nlanr.net/PMA/.
On our Web site, we are hosting the Auckland-II Trace Archive (long header traces data set from New Zealand). The Auckland-II data set is a collection of long GPS-synchronized IP header traces captured (since November 1999) with a DAG2 system at the University of Auckland Internet uplink by the Waikato Applied Network Dynamics (WAND) research group of the Computer Science Department, University of Waikato (New Zealand).
A new issue of the Network Analysis Times was produced. Printed preview editions of the issue were made for distribution at the NASA Gigabit Networking Workshop, Mountain View CA, August 14-16 2000 and the NLANR Joint Techs Workshop, Toronto Canada, August 20-24 2000. The html version of the issue was created (and posted; announcement was sent on 10/4/00). The NATimes was also distributed at the Web Technology Forum’s August monthly meeting (at SDSC) and the Extreme Networking Workshop, August 2000, SDSC.
Following the successful creation, design and distribution of an AMP poster, the new NLANR general-purpose poster, briefly covering all three NLANR site’s activities, was completed and distributed. A single-sheet NLANR brochure was also created and distributed.
The NLANR/MOAT main page (http://moat.nlanr.net) was redesigned and the content was broken into subcategories; the new header navigation bar was added to each section, making navigation easier. A new Web page was made that links to MOAT papers (authors versions of published/submitted papers): http://moat.nlanr.net/Papers/.
A new issue of "NLANR Packets" (August 15, 2000) was published (full issue posted on-line; hard copy summaries made for distribution).
McGregor, A., and Braun, H-W., "Balancing cost and utility in active monitoring: The AMP example," Accepted for publication and presentation at the INET 2000 conference, Japan, Jul. 2000.
Ritke, R. and Braun, H-W., “A comparison of two TCP connection throughput computation methods using vBNS traffic traces,” to be submitted to PAM2001.
Ryu, B., Cheney, D., and Braun, H-W., “Internet flow characterization:
Adaptive
timeout," to be submitted to PAM2001.
Tony McGregor attended INET 2000, Japan, July 2000 and presented: McGregor, A., and Braun, H-W., "Balancing cost and utility in active monitoring: The AMP example."
Ronn Ritke attended the NASA Gigabit Networking Workshop, Mountain View CA, August 14-16 2000. He served as the official note taker for the measurement breakout session and participated in the draft report.
Ronn Ritke attended the NLANR Joint Techs Workshop, Toronto Canada, August 20-24 2000; and gave a presentation on “Extending Network Connectivity using Wireless Technology.”
We hosted the EPSCOR Satellite/Wireless Workshop held at the San Diego Supercomputer Center (SDSC), July 20-22, 2000, all members of the group attended one, or more, sessions. PI Hans-Werner Braun co-hosted (with Dan Van Bellingham) and presented “Opportunities for Ubiquitous Networks, a Case Study.” (Please see the collaborations section following for more detail on the workshop.)
Ronn Ritke and Bud Hale attended several sessions of the Extreme Networking Workshop, August 2000, SDSC.
Initial coordination for our AMP demo at SC2000 in November has begun (the demo will be in the Network Operations Center [NOC]).
Envision, a publication of SDSC and NPACI recently featured MOAT in its July - September 2000 issue with an article on the Active Measurement Project (AMP) - “Active Measures to Maintain a Healthy Network, ” p. 11 (and available on-line: http://www.npaci.edu/enVision/v16.3/nlanr.html), and the Cichlid visualization of the logical vBNS network on the back cover of the issue, (http://www.npaci.edu/enVision/v16.3/back-cover.html).
Online, a Web-based publication of SDSC and NPACI featured our activities with “Wired for Weather: NLANR Installs Web-Based Meteorological Instruments,” http://www.npaci.edu/online/v4.13/nlanr_weather.html.
Our Weather Web page (http://moat.nlanr.net/Weather/) is now linked with other “SDSC Live” events on the SDSC home page (http://www.sdsc.edu/).
Details on the completed project with Les Cottrell and Warren Matthews, both of Stanford Linear Accelerator (SLAC), regarding a comparison between Surveyor, PingER, RIPE and AMP which found that while individual measurements do not correlate well - except for RIPE and Surveyor - the overall stats are closely aligned can be found at: http://www.slac.stanford.edu/comp/net/wan-mon/surveyor-vs-ripe.html.
Outreach efforts have continued with researchers at UC Los Angeles, UC Irvine and CSU Long Beach with a view to future research and collaborations.
We hosted the EPSCOR Satellite/Wireless Workshop held at the San Diego Supercomputer Center (SDSC), July 20-22, 2000. The workshop served as a platform and starting point to exchange state-of-the-art information from network researchers and communications providers with the state-of-practice information and point of view of disciplinary researchers with remote networking needs, as well as network administrators and educators from rural, underserved, areas. The core purpose was to begin to define the parameters which will be needed to reduce the performance gap between connected high performance national centers, such as major universities, and the capabilities currently available in rural areas and to scientists in the field. The ultimate goals include greater high performance network ubiquity and more transparency and answering the question of how to achieve the best service through potential hybrid situations. Participants included: Network researchers; Disciplinary researchers with needs for remote connectivity to enable and enhance the full scope of their work (in the field, at observatories, with telemetry devices placed in remote and/or distant locations); Network administrators from EPSCOR states; Representatives of the National Science Foundation (NSF); and Communications providers. Slides from some of the presentations are available at: http://moat.nlanr.net/EPSCoR.
Student researchers continue to contribute and make good progress in all areas of our activities. We hired two new students towards the end of this quarter.
Pavana Yalamanchili, a Masters graduate student in Computer Science at UCSD, joined us on 10/1/00. She will be working on network measurements and data analysis of channel characterizations, (with respect to user requirements, etc.) on the wireless network (High Performance Wireless Research and Education Network [HPWREN]).
Kuo-Wen Lo, a UCSD undergraduate student, will be working on performance measurement and analysis on the wireless network.
In preparation for the wireless project becoming a separate entity (High Performance Wireless Research and Education Network [HPWREN]), the following activities were undertaken (including several successful field installations). Progress continued to be made in the creation of the beginning of the wireless backbone, as additional wireless links were created and successfully tested from SDSC to sites in the eastern San Diego County. The experience and information gathered from these varying and contrasting sites enables continued development of the first phase of the HPWREN.
In the area of measurement and analysis relating to the wireless network:
Web pages were developed to display SNMP data (in real time). Currently, signal to noise ratios, Packets Per Second (PPS) ( unicast packets) and Kilobits Per Second (kBPS) are being plotted in twenty four intervals. With PPS and kBPS there are additional options to graph 48 hours, Weekly or Monthly plots. This data allows investigation of the wireless network at time X under circumstances Y, from which we can perhaps develop correlations between environmental factors and performance characteristics of the network. Also, a Web form with which we are archiving our progress and documenting design decisions (in a diary style format) was developed.
To access the Wireless Archives and SNMP Gateway: http://moat.nlanr.net/HPWREN/
A third weather station was deployed on the wireless network (at the Mt. Woodson site) to record weather information. Weather Web page: http://moat.nlanr.net/Weather/.
Cichlid 3-D screenshot of the wireless network. Each of the
end-points represents a link on the network. The links are pinged on a
regular basis; the number of drops indicates
the number of lost pings. The numbers between end-points are the
ping times for that link. The colors of the links are
also representative of ping time, the brighter the link, the faster it is.
It is
anticipated
that Cichlid visualizations of the wireless network and data will be
useful for
evaluating the network's status. Images such as these will also be
used for
demonstration and presentation purposes.