Engineering:
- Peering will be established with Esnet. The vBNS connection at SDSC will be upgraded to OC12.
Research:
- Measurement - The following tasks will be initiated during this quarter: development of DEC Gigaswitch encapsulation code for OC3mon; enhancement of statistics code for OC3mon data; port CVS/RCS control software from UNIX to DOS. An OC3mon monitor will be placed at FIX-West to gather data on traffic flows across this exchange point.
- Visualization - A java-based tool to visualize network topologies (MapNet) will be initiated, as will preliminary development efforts on traceroute visualization tool and a cache hierarchy visualization tool.
- MapNet will be demo'd for NSF personnel and for the Joint Engineering Team (JET) in August.
- IP v.6 - Code will be installed on the six boxes.
Image Spreadsheet Demonstration
Boulder, CO
June 23-27, 1997
Version 1.0 July 21, 1997
1.0 GOIN Overview
During the 23-27 June 1997, the working groups of the Global Observation Information Network (GOIN) met at the National Center for Atmospheric Research (NCAR), in Boulder CO. GOIN is an initiative between the President of the United States and the Prime Minister of Japan which provides cooperation between corresponding agencies of the two governments on network connectivity and interoperability to support exchanges of both satellite data and in-situ Global Observation data.
2.0 Image Spreadsheet Demonstration
Among the several applications demonstrated in Boulder, the "Image SpreadSheet" is being developed by Dr. Fritz Hasler, at NASA Goddard Space Flight Center (GSFC). It provides a user with tools to perform advanced interactive visualizations of very large datasets. The Image SpreadSheet server consists of a high performance compute and database engine at GSFC. For this demonstration the user client workstation was at the conference in Boulder, and it accessed the data stored remotely (at GSFC, about 1500 miles away) via high performance WAN.
3.0 Image SpreadSheet Demo -- WAN Configuration
The high performance network infrastructure making these demonstrations possible was provided through the support of multiple agencies, including NCAR, NASA, NSF, and DOE.
The Image SpreadSheet database server at GSFC connected directly to the GSFC campus FDDI LAN. The NASA router also connected to this LAN. Data flowed from the server over the GSFC LAN then via a Sprint OC-3 (155 Mbps) ATM Permanent Virtual Circuit (PVC) provided for the NASA Research and Education Network (NREN) to a NASA router at Ames Research Center (ARC), Mountainview, CA.
This Ames router then used a new PVC (back into the same Sprint cloud) to connect to an ATM switch at General Atomics Corp (GAC), in San Diego, CA. The GAC connection used a Sprint DS-3 (45 Mbps) circuit provided by the Department of Energy (DOE), as part of the Energy Science Network (ESNet). Thus Sprint was required to create an ATM virtual circuit between what were previously two separate Virtual Private Networks (VPNs).
The GAC switch sent the traffic across the street to the San Diego Supercomputer Center (SDSC), via an existing OC-3 circuit. At the IP level, the router at SDSC peered with the router at ARC.
The SDSC router then sent the data over the very high-speed Backbone Network Service (vBNS), an MCI corp VPN sponsored by the National Science Foundation (NSF). SDSC has an OC-12 (622 Mbps) connection to the vBNS OC-12 backbone, and NCAR has an OC-3 connection to vBNS.
The NCAR vBNS router then put the data on the NCAR FDDI LAN, for connection to the Image Spreadsheet client system at the GOIN conference.
Figure 3-1 is a simplified diagram of this connectivity.
Figure 3-1: Image SpreadSheet WAN Configuration
GOIN Demonstration
4.0 Results
As this was the first time the Image SpreadSheet had been tried across so complex a network, it was expected that the performance obtained might be less than the full network capability. This was indeed the case. The conclusion is that considerable "tuning" is and will continue to be required for high performance applications in a WAN environment.
As for specific performance results, first note that the limiting circuit speed was DS-3 (45 Mbps). After accounting for the various levels of overhead, this leaves about 34 Mbps usable for applications. The DS-3 was the ESNet/Sprint drop to GAC -- all of the rest of the circuits were at least OC-3 (155 mbps) or FDDI (100 Mbps). The Round Trip Time (RTT) was measured at about 90 ms. Note that the use of the San Diego hop instead of a direct route more than doubled the theoretical minimum RTT. Multiplying 90 ms times 34 Mbps yields a delay-bandwidth product of about 380 Kbytes -- this would be the minimum TCP window size required to achieve a 34 Mbps throughput rate over a 90 ms round trip delay.
The nodes used in these tests were both from SGI -- the server is a 4 processor Onyx, and the client an O2. Both were running version 6.2 of the O/S. The highest network throughput obtained was the result of a memory to memory data transfer test -- TTCP modified to use large windows. This test resulted consistently in about 8.6 Mbps of throughput. So one unanswered question is: Why did TTCP only get about 25% of the theoretical throughput?
At a higher layer test, FTP throughput between these same machines produced 2.5 Mbps of throughput. Only about 30 % of the machines' demonstrated capabilities in the same WAN configuration using TTCP.
But for convenience, the Image Spreadsheet demos are normally run using NFS. This allows many server files to be accessible to the client without consuming the client's disk space. However, a file copy operation using NFS resulted in only 600 Kbps of throughput -- only about 25 % of what ftp could do! And more important, with all the factors present, the NFS throughput was only about 2 % of the capacity of the circuit.
Notes received from SGI indicated several potential improvements in the use of NFS, including using NFS 3 (TCP based) rather than NSF 2 (UDP based). However, initial attempts to get NFS 3 working were unsuccessful, and time ran out before additional attempts could be made. So the demo was run on NFS 2.
Note that these changes were at the "system" level -- they applied to the end system environment, specifically, the operating system and network stack. The changes did not apply either to the network itself, or to the applications programming.
The demo ran very slowly. It may be sufficient to state that at this performance level, the Image SpreadSheet is not a viable system. The conclusion is that considerable additional effort will be required to optimize the performance of this application (and probably most high performance applications) in a WAN environment.
The ability to perform these demonstrations in this environment at all was quite remarkable, and was made possible by the very knowledgeable and timely support of engineers from NASA/NREN, ESNet/Sprint, vBNS/MCI, and NCAR.
Attachment 2
Internet Statistics and Metrics Analysis (ISMA-97)
Workshop Report - http://www.nlanr.net/ISMA/isma97_report.html
May 1-2, 1997
San Diego, CA
Contents
Introduction / Goals
Findings and Conclusions
Sessions/Presentations
Participants
Agenda
Demos
Survey Questionnaires
ISMA Mail List/Other
Workshop Organization/Acknowledgements
Introduction
The Internet Statistics and Metrics Analysis (ISMA-97) Workshop was an invitational meeting for individuals involved in developing or deploying Internet traffic measurement or analysis tools. Sixty-five (65) people attended, representing Internet service providers (ISP), the research and education (R&E) community, vendors, and end-users. The meeting was held at the San Diego Supercomputer Center (SDSC) on the campus of the University of California, San Diego (UCSD). Sponsorship was by SDSC and the National Laboratory for Applied Network Research (NLANR), with funding provided by the National Science Foundation's Division of Networking & Communications Research Infrastructure (NSF/NCRI).
The goals for the meetings included:
* fostering awareness of existing and merging tools and measurement/analysis activities;
* seeking common ground between the diverse communities and encouraging greater cooperation and sharing of information among these groups; and
* seeking inputs on the priorities for the newly formed Cooperative Association for Internet Data and Analysis (CAIDA), see www.caida.org.
Findings and Conclusions
Participants at ISMA-97 were critical of current understanding of Internet traffic behavior and the overall inability of providers and users alike to provide real-time monitoring of internetworking traffic performance. However, the overall sense of the meeting was that each segment of the Internet community has legitimate interests and needs for assessing and monitoring varying granularities of Internet performance data. Participants emerged from ISMA-97 with a greater appreciation of the unique requirements of these diverse communities (provider, researcher, vendor, and end-user) and of the inadequacies of current tools and methodologies. As explained by Steve Feldman (Worldcom),
"After a while most of us seemed to realize 'measuring the Internet' means different things to different people. It all depends on where you are watching from, and what you're trying to find out. ...Modem connection statistics won't be too useful for backbone engineers, and your typical AOL user couldn't care less about BGP packet traces. But that shouldn't mean that it's not important to collect them; each is useful to somebody."
The need for better tools to monitor and analyze routing behavior and diagnose route flaps and related problems was viewed as the most important challenge facing internetworking today. Router vendors, according to participants, have been lax in upgrading the functionality of their products, particularly as pertains to improving network management utilities. Of notable concern is the inability of tools to explain adequately the phenomenon resulting from topology changes, e.g., routing loops or black holes. The availability of realtime topology information and data on adjacency state changes (asynchronously in a vendor-independent manner) would provide valuable assistance to providers, enabling them to detect problems (e.g., brownouts) and react in a timely manner. Current tools, such as traceroute, provide some insights but are limited due to protocol (and filtering) issues. The IP record-route option is better in consistency, but has a short horizon (9 hops). Dumping routing tables is another option, however, it is time consuming and does not provide information on 'when' a particular route changed. In the near-term, instrumenting routers to answer simple questions in a timely manner, e.g., via SNMP polling, would improve the quality and quantity of information available to network operation centers.
While many new measurement research initiatives will emerge over the coming months, ISMA-97 discussions highlighted the need for the R&E community to better articulate the goals and hypotheses of their network measurement research and to identify more precisely which problems are to be solved by these efforts. Such clarity will be critical to securing necessary partnerships with industry (ISPs and vendors).
Near-term attention to end-user requirements and tools is also imperative. There is growing demand for methodologically sound approaches to end-to-end performance measurement and monitoring/analysis of ISP services. Loss measurements being taken by DOE/SLAC on behalf of the high energy physics community and commercial ISP performance assessment conducted by Inverse Technologies were cited as examples of responsible, useful efforts at data acquisition and analysis by end-users. Approaches such as the recently released NetMedic tool were sharply criticized as lacking in scalability and technical accuracy.
As Daniel McRobb (ANS) explained:
"Users trust the tools (especially if there's nothing probably better), and that's often unwise. Some tools are inconclusive or lack fundamental rigors, and not all the users have the knowledge to interpret the tool's output. This is bad. What's worse is that I'm not certain of the rigor of any of the end-user tools I saw at ISMA, and some are even obviously lacking. I want to be able to use this data, as an end user and as a provider (esp. the latter). If it lacks the necessary rigor to draw anything but highly subjective conclusions, it's of little to use to me."
re scaling, "I don't think new traffic was the real issue here (I still question how many users would use such tools, and how often. After all, most of us just use the net; only a handful of us actually poke around when things go awry). I've never heard complaints about capacity issues w/ measurement traffic of the traceroute or ping type (and it wouldn't be a valid complaint anyway, by the numbers I have). However, tools that do hammer the network could be dangerous. I just don't think I've seen a measurement tool yet that had a significant aggregate impact. Of course, pathchar isn't widely deployed yet."
"I think the complaint I heard about scaling that I took to heart was the NET.MEDIC-like issue, where ISPs might be spammed by users from tools that contain little rigor. I'm not sure how we deal with this kind of issue as technical folks, except to tell people what they're doing right and doing wrong in terms of measurement, and keep our networks in good shape. I'm sure there's some fear just from exposure. I don't think any of the good providers are so worried about that (the louder people complain about real problems, the sooner they're fixed)."
re usability, "As a provider, I'd love to have valid end-user data. The more, the better. But if it's as disparate as sitting on hard disks on everyone's PC (note I'm assuming it's useful data), that doesn't do me (as a provider) any good. I want the data, but how do I get a large aggregate in a reasonable fashion? (of course Jamshid and others talked about some ways of moving in this direction)."
Correctly interpreting measurement data is extremely difficult, particularly for individuals who lack a thorough understanding of the underlying network topologies. This point was reiterated repeatedly throughout the workshop and used to highlight weaknesses in end-user measurement activities. According to participants, direct communications with ISPs regarding anomalies and overall measurement results should be a component of any large-scale measurement endeavor. However, this approach is not scalable except for large institutional users and measurement service companies.
Accuracy and scalability are fundamental challenges that the end-user community will need to address. As Curtis Villamizar (ANS) explained,
..."there was support for institutional end users that were making measurements to assess what they were buying, that knew what they were measuring and were systematic about their measurements, and that didn't add significant traffic load in the process of making measurement."
Current public attention is focused on tasks associated with data acquisition and related tools, yet accurate storage/warehousing, analysis and correlation of these data will be as difficult a challenge as actual measurement. These technical requirements will necessitate near-term attention and resource investments by providers, researchers (government and industry), and end-users alike.
Other areas identified by ISPs which hold promise for improving their ability to engineer and operate the Internet include:
* Net-2-Net and/or AS-2-AS matrices capabilities -- either through a form of flow capture or traffic sampling built into routers or the capability to do this with an external box. Such information would enhance providers' understandings of inter-network behavior and possibly lead to geographic precision (e.g., inter-city or inter-regional traffic patterns).
* a combined probe device and a BGP listener, permitting real-time route monitoring and aggregation of data at much higher levels than host-to-host, with reasonable accuracy. [Note that Cisco version 5 flow-export provides limited route monitoring capabilities.]
* improvements in the routers' capacity to accommodate varying levels of data aggregation [reflective of various communities' needs, at granularities permitting analysis in reasonable time frames] -- note that granularity issues like prefix-based policy must also be addressed.
* expansion of flow tools such as OC3mon and Netramet to other media, e.g., FDDI and SONET.
* improvements in our understanding of flows, e.g., the effect on flow profiles when changing timeouts or other mechanisms for defining flows and the accuracy of dynamic information (i.e., queue length estimates) available through flow analysis versus analysis of packet traces.
As with many meetings of this type, the hallway discussions were among the most important features of ISMA-97. Several potential new collaborations were brought to the attention of workshop organizers during and following the workshop. Several individuals also indicated that they plan to make specific modifications to their tools and/or changes in their measurement methodologies as a result of insights gained at the workshop. Increased use of Cflowd and OC3mon/Coral flow tools by participating ISPs and large institutional participants also appear likely, as does testing of new research tools (particularly tcpanaly and pathchar).
Sessions and Presentations
A summary of the ISMA-97 sessions and discussions and links to specific presentations is available at http://www.nlanr.net/ISMA/isma97_sessions.html.
Participants
ISMA-97 was an invitational meeting, targeting participation by select Internet engineers possessing hands-on experience in Internet traffic measurement and analysis. Sixty-five people representing ISPs, the R&E community, vendors and end-users attended the meeting, see http://www.nlanr.net/ISMA/isma97_participants.html. The high caliber of these individuals and the limited attendance were essential ingredients to ISMA's success.
Unfortunately, the workshop organizers had to turn away many capable and qualified individuals who expressed interest in attending the meeting. We would like to apologize to these individuals and urge the community to continue discussions of traffic measurement and analysis and related topics given their importance to the continuing evolution of the Internet.
Demos
Twenty (20) measurement / analysis tools were demonstrated during ISMA. ISMA's goal for these demos was to bring together tool developers and users in the hopes that constructive criticisms would foster broader understanding of existing capabilities and limitations and would identify potential areas for improvement. Feedback from participants suggests that the demos were beneficial, but that future meetings should include an opportunity for a brief summary overview of available tools prior to actual demonstrations.
Survey Questionnaires
In advance of ISMA-97, organizers distributed a survey questionnaire. The purpose of the questionnaire was to provide background information on the concerns and priorities of the participants and to assist in framing discussions during the workshop. The majority of the participants responded to the survey. Their inputs are grouped according to the community they represent: commercial vendors, ISP/exchange points, or R&E community. The questions asked in this survey are included below:
* In your opinion, a. Why is it important for ISPs to invest their limited resources (people, equipment, $$$s) in traffic measurement and analysis? What type of investments would reap maximum value for the ISPs? for end users?
* The research/higher ed communities and government are engaged in efforts to develop more robust tools and deploy a measurement infrastructure across the Internet. Do you have any recommendations for how these efforts can help to produce meaningful insights into current traffic conditions and scalability issues, including what aspects of measurement/analysis these communities should pursue jointly with ISPs?
* What role should other end-users (e.g., content providers, companies with mission-critical networking requirements) play in this process?
* What do you view as today's most critical challenges relating to traffic measurement and analysis, e.g., challenges wrt WAN measurement, measurement across ISP clouds, traffic flow characterization, network simulations?
* What should the community (ISPs, research, users) be considering wrt... acquiring, storing and analyzing data? level of detail of raw data and resulting analyses available to the general public? available to participating organizations? the appropriate organizational structures / strategies for collecting and analyzing distributed Internet traffic data?
* Are there any specific measurement or analysis tools you feel are particularly useful? please describe.
* Please provide any additional comments / position statements / URLs that you would like to share with ISMA participants.
Survey responses are available at www.nlanr.net/ISMA/isma97_survey.html and a codification of the responses in aggregate is at the bottom of the summary page: http://www.nlanr.net/ISMA/isma97_sessions.html.
ISMA Mail List/Other
Several relevant mailing lists discuss traffic measurement and analysis tools. The ISMA mail list (isma@nlanr.net) can be subscribed to by sending a request to isma-request@nlanr.net. The IETF's IPPM working group mail list is also a valuable source of information in this field, see www.advanced.org/IPPM.
A preliminary copy of the Cooperative Association for Internet Data Analysis' (CAIDA) Measurement Tool Taxonomy was distributed to ISMA-97 participants as background prior to the meeting. This living tool taxonomy provides an overview of Internet and TCP/IP performance measurement tools and efforts. Its development is sponsored by the National Science Foundation (NSF) and CISCO. The taxonomy is available at www.nlanr.net/Caidants/meastools.html.
Workshop Organization/Acknowledgements
ISMA-97 was organized by Tracie Monk and k claffy (UCSD/NLANR/CAIDA).
Many thanks to the staff at SDSC and UCSD who contributed to ISMA's success. In particular, we would like to thank Charlotte Smart (SDSC) for managing the logistical arrangements and Jay Dombrowski (SDSC) for managing the demonstrations of IP and ATM-based measurement and analysis tools and coordinating the MBONE arrangements.