National Laboratory for Applied Network Research

.

Current projects

Hardware upgrades to add new functionality

Testnet upgrade to OC-12

vBNS Backbone/Access Upgrade to OC-12

New Connections

• 12 New Connections approved
  • Iowa State University
  • Ohio State University
  • Old Dominion University
  • Rice University, Baylor College of Medicine, and the University of Houston
  • University of Chicago
  • University of Colorado at Boulder
  • University of Illinois at Chicago
  • University of Minnesota - Twin Cities
  • University of Pennsylvania
• 50 or more New Connections expected in 1997
• Connectivity to other research nets is currently: Argonne, ESNet, UIC, Dartnet IDREN, NASA Science Internet, CANARIE.

vBNS Routing

• primary peers: includes SCCs and New Connections. These peers receive routes originating from other primaries and secondary peers.
• secondary peers: includes NSP-approved institutions and networks. These peers receive routes originated only by primary peers.

BGP peering at the four NAPs ensures full Internet connectivity to the vBNS for Internet-connected approved researchers.

vBNS Multicast

Another element to the monitor was activated in November: the continuous feed of active flow summaries from all monitors. This active-feed feature reports detailed information all active traffic flows in near-real-time. Working in collaboration with NLANR researchers Jamshid Mahdavi and Matt Mathis, this data feed was combined with their graphic display java applet to show dynamic vBNS traffic flows as both instananeous and averaged rates. We successfully demonstrated the overall monitoring capability at SC'96 at both the NLANR and vBNS booths. Supported by the vBNS native IP multicast service, active feeds were multicast from vBNS monitors and displayed at the booths simultaneously. Active flow feeds can now be made available to NLANR researchers via the same methods.

IP Integrated Services Development

MCI also is working with Prof. Hui Zhang and his students at CMU to integrate hierarchical WFQ code into the Netstar Gigarouter. MCI has provided the interface structure and code examples and is setting up a development environment for them for this purpose.

CBQ and H'WFQ are the two major packet scheduler algorithms we plan to evaluate in the first half of 1997. Using vBNS traffic, the goal is to get production environment experience with link sharing and measurements of the worst case packet delay. We have integrated but not yet fully tested a packet classifier based on CBQ code

An integration interface structure similar to the one for the FDDI card has been designed and is now being implemented on the Netstar ATM card. We hope to deploy the same packet scheduler and classifier code on this new interface.

Another important component of integrated services is traffic admission control. Realistic traffic loading is required for evaluation of a measurement-based approach to admission control. MCI talked a bit with Prof. Sugih Jamin at UMich on on how to integrate his code into Netstar and to evaluate the algorithm in the vBNS environment. MCI is still working on the NDA issues.

As part of the work on integrated services, there are plans to produce a general integration structure on both the system and the interface cards and to provide it as a package to other researchers using Netstars. On the router system, RSVP code has been ported but not yet tested. A set of configuration utilities will be written in the interim. We are still studying of scalability issues on the backbone, with mechanisms based on the ideas of traffic profiles and bit-marking proposed by Dave Clark. Evaluation of the effectiveness of these schemes will involve submitting vBNS access traffic through a Netstar router that implements such mechanisms.

RSVP

Realize vBNS upgrade to OC-12

Improved Reliability and Performance Guarantees

New Connections

ATM signalling

OC3 Traffic Monitor

Integrated Services

Statistics are available at http://www.vbns.net/reports/1996/. Joel Apisdorf at MCI has fielded the OC-3 packet/cell monitor. (http://www.nlanr.net/NA/Oc3mon/), which ports the functionality of the SDSC flow software (described later in this report) to a stand-alone passive monitor (http://www.nlanr.net/NA/Oc3mon/) that can attach directly to serial (e.g., OC-3) optical fiber. K. Claffy, Joel Apisdorf, and Rick Wilder presented a paper on this monitor at Usenix LISA '96.

Cornell Theory Center (CTC)

Personnel

Activities

Cornell connected their campus FORE-based network to the vBNS Lightstream with an upgrade of the FORE ASX-200 switch software to version 3.4. They also connected their IPOP machine via 155 Mbps ATM interface to this FORE switch and successfully tested IP connectivity across the vBNS, as well as testing PVC-based IP connectivity from NYNET through this FORE switch to the vBNS. These PVC connections between NYNET and the vBNS were used for demos from Rome Laboratories to SC95 and for demos from Syracuse University for SC95 and to the CRPC meeting at ANL.

CTC established a desktop video conference routed across the vBNS for a vTCC, using a CU-SeeMe reflector (logic.tc.cornell.edu). The reflector attached to a multicast group with nv clients using CU-SeeMe encoding and CU-SeeMe clients connected to the reflector. The conference worked but had few participants. When trying to add a second CU-SeeMe reflector to this multicast group, it uncovered a bug that multiple CU-SeeMe video streams between the reflectors were not distinguished across the multicast path and were combined as one. Bruce Johnson will work with the developers to resolve this bug.

CTC extended the FDDI LAN which connects to the vBNS routers from Frank H. T. Rhodes hall to the Computer and Communications Center to connect to the two Cisco 7000 routers which provide Cornell's "commodity" path to the Internet via NYSERNET. Cornell then installed new FDDI interfaces in these routers and established BGP peering with the vBNS Cisco router. Cornell is currently advertising routes to all three of their Class B addresses, 128.84.0.0, 128.253.0.0, and 132.236.0.0, since subnets of these have been intermixed on the Campus. However, work has begun to investigate consolidation of the Theory Center's Class B, 128.84.0.0, such that only one network or two quarters of 128.84.0.0 would be dynamically routed via the vBNS. Working with MCI, Cornell completed configuration and testing of BGP peering between Cornell's two Cisco 7000 routers which connect the Campus to NYSERNet and the vBNS routers.

Several special static routes were configured via the CTC HIPPI network for high bandwidth applications. These included routes to NCAR/UCAR for the HPCC Grand Challenge Geophysical Turbulence Project, ATM PVCs to Syracuse University via NYNET and Argonne National Lab for Roman Markowski's demo at the CRPC meeting, and a route to PSC to enable researchers in New York City, who were bottlenecked by the commercial Internet connection to Pittsburgh, to obtain high performance connectivity. With the current IBM SP2 implementation at CTC, these routes must be statically configured on all 512 nodes to route through a node which is attached to the HIPPI network- a manual procedure for removing these routes in the event of a vBNS outage has been established and CTC is working with IBM to make this type of routing dynamic.

Phil Pishioneri contributed information on TCP tuning for high performance data transfers for AIX systems to Jamshid Mahdavi's Web page. Phil also established an experimental mbone tunnel from logic.tc.cornell.edu to a multicast router in the Program of Computer Graphics to enable video conferencing between PCG and collaborators at SDSC.

Cornell demonstrated a version of CU-SeeMe which was modified for high bandwidth over Cornell's ATM network- with a maximum of 1 Mb, it produced nearly 30 frames per second of video and good quality audio.

CTC replaced the FORE ASX-200 switch which connects to the vBNS Lightstream 2020 with a FORE ASX-1000 switch. The ASX-1000 also connects to other ASX-200s at Cornell and to the IBM 8260 which has ATM connections to all AFS servers, several SP2 nodes, and RS6000s acting as routers to a FDDI network.

Using a Cisco 7507 on loan from MCI, Cornell loaded Cisco's beta RSVP code and is working with MCI and NCSA on testing and preparing for a demo at SC96. Cornell will connect from the FDDI of the Cisco to a SGI Indigo running IRIX 6.2 with RSVP and to a FORE 7000 PowerHub which will route to dedicated ethernets to two SGI Onyx machines running 6.2. The SC96 RSVP demo will use SGI Indys at CTC and NCSA generating vic streams to an SGI Impact on the other side of an MCI supplied Cisco 4700 router in the NLANR booth- one vic stream will utilize an RSVP reservation and the other will not. A machine generating traffic will create congestion on the ethernet with the 4700 and Impact.

National Center for Atmospheric Research (NCAR)

Large Dataset Transfer

Routing and ATM PVC configurations for CTC to NCAR connectivity were completed pending startup of the Large Dataset Transfer file transfers between NCAR and NLANR worked with CTC and PSC to provide connectivity to PSC C-90 and NCAR J-9 for Don Middleton's Cray ftp client to be used for the NCAR Large Dataset Transfer and the CU Turbulence applications. Large files (25 G bytes nominal) for these applications were successfully transferred between NCAR and the PSC Mass Storage System (MSS) via tuned, multiple ftp sessions over the vBNS. Throughput was observed in the 20-40 Mbps (avg) range using ncftp and in the 70 Mbps range with the nettest/nettestd utility.

Don Middleton has made a BSD ftp client with TCP RFC 1323 performance extensions binary available via the LDT WWW homepage. Through the course of our research into TCP performance research we have found that while we are capable of tuning the UNIX kernels of our machines for the performance extensions outlined in RFC 1323, most of the applications that use TCP as the layer four transport mechanism such as ftp and PVM, do not facilitate such performance extensions. Our work with a tunable ftp client/server, while verifying this assumption, has allowed such applications to take advantage of tuned TCP and increase overall performance.

Internet Data Distribution System

While the original Unidata application was submitted in 1995, the vBNS AUP has hampered progress of expansion of the test system that would benefit the wider Unidata user community. As a result of the announcement of the NSF New Connections '96 awards to IDD-server university, Chris Fair and Marla Meehl met with Unidata personnel to discuss resubmission of the application for vBNS experiments involving the Unidata Internet Data Distribution System (IDD). Initial connectivity will be set up to those sites with progress toward other IDD sites as the New Connections reach other IDD university sites.

Distributed Climate System Laboratory Project

NCAR and SDSC DCSL researchers worked on MPI_IO, a parallel I/O stream architecture built on top of the MPI message passing library and parallel I/O streams for network applications such as ftp to extract better performance from the vBNS. We have not yet identified an application to take advantage of the library, although Don Middleton is interested in using it for the LDT application. http://www.scd.ucar.edu/vg/DCSL/DCSM/DCSL.V1_1.html

HPCC Grand Challenge Astrophysical Turbulence (University of Colorado)

The application for the University of Colorado turbulence application was originally submitted in August 1995. At that time, WESTNET/CU did not have the required high-speed connectivity to NCAR or any other vBNS access point. As a result of the delay NCAR resubmitted the application in March 1996.

NCAR, PSC and CTC worked to establish ATM connectivity over the vBNS to the Crays at PSC and the SGI Onyx at NCAR and the IBM SP2 at CTC and the SGI Onyx at NCAR, in support of the expected U. Colorado turbulence application. Then starting in April 1996 they supported the OC-3 connectivity between NCAR and CU, including Permanent Virtual Circuits (PVC) support via the NCAR ATM fabric to four hosts on the CU campus. http://lcd-www.colorado.edu/Home.html

Other Activities

NCSA is working with Paul Woodward, from the Laboratory for Computational Science and Engineering (LCSE) at the University of Minnesota to demonstrate similar capabilities over the vBNS, using facilities at LCSE as well as Boston University as soon as the Minnesota and BU high performance connections are in place (both awarded High Performance Connections grants). As of October 1996 NCSA demonstrated the ability to seamlessly queue jobs between SGI environments at BU and NCSA using the LSF scheduling software from Platform Computing. They have been working with SGI to address limitations of their high performance MPI library, specifically those related to the very large messages needed in high speed applications (e.g, 40 Mbyte messages for the LCSE PPM code). Using innovative methods to overlap communication and computation, Woodward's team is achieving tolerable delays (in the range of 100ms), rendering this application ideally suited to the vBNS.

Remote Instrument Control using vBNS

Application scientists at NCSA and the University of Illinois Beckman Institute used the vBNS to remotely run EVAC (Virtual Environment for Control of Remote Imaging Instrumentation) between the Beckman Institute and the Scripps Research Institute in San Diego. The demonstration, which was for the ``Molecular Microscopy in the 90's''; workshop at Scripps, required more bandwidth than available via the commodity Internet. NCSA networking staff coordinated routing among NCSA, the University of Illinois, MCI vBNS engineering, SDSC, and Scripps.

Remote Volumetric Visualization

Computational scientists from the research group of Paul C. Lauterbur of the Department of Chemistry at UIUC, NCSA, Cornell's chemistry departement, and the Cornell Theory Center (CTC) have used the vBNS for computation-intensive research in magnetic resonance imaging (MRI). Diffusional Enhancement of Signal Intensity and REsolution, or DESIRE, is a new paradigm for MRI at resolutions far below ten microns, the current state of the art. Numerical simulations governing partial differential equations of motion have a crucial role in the design of experiments and the determination of fundamental, physical limitations. Simulations are running on the 512-node IBM RS/6000 SP2 at the CTC by way of a 28,000 SU allocation awarded to Paul Lauterbur. A typical, single, simulation run generates 2 gigabytes of data on CTC storage devices which must be processed and visualized by scientists at UIUC; up to 100 simulation runs are planned for the current SU allocation. The experience of the research groups has made clear that remote access to data generated at the CTC and collaborative sharing of results would not be feasible without the vBNS.

Paul Lauterbur has presented results of simulations to date at the XVIIth meeting of the International Conference on Magnetic Resonance in Biological Systems, Keystone, Colorado, August 18-23, 1996.

International High Performance Distributed Applications

NCSA worked with NSF NCRI's international connections program and the White House G7 program to solicit and process proposals for collaborative applications that could utilize high performance international connectivity. Well over 50 applications, from researchers from all parts of the US, Europe, and Asia. were submitted and are now documented. NCSA produced a hard copy document detailing the applications, distributing it at the Paris GIBN meeting in February. This document later became the standard that the other GIBN partners used as a model to summarize additional GIBN applications. This has the potential to greatly increase the number of distributed applications that can utilize the vBNS. NCSA is already targeting a major application for use over the high speed link between the United States and Europe, building on NCSA work in homogeneous wide area metacomputers (see above section on Latency Tolerant High Performance Distributed Applications): distribution of a job between NCSA's SGI Power Challenge Array and the Max Plank Institute at Potsdam Germany.

Application Performance Tuning Primer

As a result of working with several dozen application teams during FY95 and early FY96 in the context of the I-WAY project, Von Welch developed a primer for using TCP window scaling. This user guide is one of several NLANR resources available to users in order to help them achieve high performance over the vBNS.

Performance Tools, Measurement and Tuning

Tuned ftp Tools for vBNS

NCSA ported public-domain ftp tools to their SGI Power Challenge array and modified them to be able to use large TCP window sizes, allowing users to take full advantage of the vBNS high-bandwidth, resulting in ftp throughput between NCSA and PSC increasing by nearly an order of magnitude from 2 megabytes/second to 13 megabytes/second. ( ftp performance details here)

Internet End-to-end Performance: The User View

In response to many user complaints about Internet performance Von Welch and Charlie Catlett initiated a long term quantitative study of Internet performance between NCSA and a number of key remote user sites. The testing, which spans from March of 1996 to the present time, involve throughput, latency, and packet loss measurement. Treno, from PSC, was also used to attempt to determine bottlenecks.

The tests have already proven useful in documenting the actual performance impact on real users of several changes in networking infrastructure. Two sites (OSC and UCLA) upgraded from multiple T1s to T3s during the course of the study. Comparing data before and after these upgrades reveals the impact on end users, jumping by nearly an order of magnitude in both cases.

The study also documented the performance degradation caused by a problem in a network link and the resulting improvement when it was corrected. The study documented the exact day and time when performance to U. of Arizona fell sharply. This lent networking personnel there definitive proof that a serious problem existed. After much debugging, they determined the problem was in Arizona's link to their ISP.

vBNS Performance Testing: U. of Illinois/Scripps

Although performance seen between the University of Illinois and Scripps Lab for the aforementioned EVAC demo was sufficient for the demo, it was much worse than expected. NCSA staff ran performance tests between different end points on the vBNS in an attempt to isolate the problem. The testing indicated that the performance bottleneck was not on the vBNS, but instead exposed a bottleneck on Scripps' internal network. Staff at Scripps believe they understand the (firewall-related) problem and are deploying a solution. The performance tests also serve to document vBNS performance so that we have a baseline for future investigations.

NLANR Monitor/CICNet Traffic Monitoring

As a result of the highly-successful I-WAY demonstration at SC95, NCSA UIUC-CS dept, and ANL/EVL submitted a proposal to ARPA to fund a persistent version of the I-WAY distributed applications testbed. One component of this proposal was to develop real-time network performance tools, leveraging MCI's OC3MON, SDSC's flows research, and Dan Reed's Pablo project at UIUC/NCSA. Pablo was demonstrated during the I-WAY project, and has also been used to examine real-time performance data using virtual environments to allow scientists to interact with system performance information flows such as from networks, parallel applications, or information servers.

The goal of the networking performance tools is to facilitate a non-specialist in networking to analyze and tune application performance. Currently end users access to real information about the network behavior of a given applications is limited to what they can get from networking specialists, and typically via the use of insecure promiscuous traffic sniffers that create privacy and security concerns.

The developed network performance assessment system will have authentication, authorization and traffic filtering capabilities so that users can access data about their network application in a secure manner. The group will also develop an API to allow the system to be integrated with existing networking libraries, e.g., MPI.

Since writing the proposal, NCSA has also joined in a CICNet-wide effort to gather IP flow statistics in pursuit of a better understanding of local traffic flow characteristics. An PC with a FDDI interface will serve as a monitoring platform for NCSA's Internet DMZ. A version that can monitor an FDDI network with the Flow software from SDSC has been ported and is ready for field testing. The remaining PC and ATM hardware is on order and should arrive by January 1997, at which point NCSA can port an ATM version of the software.

vBNS RSVP Testing

NCSA is working with MCI and Cornell to test and deploy early implementations of the RSVP reservation protocol from both Cisco and SGI over the vBNS. NLANR will demo RSVP over the vBNS at Supercomputing '96, comparing performance of an application competing with a `network-hog' application both with and without RSVP support.

Programmatic Support and Promoting vBNS External Connections

Support for High Performance Connections Program

During the past year, NCSA staff have worked with a number of sites to help them understand the operational and technical aspects to engineering and proposing a high performance network connection. This involved assisting them in selecting promising driving applications, helping them understand the technologies involved in connecting LANs with high performance WANs, and giving them guidance on the economics of high performance telecommunications offerings.

G7 Global Interoperability of Broadband Networks (GIBN)

NCSA has been working closely with NSF NCRI's international connections program to provide technical advice regarding the White House G7 initiatives such as Global Interoperability of Broadband Networks (GIBN) project. Details about the GIBN project applications are at http://www.ncsa.uiuc.edu/General/GIBN/. Charlie Catlett and Randy Butler have been actively involved as USA technical advisors to the project, focusing on interconnection of GIBN partners. One plan is to build an interconnection site in the USA that would host links to Japan, Canada (and from there to Europe) as well as to the major USA research networks led by the vBNS. While the proposed interconnection plan did not meet with approval it did bring together the carriers (AT&T, Sprint and MCI) and networking leaders at NSF and DOE to discuss interconnection issues and strategies.

A direct result of the GIBN activity is the interconnection between CANARIE and the vBNS via the Chicago NAP, and concomitant plans to for a CANARIE-vBNS application demo in the NLANR booth at SC96. During FY97 we expect to use this interconnection to demonstrate wide area homogeneous metacomputing with the Max Planck Institute using Silicon Graphics Origin-2000 supercomputers. We will also work with the European Community Advanced Communications Technologies and Services (ACTS) program to use the interconnection of the vBNS, CANARIE, and the European ATM testbed infrastructure to demonstrate the use of shared virtual environments for supporting collaborative engineering in the context of vehicle design and manufacturing.

CEWES applications

NCSA worked with Interim-DREN staff and NSF to drive the peering of the vBNS and iDREN as required to support environmental applications needing coupled virtual environments between the US Army Corps of Engineers Waterways Experiments Station (CE-WES) and NCSA. NCSA is assisting CEWES in generating an Immersadesk application for data collected at Chesapeake Bay. The Chesapeake Bay Immersadesk application models salinity, chlorophyll, and dissolved oxygen content, in addition to inflow and outflow from the many rivers that enter the bay and the Atlantic tidal flow. I-Desk to I-Desk collaboration currently allows a researcher at one location to follow a researcher at another location while s/he navigates through the data space. Allowing each collaborator to watch everything his/her counterpart is doing in the Bay (data space) requires a high-speed, reliable connection. This application will serve as a stepping stone to more in depth collaborative applications. CEWES is planning to use this application at Supercomputing '96 in the Department of Defense exhibit. The I-Desk at NCSA will also be at SC '96, and some interaction between the two exhibits will take place.

Interop Panel on NLANR Activities

Charlie Catlett gave an overview of NLANR activities on a panel at Interop'96 in April along with George Strawn (NSF) and Charles Lee (MCI). Strawn's presentation gave an NSF policy perspective, Lee's an MCI operational view, and Catlett's a research and applications overview.

Supporting vBNS Connectivity to NCSA Computational and Information Resources

Routing Deployment

During the past year NCSA made significant advances in integrating the vBNS with its internal network. A route server was designed, tested and installed in operation to do BGP peering with the vBNS allowing for automated exchange of routing information. The system was successfully tested on June 12 when NCSA lost their Internet connectivity due to a failure of a MCI router at Downers Grove. In response to the crisis, NCSA and MCI vBNS Engineering staff cut over NCSA's Internet connectivity to the vBNS for the day.

NCSA LAN ATM Migration

NCSA developed and implemented a plan to convert thier internal FDDI backbone/shared Ethernet LAN to an ATM based LAN with switched Ethernets. This deployment, planned for completion by the end of 1996, will not only offer users higher bandwidth but also the means to runs application directly over ATM from NCSA machines across the vBNS. Direct high speed coupling of users to the vBNS at high speed will hopefully encourage the development of applications.

Pittsburgh Supercomputing Center (PSC)

In addition to general descriptions of our work in these focus areas over the lifetime of the project, a detailed 4th quarter report is included to cover work not reported on elsewhere.

Operational vBNS support

Over the course of the project we have provided support to numerous vBNS applications, documented in quarterly reports and elsewhere online. In addition, we have supported several demonstrations of vBNS capabilities at conferences and meetings.

Organizational Support of vBNS and NLANR activities

The PSC played a leadership role in planning for an NLANR booth at the Supercomputing'96 conference. At this conference, we hope to demonstrate both the research aspects of the NLANR project as well as the capabilities of the next generation of high performance networking technology. Some of the planned demonstrations include Resource Reservation capabilities (RSVP), next generation Internet protocol (IPv6), OC3 Monitoring capabilities (OC3-mon), TCP Selective Acknowledgments (TCP SACK), and WWW caching. In addition, several applications which demonstrate the capabilities of high performance networks are planned.

Network Performance Research

In addition to work on TCP protocol enhancements, we have been studying theoretical models for TCP performance, and relating them to Internet measurement efforts. PSC has teamed up with Vern Paxson at LBNL in a proposal to research scalable techniques for measurement of Internet performance. This proposal was submitted to NSF earlier this year and is working its way through the authorization process.

Education and Demonstration of New Technologies

The capabilities for collaboration using high performance networking technologies are not well understood outside of the small group of researchers who work with these technologies on a day-to-day basis. In order to demonstrate the possibilities for use of the network in educational and collaborative environments, the PSC took advantage of an MBone talk presented by Chris Fair at NCAR in June, 1996. This talk was displayed live in two large lecture rooms at PSC utilizing vic, vat, and netscape. Roughly 75 people including staff at PSC, members of the CMU community, and about 20 PSC users who were on site for training were present for the seminar.

PSC Report on Detailed 4th Quarter Activities

vBNS Support Activities

In addition to these hardware activities, our engineering staff configured BGP peering with the vBNS and performed general routing maintenance throughout the quarter. We also performed various monitoring and troubleshooting activities throughout the quarter to ensure that PSC users were getting the best possible use from our vBNS connectivity.

vBNS/NLANR Organizational activities

Mr. Mahdavi spearheaded the effort to include an NLANR booth at the SC96 conference. The NLANR booth will showcase both the vBNS activities and networking research activities of recent months. Some examples of technology demos planned for the NLANR booth include IPv6, SACK TCP, use of the OC-12 vBNS Testnet, and the MCI OC3 monitors.

Mr. Mahdavi attended the High Performance Campus Networking (HPCN) meeting in Boulder, Colorado.

Networking Research Activities

Applications

Heterogeneous MetaCluster
http://www.nlanr.net/VBNS/Alloc/metacluster.html

SDSC

SDSC NLANR published papers (http://www.nlanr.net/Papers/)

Coordination and Administration of the Internet, Harvard JFK School workshop, August 1996:

1. cooperation in Internet data acquisition and analysis (with Tracie Monk) describes the proposed Caida (http://www.nlanr.net/Caida/)
2. evolution of the nlanr cache hierarchy (with Duane Wessels) describes experiences with the NLANR caching hierarchy
3. address administration in ipv6 (with Eric Hoffman) proposes a geographically based metro addressing model
4. in whose domain: name service in adolescence (with Don Mitchell and Scott Bradner), commentary on the undue community attention to the Top Level Domain issue while other more important, but conceptually more complex, issues remain ignored.

IEEE 1996 IEEE Symposium on Information Visualization, San Francisco, 30 October 1996

• Visualizing the Global Topology of the MBone

Usenix '96 LISA, Chicago, 3 October 1996

• oc3mon: flexible, affordable, high performance statistics collection (with Joel Apisdorf, Rick Wilder and Kevin Thompson) describes the MCI OC3MON and provides sample OC3MON data from MCI bakcbone

• `but some data is worse than others': measurement of the global Internet

SDSC NLANR presentations/meetings (http://www.nlanr.net/Presentations/)

1. invited presentations to Federal Networking Council meetings, February and October. (K Claffy and Hans-Werner Braun)
2. met with NSF regarding next steps for NLANR in the context of recent developments (Internet II, next generation stuff, new connections), as well at the CAIDA efforts and their relationship to NLANR. (K Claffy)
3. IETF plenary session, June 1996 (Montreal), to present video shorts of various NLANR visualization projects. (K Claffy)
4. October NANOG meeting (Ann Arbor), focused on measurement tools and describing the OC3MON architecture and data examples. (K Claffy)
5. visited the Geometry Center in Minnesota to discuss using their very capable visualization tools and expertise on Internet data. (K Claffy)
6. presentations all over California (some Mboned) on the Caching project: NCAR, IETF (Montreal), FIX-west, Ipsilon, Digital-NSL, AOL, Xerox Parc, SDSC (Duane Wessels)
7. invited to give talks at special workshop on caching in Warsaw, Poland. (Duane Wessels)

SDSC NLANR software/web repositories released/supported (http://www.nlanr.net)

1. http://www.nlanr.net/Caidants CAIDA network tool set, used to support NLANR Internet visualzation projects
2. ftp://www.nlanr.net/Traces maintaining current online address-masked trace data (packet traces from FIX-west, cache logs from the NLANR caching system) (e.g,. FIX-west, NLANR caching system), for the research community to use in traffic studies.
3. ftp://www.nlanr.net/Software/Tcpdpriv software to hide raw network address from tcpdump file output, in order that it may be released to the research community safe from violating privacy considerations.
4. http://www.nlanr.net/INFO maintaining a respository of provider-supported Internet statistics and
5. (http://www.nlanr.net/Viz/Deunion) Internet data visualization projects, many of the latter being NLANR-supported

Cooperative Association for Internet Data Analysis (CAIDA)

Examples of statistics visualization that would be useful include:

1. Mbone tunnel topology to illustrate the multicast tunnel infrastrcuture that carries an increasing proportion of network traffic, but unfortunately often with suboptimal configuration.
2. workload flow profiles, including by application, source-destination matrices (by country, although other granularities are possible), and graphic illustrations of how packet, byte, and flow durations correlate for a given protocol. The flow analysis methodology and (gzipped, tarred) previous unix software and OC3MON platform software are publicly available. K Claffy, Joel Apisdorf, Kevin Thompson, and Rick Wilder of MCI/vBNS submitted to and presented a paper, oc3mon: `flexible, affordable, high performance statistics collection', on OC3MON at Usenix LISA '96. Future OC3MON plans include refining the statistics collected, support xterm-based remote control rather than only a PC, porting to unix, supporting AS matrix instead of classful network extraction from IP address, and integrating FPGA design tools.
3. Performance characteristics of a path from a fixed server in the Internet to a specified destination host. Jamshid Mahdavi and Matt Mathis have deployed the Treno server at PSC to perform this function via a form-based web interface, where the user may set parameters according to his performance interest.
4. Lower-cost performance metrics. One drawback of many commonly used performance tools is their somewhat aggressive use of the path to measure the path itself. We seek to minimize the burden imposed on the network, and NLANR seeks to foster the development of other tools and supporting models that will enable estimates of performance statistics with minimal imposition on the network. Inspired by Braun's extended low-impact measurement study (using a modified version of ping to timestamp packets at a microsecond level) and visualization, Claffy and Hoffman have been developing a tool to explore the quality of underlying unicast paths in such end-to-end measurements. Results of initial tests with this tool were the subject of one of the NLANR videos shown at the June IETF.
5. Visualization of caching infrastructure and a daily update of the caching hierarchy. Although these are rough VRML-based visualizations, we are currently developing tools similar to the Mbone tools above to enable greater manipulation of topological information.

Internet infrastructure information repository

Caching system prototype

In line with the goal of using the information gathered from network analysis to improve the performance of the architecture, NLANR continues work on its information caching prototype, and has submitted a proposal to NSF for continued evolution of the system. A key result of Internet workload analysis has been the recognition of the existence of an increasing proportion of web traffic as measured at reachable locations, as well as simulation results that indicate that caching could leverage the high degree of redundancy in web document transmissions.

Participation in the NLANR caching project continues to grow. We have upgraded the machines from 128 to 256 Mbytes of RAM for improved performance. The caches originally used the original Harvest cache software, designed for performance and hierarchical scaling. Since May 1996 we have been using the public domain squid software, a community effort led by Duane Wessels, NLANR technical PI for the caching project. Peter Danzig, architect of the original Harvest cache, is providing commercialization support for the Harvest cache.

international participation is still extraordinary, and dwarfs U.S. interest in caching in general. We began in July to place restrictions on using the caches, to encourage a more sound hierarchical topology. We announced this policy in May in response to visualization analysis of the neighbor database. An example configuration for access control at each site is generated by ranking all NLANR clients for the month of June by number of requests. Low ranking sites from countries with higher ranking caches have been dropped.

NLANR operates a number of mailing lists related to this project:
ircache, ircache-dev, squid-users, squid-bugs, cache-coders. Requests to info@nlanr.net.

Additional Information
Sample Statistics are available, as are Issues, Future Work, Open Research Questions, usage patterns, network configuration, VRML Visualization, Getting Caches into NAPs, as well as code status, published access logs, cache registration service, daily access summaries and statistics, and problems encountered.

Multicast infrastructure

NLANR continues to support multimedia conferencing and supporting multicast communications infrastructure to use and provide a development environment for such tools on the vBNS.

Claffy is also working on developing tools for Mbone visualization, as mentioned earlier, with Tamara Munzner at Stanford, Bill Fenner at Xerox, and Eric Hoffman at Ipsilon. Their paper, Visualizing the Global Topology of the MBONE was accepted to Information Visualization '96, a symposium associated with IEEE Visualization '96, to be held in San Francisco, Oct 27 - Nov 1 1996. The tool outlined in the work enables one to zoom in on logical subsets of the Mbone infrastructure such as the vBNS-related subset of tunnels, illustrated to the right.

Collaboration environments

NLANR still supports text-based collaborative environments, two in particualr: NLANR moo, an environment for NLANR researchers to chat, explore, and build within an online community, and IPNmoo, a collaboration with the Routing Arbiter's IPN project (established and led by Craig Labovitz) to provide an online collaborative environments for use in real-time ISP communication to debug network problems. Both moo's have already proven a useful resource for the IPN community. ( http://www.nlanr.net/Ipnmoo)

Toward development of an undergraduate Internet engineering curriculum

Claffy has hired some students to start building a prototype web repository of links to resources for an an Internet engineering curriculum ( http://iec.nlanr.net/). Intended as resource for instructors at all levels, as well as students, the repository will facilitate sharing of instructional resources (e.g., slides, problem sets, exams, outlines, notes).

NLANR agenda for coming year

1. helping NCRI to provide a viable network environment to the R&E community, in order that scientists can take maximum advantage of community computational resources, many of which are also provided by the National Science Foundation (i.e., the supercomputers (vector and mpp), visualization/graphics engines, workstation clusters, data archives, mass storage, and AFS services of the NSF-sponsored supercomputer centers)
2. explore larger issues in evolution of the commodity Internet, including metrics and statistics of workload and performance, cooperation among ISPs, charging and settlement models.

NCRI recognizes that many of the networking as well as computational technologies will find their way outside of the NSF-spawned R&E environment, just as they did with the NSFNET program. NCRI continues to take care to optimize the interaction between the research infrastructure and the market-provided infrastructure, while pushing the technology envelope gracefully, such as with the recently announced NSFNET Connections Program. The neutrality of the constituents of NLANR also make it a good basis for infrastructural and interagency network activities, such as assisting NSF with this program, and continued collaboration among U.S. federal agencies. NLANR facilitates collaboration in conjunction with various other organizations, some of which are actively supporting NLANR activities.

For the remainder of the NLANR cooperative agreement, currently scheduled to end in October 1997, we plan to gear our activities towards supporting the larger NSF high performance connections communnity, leveraging off the experience of the initial vBNS sites over the last two years. Over this time, NLANR has served the NSF-sponsored R&E networking community in at least four separate capacities:

1. helping R&E users effectively run their applications on the vBNS
2. providing technical and operational support for NSF connections equipment and configuration at each site
3. providing technical feedback relevant to NSF connections policy issues
4. pursuing and coordinating forward-looking research and immediately apply to NSF research infrastructure

For the remainder of the duration of the cooperative agreement, as well as for follow on agreements we expect to jointly support, NLANR would like to serve NSF high performance connection awardees in the following capacities:

1. user support: 5.25 FTE (user support, tracking, and training: 4.25; public relations: 1)
   Each user support person would be responsible for seeking out and supporting a set of NSF-approved applications. User support would include: providing user education through multicast conferencing as well as on-site workshops; providing written reports; and arranging for demonstrations at fora such as SC97. Since the NLANR user services group will be distributed across the five sites (as well as perhaps at new sites), a specific 0.25 FTE would be responsible for coordinating this distributed organization.

   Throughout the year, the NLANR user services group will schedule higher performance network usage training workshops to cover specifics of the infrastructure as well as general high performance networking technologies and issues such as ATM, RSVP, MPI and tips for achieving high performance over high-speed, high-latency networks. NCSA will coordinate the user support efforts spread across multiple sites, including identification of application project opportunities, matching these with user support staff, and ensuring that plans and results are made available.

   To aggressively attract more applications across the new connections sites, NLANR will seek strategic partnerships with other networks, finding and assisting end users in using the infrastructure to full advantage, and further pursuit of latency-tolerant applications such as those already started with Boston U. and LCSE/Minnesota. We will continue to support existing applications, tuning where necessary to take advantage of the FY97 OC-12 upgrade.

   Of particular interest during FY97 in the application area will be the support of coupled virtual environments (see CEWES application above), involving CAVEs as well as Immersadesks and Infinity-Walls. In addition, latency tolerant applications such as the LCSE/Minnesota application will be pursued as more high performance resources become available through the NSF high performance connections program.

   The user services group will also continue to be available to assist users with technical and operational considerations to engineering a high performance connection, and developing applications to utilize it.

   We will establish and maintain a web page describing how to administer a responsible NSF New Connections: http://www.nlanr.net/VBNS/goodneighbor.html.

2. technical/operations support coordination: 2.5 FTE.
   PSC will coordinate technical discussions, i.e., planning general meetings of new sites, perhaps co-located with NANOG meetings. NLANR will also monitor the performance of applications running currently over the vBNS and new connections research infrastructure concentrated mainly on the large number of un-instrumented applications that have no way to determine what level of network bandwidth they are consuming. Finally, NLANR will continue to forge technical exchanges with CANARIE, DREN, ESNET, and other high performance network infrastructure groups during FY97.

3. technical feedback on policy .6 FTE (PI responsibilities)
   to represent the research community beyond the SC centers.

4. research coordination and supporting tool development 2.5 FTE
   1. IPv6 experiments, in addition to Mbone support that the vBNS and sites have provided this year.
   2. visualization tools ( http://www.nlanr.net/Caidants) and projects (http://www.nlanr.net/Viz/Deunion), including topology mapping for the Mbone, NLANR caching system, and IPv6bone, and progress to mapping more than just the topology but actual traffic/routing flow across the infrastructures. In addition, Intervu Corporation has approached NLANR with one of their projects which has yielded a lot of delay and throughput data from a wide variety of network locations. We will pursue visualization of their data set. We also plan to collaborate with the Routing Arbiter in developing better visualization methodologies and tools for the vast amounts of BGP data they collect.
   3. measurement tools non-invasive, low-cost metrics and data collection tools for more useful description of Internet workload, topologies, performance, instability. Several NLANR researchers are active members of the IETF's IPPM (Internet provider performance measurements) working group, chartered to develop stronger specifications of Internet data metrics and tools.
   4. Internet engineering curriculum further develop, refine, and respond to feedback on the IEC prototype at http://iec.nlanr.net/ (separate proposal pending with NCRI)
   5. Network collaborative environment development (2 students, UCSD):
      • NLANRmoo (telnet://www.nlar.net:8267), for NLANR researchers to get and share help with NLANR applications and related issues
      • IPNmoo (http://www.nlanr.net/Ipnmoo, an experimental prototype channel for ISPs to communicate information about network status in (nearly) real-time

By site, this breaks down into (units of FTEs):

1. Cornell: .5 tech/ops, 1 user services
2. PSC: 1.25 user services/mgt, .5 tech/ops, .25 PI
3. NCAR: .5 tech/ops, 1 user services
4. NCSA: .5 tech/ops, 1 user services , .1 PI
5. SDSC: .5 research coord., 1 PR/admin, .25 PI 2 FTEs stats/tools, 1.5 web pages (students), 2 student programmers, .5 caching outreach

• 1996 quarter 2 report (1 apr 95 - 30 nov 96)
  
• 1996 quarter 2 report (1 apr - 31 june 96)
  
• 1996 quarter 1 report (1 jan - 31 mar 96)
  
• 1995 annual report (1 may - 31 dec 95)
• NLANR/NCRI Meeting 7/96 on Project Directions

Contents

• Cache access control
• Example access control configuration
• Usage Patterns
• Network Configuration
• Code status
• Sanitized access logs made public
• Cache Registration service
• Daily access log summaries and statistics
• Problems Encountered
• Status reports and presentations
• Mailing lists
• VRML Visualization
• Getting caches at the NAPs
• sample statistics
• Issues and Future Work
• Research Questions

In line with the goal of using the information gathered from network analysis to improve the performance of the architecture, NLANR continues work on its information caching prototype, and has submitted a proposal to NSF for continued evolution of the system. A key result of Internet workload analysis has been the recognition of the existence of an increasing proportion of web traffic as measured at reachable locations, as well as simulation results that indicate that caching could leverage the high degree of redundancy in web document transmissions.

Participation in the NLANR caching project continues to grow. We have upgraded the machines from 128 to 256 Mbytes of RAM for improved performance. The caches originally used the original Harvest cache software ( http://harvest.cs.colorado.edu/ ), designed for performance and hierarchical scaling. Since May 1996 we have been using the public domain squid software ( http://www.nlanr.net/Squid/ ), a community effort led by Duane Wessels, NLANR technical lead for the caching project. (Peter Danzig, architect of the original Harvest cache, is providing commercialization support for the harvest cache.)

The international participation is still extraordinary, and dwarfs U.S. interest in caching in general. Following is a list of sites using the NLANR caches:

• Africa: iafrica.com
• Austria: univie.ac.at
• Australia: ausom.net.au, bns.com.au, connect.com.au, flex.com.au, geog.unsw.edu.au, mq.edu.au, senet.com.au, starwon.com.au unsw.edu.au, uts.edu.au, xavier.sa.edu.au
• Belgium: hades.sckcen.be, ucl.ac.be
• Canada: gmcc.ab.ca
• China: news.fudan.sh.cn
• Commercial: ange.nuri.net, compuserve.com, cray.com, gdscorp.com, hp.com, hunter.com, mot.com, netcom.com, nttlabs.com, pics.com, sony.com, wmsoft.com
• Czech Republic: cesnet.cz, feld.cvut.ca, liff.jcu.cz, vslib.cz,
• Denmark: netman.dk
• Germany: dkrz.de, fh-rpl.de mvv.de, netman.de, ruhr-uni-bochum.de, tu-chemnitz.de, uni-bonn.de, uni-frankfurt.de, uni-kiel.de, uni-kl.de
• U.S. Educational: binghamton.edu, colorado.edu, colostate.edu, cornell.edu, duke.edu, georgetown.edu, missouri.edu, msu.edu, psc.edu, uogi.edu, sdsc.edu, stanford.edu, sunysb.edu, syr.edu, trincoll.edu, usc.edu, utk.edu
• U.S. Government: lbl.gov
• India: ernet.in
• Israel: tau.ac.il
• Italy: forza-italia.it, lnf.infn.it unipd.it unive.it
• Japan: imasy.or.jp nttlabs.com
• Korea: kaist.ac.kr
• Networks: cic.net, cni.net, fugger.net, iphil.net, jwindow.net
• New Zealand: nzgate.net.nz,
• New Zealand: nzgate.net.nz, waikato.ac.nz
• Non-Profit: hall.org, loop.org, oceana.org, pacbell-sucks.org, park.org, seld.org
• Norway: uninett.no
• Philippines: admu.edu.ph i-manila.com.ph
• Poland: icm.edu.pl, pw.edu.pl, wsub.waw.pl
• Singapore: pacific.net.sg,
• South Africa: momentum.co.za
• (Former) Soviet Union: arcom.spb.su, sovam.com jet.msk.su
• Sweden: nada.kth.se
• United Kingdom: barclays.co.uk, demon.co.uk, gla.ac.uk, ic.ac.uk, ph.ed.ac.uk, mcc.ac.uk, tadpole.co.uk, tcp.co.uk

The sites shown in bold have established ``mutual parent'' relationships with us. That is, we agree to be their parent cache for URLs located in the U.S., and they are our parent for URLs located within their country.

We began in July to place restrictions on using the caches, to encourage a more sound hierarchical topology. accept connections from any address and will process any URL request. We announced this policy in May in response to visualization analysis of the neighbor database.

Cache access control includes the following objectives:

• Each peer cache will be allowed to use two NLANR caches, basically determined by geography/topology. By default European sites will use our East-coast caches, and Asia-Pacific sites will use our West-coast caches. U.S. sites should use whichever NLANR caches are appropriate
• The class A/B/C network of each peer cache is specified in the config file, to allow backup/replacement machines and some address renumbering with minimal config changes.
• We strongly encouraged new sites to join this hierarchy/mesh. We may first ask non-U.S. sites to peer with an existing cache in their country.

Note these are only guidelines, not hard-and-fast rules. We are very open to other arrangements which make sense.

The list below shows an example access configured for each site, generated by ranking all NLANR clients for the month of June by number of requests. Low ranking sites from countries with higher ranking caches have been dropped.

ebone-proxy.univie.ac.at        192.174.65.0/24      it pb
cardrout.card.ucl.ac.be         130.104.0.0/16       it pb
gateway.gmcc.ab.ca              198.161.33.0/24      bo it
news.fudan.sh.cn                202.120.224.0/24     sv sd
tinderbox.netman.dk             193.88.72.0/24       it pb
ababel.ecm.ub.es                161.116.0.0/16       it pb
iitk.ernet.in                   202.141.40.0/24      it pb
light.imasy.or.jp               202.227.24.0/24      sv sd
cosmos.kaist.ac.kr              143.248.0.0/16       sv sd
isis.waikato.ac.nz              140.200.0.0/16       sv sd
sunsite.icm.edu.pl              148.81.0.0/16        it pb
sunsite.nada.kth.se             130.237.0.0/16       it pb
pacific.net.sg                  203.120.90.0/24      sv sd
zaphod.arcom.spb.su             194.135.104.0/24     bo it
alpha.icmp.lviv.ua              193.124.228.0/24     bo it
norse.mcc.ac.uk                 130.88.0.0/16        it pb
phoenix.doc.ic.ac.uk            193.63.255.0/24      it pb
sirius.tadpole.co.uk            192.52.161.0/24      it pb
connect.com.au                  192.189.54.0/24      sv sd
manifera.itd.uts.edu.au         138.25.0.0/16        sv sd
iliad.lib.mq.edu.au             137.111.0.0/16       sv sd
netra.geko.net.au               203.2.239.0/24       sv sd
holly.flex.com.au               203.19.214.0/24      sv sd
starwon.com.au                  203.15.243.0/24      sv sd
geog.unsw.edu.au                149.171.0.0/16       sv sd
topdown.bns.com.au              203.19.43.0/24       sv sd
bbs.ausom.net.au                203.8.161.0/24       sv sd
vweb.access.net.au              203.13.25.0/24       sv sd
eyre.iinet.net.au               203.19.149.0/24      sv sd
alpha.xavier.sa.edu.au          203.17.192.0/24      sv sd
spock.rz.uni-frankfurt.de       141.2.0.0/16         it pb
rz.ruhr-uni-bochum.de           134.147.0.0/16       it pb
uran.informatik.uni-bonn.de     131.220.0.0/16       it pb
doener.unix-ag.uni-kl.de        131.246.0.0/16       it pb
aiken.zw.klautern.fh-rpl.de     143.93.0.0/16        it pb
proxy.germany.eu.net            194.175.173.0/24     it pb
www-proxy.iafrica.com           196.7.0.0/24         sv sd
umomware.momentum.co.za         196.13.225.0/24      sv sd
groa.uct.ac.za                  137.158.128.0/24     sv sd
iphil-communications.ph         204.70.35.0/24       sv sd
crispin.i-manila.com.ph         203.172.14.0/24      sv sd
vultur.vslib.cz                 147.230.0.0/16       it pb
adis.cesnet.cz                  194.50.6.0/24        it pb
digi.feld.cvut.cz               147.32.0.0/16        it pb
ax0rm1.roma1.infn.it            141.108.0.0/16       it pb
server1.infn.it                 131.154.0.0/16       it pb
www.forza-italia.it             194.166.23.0/24      it pb
mail1.fw-bc.sony.com            198.83.177.0/24      it sv
telford.nsa.hp.com              15.0.0.0/8           sv sd
gatekeeper.hunter.com           199.217.148.0/24     uc
thesparc.wmsoft.com             194.64.125.0/24      it uc
ns.nttlabs.com                  204.162.36.0/24      sv
zmark.cts.com                   204.94.75.0/24       sd
ucar.edu                        128.117.0.0/16       bo
colorado.edu                    128.138.0.0/16       bo
msu-buzz.edu                    206.26.77.0/24       bo
syr.edu                         128.230.0.0/16       it sv
cornell.edu                     128.84.0.0/16        it
trincoll.edu                    157.252.0.0/16       bo
bxscience.edu                   204.217.8.0/24       uc bo
colostate.edu                   129.82.0.0/16        bo
missouri.edu                    128.206.0.0/16       it sv
psc.edu                         147.72.0.0/16        pb
sunysb.edu                      129.49.0.0/16        it
diogenes.sedl.org               198.213.9.0/24       sv
valinor.oceana.org              205.233.219.0/24     pb uc
janus.sedl.org                  198.213.9.0/24       uc bo
ren.ee.lbl.gov                  131.243.0.0/16       sv
ange.nuri.net                   203.255.112.0/24     sv
zuse.space.net                  194.59.182.0/24      it uc
isca10.fugger.net               194.77.44.0/24       uc
tm3.cni.net                     194.115.51.0/24      pb
binabik.gatewest.net            198.163.227.0/24     uc
musclewood.hq.cic.net           198.108.58.0/24      uc
shell.wisp.net                  198.67.33.0/24       it

Usage Patterns

A busy cache will typically handle 150,000 requests and serve 3 Gbytes of Web objects. Cache hit rates are usually in the range of 5-10% percent and a couple of percentage points higher when weighted by byte volume.

By far, the largest number of requests are in the .com domain. Often 60% of all requests are for .com URLs. Another 20% are for .edu, .net, and .org. We calculate that the caches are currently only being used to about 15% of their capacity. The cache software should be able to handle one million requests and 16 Gbytes per day. However, at this rate, we would be serving twice as much data per day than we could store on disk; additional disk space would be desirable.

We have also been exploring metrics to indicate how well client caches are using parent caches. One measure is relative number of ICP requests vs HTTP requests (queries vs fetches). We probably do not want to see only one HTTP fetch for every 10 ICP queries. However, for sibling caches we could expect a pretty low HTTP:ICP ratio because a cache only fetches from siblings on a HIT.

So another measurement is hit rate. If a cache gets good hit rates from us, then maybe the HTTP:ICP ratio doesn't matter so much.

So what we are tentatively calling "cache utilization" is

	     HTTP_COUNTS      HTTP_HITS
utilization = ------------- + --------------
	      ICP_COUNTS     HTTP_COUNTS

Here's calculated utilization values for 'sv' on 7 May 1996:

		HIT    H/I
REVERSE FQDN            RATE   RATIO  UTIL
----------------------- ----   ----   ----
edu.georgetown.webcache 0.56 + 1.00 = 1.56
gov.lbl.ee.ren          0.24 + 0.99 = 1.23
org.sedl.diogenes       1.00 + 0.21 = 1.21
edu.syr.rocket          1.00 + 0.17 = 1.17
[206.106.252.226]       1.00 + 0.14 = 1.14
br.com.gns.srv1-bsb     0.33 + 0.80 = 1.13
net.nuri.ange           0.43 + 0.63 = 1.06
[193.175.197.97]        1.00 + 0.04 = 1.04
uk.ac.ic.doc.phoenix    1.00 + 0.04 = 1.04
IT                      0.07 + 0.94 = 1.01
com.hp.access.paloalto  0.20 + 0.80 = 1.00
BO                      0.15 + 0.85 = 1.00
com.sony.fw-sj.user1    0.00 + 1.00 = 1.00
au.com.flex.holly       0.37 + 0.62 = 1.00
SD                      0.08 + 0.91 = 0.99
com.nttlabs.ns          0.03 + 0.95 = 0.98
com.vivid.greedo        0.14 + 0.80 = 0.94
PB                      0.07 + 0.87 = 0.94
dk.netman.tinderbox     0.00 + 0.94 = 0.94
jp.or.imasy.light       0.27 + 0.66 = 0.93
net.mci.sanfrancisco.ip 0.16 + 0.72 = 0.88
nz.ac.waikato.isis      0.26 + 0.62 = 0.87
au.edu.uts.itd.manifera 0.22 + 0.64 = 0.85
cz.vslib.vultur         0.78 + 0.07 = 0.85
net.nlanr.oceana        0.00 + 0.80 = 0.80
jp.or.imasy.luck.yebisu 0.50 + 0.26 = 0.76
au.edu.unsw.geog.electr 0.17 + 0.55 = 0.72
localhost               0.01 + 0.69 = 0.70
edu.ucar.scd.surf       0.20 + 0.50 = 0.70
au.net.access.power4    0.64 + 0.06 = 0.70
au.com.connect.myangup  0.17 + 0.52 = 0.69
it.infn.lnf.pccarboni   0.40 + 0.28 = 0.68
uk.ac.mcc.norse         0.64 + 0.04 = 0.68
de.tu-chemnitz.hrz.hasi 0.60 + 0.03 = 0.63
au.net.geko.netra       0.08 + 0.48 = 0.57
de.uni-frankfurt.rz.spo 0.30 + 0.25 = 0.55
uk.co.tadpole.sirius    0.50 + 0.02 = 0.52
au.edu.mq.lib.iliad     0.18 + 0.34 = 0.52
au.net.access.vweb      0.33 + 0.11 = 0.45
de.uni-bonn.informatik. 0.19 + 0.10 = 0.29
de.uni-kl.unix-ag.doene 0.00 + 0.00 = 0.00
org.sedl.janus          0.00 + 0.00 = 0.00
br.com.gns.pixel-bsb    0.00 + 0.00 = 0.00
de.dkrz.www             0.00 + 0.00 = 0.00

We expect that around 0.80 utilization is a reasonable threshold; we may want to ask any child cache falling below that point to use a different (or fewer) parent cache(s).

A full collection of statistics can be found at http://www.nlanr.net/Cache/Statistics/ .

Network Configuration

Because some of the cache machines are located at supercomputer center sites, they are able to communicate with each other over the vBNS. This is advantegeous because it provides a high-speed cache-to-cache communication channel. It means that in most cases, there is very little penalty to retrieve an object through another NLANR cache. The FIX-West cache is not on the vBNS, but we have established a collaboration with DREN which enables us to leverage the DREN connectivity between NOSC in San Diego and FIX-West.

Because the supercomputer centers are not optimal locations within the Internet topology, we are still trying to locate caches at top-level interconnection points (such as the FIXes and NAPs). Because the supercomputer sites are one or two ISPs ``underneath'' from the backbone providers, U.S. users may find little benefit in using an NLANR cache. The FIX-West cache is actually in a very good location and this makes it more popular than the others, and we continue trying to convince NAP service providers to participate in this project.

Code status

The current version of the caching code, squid has many powerful features: Support for SSL, graceful shutdown without loss of current cache obejects, store.log that logs the actions of the storage manager and swap status/ttl of a requested object. private objects, proper parsing of HTTP reply codes, support for If-Modified-Since GET, improvements to the access log, metadata reloads in the background, flexible access control scheme, using SIGHUP to reconfigure the cache, an ftpget server to avoid forking, and assigning weights to cache neighbors.

We also purchased and installed an NLANR copy of Purify software for use with Squid code development, which has been a great help (no advertisting intended, the software speaks for itself.)

Sanitized access logs made public

We make sanitized versions of our daily cache access log files available for anyone to use. Some organizations are using these logs to analyze trends in WWW access patterns. At least one is using them to determine which Web sites are the most popular. The log files are available via anonymous ftp from ftp://oceana.nlanr.net/Traces/Caching/.

Cache Registration service

The Harvest project included the idea of providing a registration service for caches. This would allow cache users to locate one another and set up local cache hierarchies. The initial registration service was via email, which proved to be problematic.

NLANR now operates a cache registration service based on UDP announcement messages. If configured, a cache will send an announcement at regular intervals (i.e. once per day) or every time the cache daemon starts. The announcement includes hostname and ports, software version, contact information, geographical location (latitude/longitude), and any additional comments the administrator wishes to add. The registration database can be viewed at http://www.nlanr.net/Cache/Tracker/ .

Daily access log summaries and statistics

We provide daily summary reports of cache activity. These reports include total traffic through the caches, hit-rate percent for number of requests and by traffic volume, and a breakdown of the request result types (hit, miss, error, etc). The reports also give the top 20 server sites being accessed, the top clients, the top URLs, and a list of object types (loosely determined from the URL). These are available at http://www.nlanr.net/Cache/Statistics/ .

Problems Encountered

Unfortunately we had a streak of bad luck with Digital's Unix operating system. Our moderately busy, FDDI-attached machines exercised a few different bugs which caused the machines to hang very often (i.e. daily). First, a problem with the FDDI driver would hang a machine if it received too many illegal packets coming from a Cisco AGS+ router. Second, every received (small) packet would be placed into a separate `mbuf' of 8192 bytes. After some time, megabytes of RAM would be consumed by the network. Third, a bug in the TCP code would cause the maximum window size to become huge. This caused megabytes of data to become stored in a socket receive queue and led to a shortage of mbufs described above.

After two months of dealing with Digital technical support we were finally put in touch with kernel developers who provided us with fixes for these problems. Since then, the machines have been operating without any unexpected hangs.

Status reports and presentations

We have written status reports and given a number presentations on this project. Claffy has given presentations at NANOG, IETF the Federal Networking Council (FNC), and numerous ISPs. Wessels has presented the project at NCAR and SDSC. Wessels was an invited speaker to a workshop on caching in Warsaw, Poland. He gave about four talks during the workshop.

Mailing lists

NLANR operates a number of mailing lists related to this project:

• ircache
• ircache-dev
• squid-users
• squid-bugs
• cache-coders

All available by requesting to info@nlanr.net.

VRML Visualization

Claffy, Tamara Munzner (Stanford), and Eric Hoffman (Ipsilon) have produced 3-D VRML representations of the NLANR cache hierarchy. Using a database which translates IP addresses into geographical lat/long coordinates, they are able to generate a global view of which sites are using the NLANR root caches. These visualizations are available at http://www.nlanr.net/Cache/cacheviz.html , with a view updated daily at http://www.nlanr.net/Cache/daily.html .

Getting caches at the NAPs

Claffy has been working with the NAPs, encouraging them to install caches at their sites. So far, two have placed orders for machines. We are excited about getting caches at the NAPs since they will be much more logical locations for many U.S. users.

---

sample statistics

We make current usage statistics publicly available at http://www.nlanr.net/Cache/Statistics in several formats

Amount of data served by the NLANR caches (per day)
since project beginning (27 November 1995)

---

We also provide graphs for each cache separately, at http://www.nlanr.net/Cache/Statistics/Graphs/, and reports and summaries at http://www.nlanr.net/Cache/Statistics/Reports/ and http://www.nlanr.net/Cache/Statistics/Summaries/, respectively.

Each of the cache machines is a high-end Digital Unix server with 10G of disk, 256M of RAM, and FDDI interfaces. The cache software keeps object metadata and hot objects in virtual memory. We found that with 128M of RAM we must limit our cache size to around 200,000 objects or 3.3G bytes. When the cache process grows too large, the system has swap the process to disk, which degrades overall performance. We increased the memory size to avoid this behavior as much as possible. Still, we're working those puppies pretty hard.

As the above graph indicates, a cache will often serve 2G bytes of data per day, more than half of its entire cache size. We typically observe cache hit rates ranging from 10-20%, not so surprising since

• almost all of our clients are other caches which handle the majority of cache hits.
• our cache size is too small for the load it is handling. We need to increase our cache size which will require more physical memory.

Issues and Future Work

Ongoing activities will be in areas to:

1. encourage greater U.S. participation. we have started to hear that the NAPs are finally installing cache hardware and we will be welcoming them to the NLANR hierarchy
2. standardize ICP protocol used for caches to communicate with one another, and develop more sophisticated automatic neighbor selection protocols and algorithms. Configuration of cache parents and neighbors is still quite manual, and only incorporates delay data rather than available bandwidth data in neighbor selection.
3. continue development of cache software (http://squid.nlanr.net/. Items for future releases include support for HTTP/1.1 protocol features, generic redirection to for access controls and automatic mirroring, If-modified-since requests between caches, reviving the Harvest encapsulation protocol for multiplexing cache-cache traffic over a single persistent TCP connection.
4. workshop participation and hosting. NLANR will participate in international as well as domestic workshops in the caching community.
5. investigate further the benefit of mutual-parenting relationships. Intuitively, this makes sense so long as the parent caches are located at ``high spots'' in the network (for example near each end of an international link), but we need empirical data to support the hypothesis.
6. develop a scalable cache discovery system: will likely involve using the DNS domain ircache.net for top-level domain cache location.
7. pushing objects into caches, i.e., server push
8. multicast: collaborate with multicast researchers (at LBL, UCLA) to determine how to best integrate multicast technology into the fielded hierarchy
9. benchmark hardware/software platform performance as high-end caches
10. explore how to accommodate the information provider need to obtain cache-hit feedback on their documents. Obviously providers want to collect demographics on visitors to their sites. From the user-perspective, some organizations prefer the fact that proxies and caches hide some information about the end users. Is there a middle ground? Should the protocol specify exactly what sort of demographics must be available to the information providers?

Research Questions

Other research questions that require attention include:

1. What obstacles will be encountered as we scale up the caches? We find that we are currently limited by the amount of physical memory and seek time of hard disks. Will arrays or clusters of caches be necessary and/or viable?
2. How do we enable providers to retain accurate hit count data without writing cache-unfriendly pages? We fear that a number of information providers make an effort to thwart caching so they can accurately log every access to their site. We would much rather develop a general-purpose access-reporting mechanism to satisfy both requirements: serving objects from a cache to save bandwidth and improve performance, and still provide a mechanism for information providers to obtain hit count data for their objects.
3. Do cache access traces represent an accurate slice of WWW traffic? A few people are using our sanitized log files for simulations and as a way to measure which web sites are popular. Because our caches are at the top levels of a hierarchy, their access logs do not likely represent end user browsing patterns.

---

NLANR quarterly meeting (Tuesday 27 june 1996, IETF, Montreal)

The following meeting summary is included as it outlines NLANR strategies and future emphases.

Goal of meeting: focus NLANR on key areas where we can make an impact.

1. What is NSF getting for its investment in the vBNS
   1. Get information back to NSF on what is going on with the vBNS
   2. Produce quarterly reports modeled on SCC quarterly reports
   3. Demonstrate overall plan, tangible results (milestones) each quarter
   4. Include plans/goals for next quarter in each quarterly report
2. Increase application usage
   1. Better vBNS PR to SCC users via newsletter articles and training
   2. Solicit high profile applications (i.e. GC3, IWAY)
   3. Prepare better documentation for new users of the vBNS
3. Prepare vBNS for the influx of new connections
   1. Work with sites on planning & overall network architecture
   2. Attempt to implement technical solutions for congestion
   3. Help make sense of conflicting requests to MCI
   4. Provide ongoing support for specialized routing needs. Include in this addressing fundamental tensions between IP users and ATM users.
4. Promote research into new networking technologies on the vBNS
5. Specific operational needs
   1. Strong need for network monitoring and real data on reliability and utilization.
   2. Generate a technical plan to implement something by 7/15
   3. Review current routing architecture and discuss any desired changes.
6. Design/Architecture for new sites
7. Applications Working Group
   1. Define goals for an applications working group
   2. Discuss strategies for increasing usage by current apps and bringing on new apps
   3. Select a core group to implement strategies as rapidly as possible.
8. Networking Research Working Groups
   1. Brainstorm possible areas of interest
   2. Select networking research areas which will have maximum impact
   3. Select a core group (or groups) to forge ahead in these areas