National Laboratory for Applied Network Research

NCSA is working with Paul Woodward, from the Laboratory for Computational Science and Engineering (LCSE) at the University of Minnesota to demonstrate similar capabilities over the vBNS, using facilities at LCSE as well as Boston University as soon as the Minnesota and BU high performance connections are in place (both awarded High Performance Connections grants). As of October 1996 NCSA demonstrated the ability to seamlessly queue jobs between SGI environments at BU and NCSA using the LSF scheduling software from Platform Computing. They have been working with SGI to address limitations of their high performance MPI library, specifically those related to the very large messages needed in high speed applications (e.g, 40 Mbyte messages for the LCSE PPM code). Using innovative methods to overlap communication and computation, Woodward's team is achieving tolerable delays (in the range of 100ms), rendering this application ideally suited to the vBNS.

Remote Instrument Control using vBNS

Application scientists at NCSA and the University of Illinois Beckman Institute used the vBNS to remotely run EVAC (Virtual Environment for Control of Remote Imaging Instrumentation) between the Beckman Institute and the Scripps Research Institute in San Diego. The demonstration, which was for the ``Molecular Microscopy in the 90's''; workshop at Scripps, required more bandwidth than available via the commodity Internet. NCSA networking staff coordinated routing among NCSA, the University of Illinois, MCI vBNS engineering, SDSC, and Scripps.

Remote Volumetric Visualization

Computational scientists from the research group of Paul C. Lauterbur of the Department of Chemistry at UIUC, NCSA, Cornell's chemistry departement, and the Cornell Theory Center (CTC) have used the vBNS for computation-intensive research in magnetic resonance imaging (MRI). Diffusional Enhancement of Signal Intensity and REsolution, or DESIRE, is a new paradigm for MRI at resolutions far below ten microns, the current state of the art. Numerical simulations governing partial differential equations of motion have a crucial role in the design of experiments and the determination of fundamental, physical limitations. Simulations are running on the 512-node IBM RS/6000 SP2 at the CTC by way of a 28,000 SU allocation awarded to Paul Lauterbur. A typical, single, simulation run generates 2 gigabytes of data on CTC storage devices which must be processed and visualized by scientists at UIUC; up to 100 simulation runs are planned for the current SU allocation. The experience of the research groups has made clear that remote access to data generated at the CTC and collaborative sharing of results would not be feasible without the vBNS.

Paul Lauterbur has presented results of simulations to date at the XVIIth meeting of the International Conference on Magnetic Resonance in Biological Systems, Keystone, Colorado, August 18-23, 1996.

International High Performance Distributed Applications

NCSA worked with NSF NCRI's international connections program and the White House G7 program to solicit and process proposals for collaborative applications that could utilize high performance international connectivity. Well over 50 applications, from researchers from all parts of the US, Europe, and Asia. were submitted and are now documented. NCSA produced a hard copy document detailing the applications, distributing it at the Paris GIBN meeting in February. This document later became the standard that the other GIBN partners used as a model to summarize additional GIBN applications. This has the potential to greatly increase the number of distributed applications that can utilize the vBNS. NCSA is already targeting a major application for use over the high speed link between the United States and Europe, building on NCSA work in homogeneous wide area metacomputers (see above section on Latency Tolerant High Performance Distributed Applications): distribution of a job between NCSA's SGI Power Challenge Array and the Max Plank Institute at Potsdam Germany.

Application Performance Tuning Primer

As a result of working with several dozen application teams during FY95 and early FY96 in the context of the I-WAY project, Von Welch developed a primer for using TCP window scaling. This user guide is one of several NLANR resources available to users in order to help them achieve high performance over the vBNS.

Performance Tools, Measurement and Tuning

Tuned ftp Tools for vBNS

NCSA ported public-domain ftp tools to their SGI Power Challenge array and modified them to be able to use large TCP window sizes, allowing users to take full advantage of the vBNS high-bandwidth, resulting in ftp throughput between NCSA and PSC increasing by nearly an order of magnitude from 2 megabytes/second to 13 megabytes/second. ( ftp performance details here)

Internet End-to-end Performance: The User View

In response to many user complaints about Internet performance Von Welch and Charlie Catlett initiated a long term quantitative study of Internet performance between NCSA and a number of key remote user sites. The testing, which spans from March of 1996 to the present time, involve throughput, latency, and packet loss measurement. Treno, from PSC, was also used to attempt to determine bottlenecks.

The tests have already proven useful in documenting the actual performance impact on real users of several changes in networking infrastructure. Two sites (OSC and UCLA) upgraded from multiple T1s to T3s during the course of the study. Comparing data before and after these upgrades reveals the impact on end users, jumping by nearly an order of magnitude in both cases.

The study also documented the performance degradation caused by a problem in a network link and the resulting improvement when it was corrected. The study documented the exact day and time when performance to U. of Arizona fell sharply. This lent networking personnel there definitive proof that a serious problem existed. After much debugging, they determined the problem was in Arizona's link to their ISP.

vBNS Performance Testing: U. of Illinois/Scripps

Although performance seen between the University of Illinois and Scripps Lab for the aforementioned EVAC demo was sufficient for the demo, it was much worse than expected. NCSA staff ran performance tests between different end points on the vBNS in an attempt to isolate the problem. The testing indicated that the performance bottleneck was not on the vBNS, but instead exposed a bottleneck on Scripps' internal network. Staff at Scripps believe they understand the (firewall-related) problem and are deploying a solution. The performance tests also serve to document vBNS performance so that we have a baseline for future investigations.

NLANR Monitor/CICNet Traffic Monitoring

As a result of the highly-successful I-WAY demonstration at SC95, NCSA UIUC-CS dept, and ANL/EVL submitted a proposal to ARPA to fund a persistent version of the I-WAY distributed applications testbed. One component of this proposal was to develop real-time network performance tools, leveraging MCI's OC3MON, SDSC's flows research, and Dan Reed's Pablo project at UIUC/NCSA. Pablo was demonstrated during the I-WAY project, and has also been used to examine real-time performance data using virtual environments to allow scientists to interact with system performance information flows such as from networks, parallel applications, or information servers.

The goal of the networking performance tools is to facilitate a non-specialist in networking to analyze and tune application performance. Currently end users access to real information about the network behavior of a given applications is limited to what they can get from networking specialists, and typically via the use of insecure promiscuous traffic sniffers that create privacy and security concerns.

The developed network performance assessment system will have authentication, authorization and traffic filtering capabilities so that users can access data about their network application in a secure manner. The group will also develop an API to allow the system to be integrated with existing networking libraries, e.g., MPI.

Since writing the proposal, NCSA has also joined in a CICNet-wide effort to gather IP flow statistics in pursuit of a better understanding of local traffic flow characteristics. An PC with a FDDI interface will serve as a monitoring platform for NCSA's Internet DMZ. A version that can monitor an FDDI network with the Flow software from SDSC has been ported and is ready for field testing. The remaining PC and ATM hardware is on order and should arrive by January 1997, at which point NCSA can port an ATM version of the software.

vBNS RSVP Testing

NCSA is working with MCI and Cornell to test and deploy early implementations of the RSVP reservation protocol from both Cisco and SGI over the vBNS. NLANR will demo RSVP over the vBNS at Supercomputing '96, comparing performance of an application competing with a `network-hog' application both with and without RSVP support.

Programmatic Support and Promoting vBNS External Connections

Support for High Performance Connections Program

During the past year, NCSA staff have worked with a number of sites to help them understand the operational and technical aspects to engineering and proposing a high performance network connection. This involved assisting them in selecting promising driving applications, helping them understand the technologies involved in connecting LANs with high performance WANs, and giving them guidance on the economics of high performance telecommunications offerings.

G7 Global Interoperability of Broadband Networks (GIBN)

NCSA has been working closely with NSF NCRI's international connections program to provide technical advice regarding the White House G7 initiatives such as Global Interoperability of Broadband Networks (GIBN) project. Details about the GIBN project applications are at http://www.ncsa.uiuc.edu/General/GIBN/. Charlie Catlett and Randy Butler have been actively involved as USA technical advisors to the project, focusing on interconnection of GIBN partners. One plan is to build an interconnection site in the USA that would host links to Japan, Canada (and from there to Europe) as well as to the major USA research networks led by the vBNS. While the proposed interconnection plan did not meet with approval it did bring together the carriers (AT&T, Sprint and MCI) and networking leaders at NSF and DOE to discuss interconnection issues and strategies.

A direct result of the GIBN activity is the interconnection between CANARIE and the vBNS via the Chicago NAP, and concomitant plans to for a CANARIE-vBNS application demo in the NLANR booth at SC96. During FY97 we expect to use this interconnection to demonstrate wide area homogeneous metacomputing with the Max Planck Institute using Silicon Graphics Origin-2000 supercomputers. We will also work with the European Community Advanced Communications Technologies and Services (ACTS) program to use the interconnection of the vBNS, CANARIE, and the European ATM testbed infrastructure to demonstrate the use of shared virtual environments for supporting collaborative engineering in the context of vehicle design and manufacturing.

CEWES applications

NCSA worked with Interim-DREN staff and NSF to drive the peering of the vBNS and iDREN as required to support environmental applications needing coupled virtual environments between the US Army Corps of Engineers Waterways Experiments Station (CE-WES) and NCSA. NCSA is assisting CEWES in generating an Immersadesk application for data collected at Chesapeake Bay. The Chesapeake Bay Immersadesk application models salinity, chlorophyll, and dissolved oxygen content, in addition to inflow and outflow from the many rivers that enter the bay and the Atlantic tidal flow. I-Desk to I-Desk collaboration currently allows a researcher at one location to follow a researcher at another location while s/he navigates through the data space. Allowing each collaborator to watch everything his/her counterpart is doing in the Bay (data space) requires a high-speed, reliable connection. This application will serve as a stepping stone to more in depth collaborative applications. CEWES is planning to use this application at Supercomputing '96 in the Department of Defense exhibit. The I-Desk at NCSA will also be at SC '96, and some interaction between the two exhibits will take place.

Interop Panel on NLANR Activities

Charlie Catlett gave an overview of NLANR activities on a panel at Interop'96 in April along with George Strawn (NSF) and Charles Lee (MCI). Strawn's presentation gave an NSF policy perspective, Lee's an MCI operational view, and Catlett's a research and applications overview.

Supporting vBNS Connectivity to NCSA Computational and Information Resources

Routing Deployment

During the past year NCSA made significant advances in integrating the vBNS with its internal network. A route server was designed, tested and installed in operation to do BGP peering with the vBNS allowing for automated exchange of routing information. The system was successfully tested on June 12 when NCSA lost their Internet connectivity due to a failure of a MCI router at Downers Grove. In response to the crisis, NCSA and MCI vBNS Engineering staff cut over NCSA's Internet connectivity to the vBNS for the day.

NCSA LAN ATM Migration

NCSA developed and implemented a plan to convert thier internal FDDI backbone/shared Ethernet LAN to an ATM based LAN with switched Ethernets. This deployment, planned for completion by the end of 1996, will not only offer users higher bandwidth but also the means to runs application directly over ATM from NCSA machines across the vBNS. Direct high speed coupling of users to the vBNS at high speed will hopefully encourage the development of applications.

National Center for Supercomputing Applications FY97 Plans

Pittsburgh Supercomputing Center (PSC)

In addition to general descriptions of our work in these focus areas over the lifetime of the project, a detailed 4th quarter report is included to cover work not reported on elsewhere.

Operational vBNS support

Over the course of the project we have provided support to numerous vBNS applications, documented in quarterly reports and elsewhere online. In addition, we have supported several demonstrations of vBNS capabilities at conferences and meetings.

Organizational Support of vBNS and NLANR activities

The PSC played a leadership role in planning for an NLANR booth at the Supercomputing'96 conference. At this conference, we hope to demonstrate both the research aspects of the NLANR project as well as the capabilities of the next generation of high performance networking technology. Some of the planned demonstrations include Resource Reservation capabilities (RSVP), next generation Internet protocol (IPv6), OC3 Monitoring capabilities (OC3-mon), TCP Selective Acknowledgments (TCP SACK), and WWW caching. In addition, several applications which demonstrate the capabilities of high performance networks are planned.

Network Performance Research

In addition to work on TCP protocol enhancements, we have been studying theoretical models for TCP performance, and relating them to Internet measurement efforts. PSC has teamed up with Vern Paxson at LBNL in a proposal to research scalable techniques for measurement of Internet performance. This proposal was submitted to NSF earlier this year and is working its way through the authorization process.

Education and Demonstration of New Technologies

The capabilities for collaboration using high performance networking technologies are not well understood outside of the small group of researchers who work with these technologies on a day-to-day basis. In order to demonstrate the possibilities for use of the network in educational and collaborative environments, the PSC took advantage of an MBone talk presented by Chris Fair at NCAR in June, 1996. This talk was displayed live in two large lecture rooms at PSC utilizing vic, vat, and netscape. Roughly 75 people including staff at PSC, members of the CMU community, and about 20 PSC users who were on site for training were present for the seminar.

PSC Report on Detailed 4th Quarter Activities

vBNS Support Activities

In addition to these hardware activities, our engineering staff configured BGP peering with the vBNS and performed general routing maintenance throughout the quarter. We also performed various monitoring and troubleshooting activities throughout the quarter to ensure that PSC users were getting the best possible use from our vBNS connectivity.

vBNS/NLANR Organizational activities

Mr. Mahdavi spearheaded the effort to include an NLANR booth at the SC96 conference. The NLANR booth will showcase both the vBNS activities and networking research activities of recent months. Some examples of technology demos planned for the NLANR booth include IPv6, SACK TCP, use of the OC-12 vBNS Testnet, and the MCI OC3 monitors.

Mr. Mahdavi attended the High Performance Campus Networking (HPCN) meeting in Boulder, Colorado.

Networking Research Activities

Applications

Heterogeneous MetaCluster
http://www.nlanr.net/VBNS/Alloc/metacluster.html

Pittsburgh Supercomputer Center FY97 Plans

SDSC

SDSC NLANR published papers (http://www.nlanr.net/Papers/)

Coordination and Administration of the Internet, Harvard JFK School workshop, August 1996:

1. cooperation in Internet data acquisition and analysis (with Tracie Monk) describes the proposed Caida (http://www.nlanr.net/Caida/)
2. evolution of the nlanr cache hierarchy (with Duane Wessels) describes experiences with the NLANR caching hierarchy
3. address administration in ipv6 (with Eric Hoffman) proposes a geographically based metro addressing model
4. in whose domain: name service in adolescence (with Don Mitchell and Scott Bradner), commentary on the undue community attention to the Top Level Domain issue while other more important, but conceptually more complex, issues remain ignored.

IEEE 1996 IEEE Symposium on Information Visualization, San Francisco, 30 October 1996

• Visualizing the Global Topology of the MBone

Usenix '96 LISA, Chicago, 3 October 1996

• oc3mon: flexible, affordable, high performance statistics collection (with Joel Apisdorf, Rick Wilder and Kevin Thompson) describes the MCI OC3MON and provides sample OC3MON data from MCI bakcbone

• `but some data is worse than others': measurement of the global Internet

SDSC NLANR presentations/meetings (http://www.nlanr.net/Presentations/)

1. invited presentations to Federal Networking Council meetings, February and October. (K Claffy and Hans-Werner Braun)
2. met with NSF regarding next steps for NLANR in the context of recent developments (Internet II, next generation stuff, new connections), as well at the CAIDA efforts and their relationship to NLANR. (K Claffy)
3. IETF plenary session, June 1996 (Montreal), to present video shorts of various NLANR visualization projects. (K Claffy)
4. October NANOG meeting (Ann Arbor), focused on measurement tools and describing the OC3MON architecture and data examples. (K Claffy)
5. visited the Geometry Center in Minnesota to discuss using their very capable visualization tools and expertise on Internet data. (K Claffy)
6. presentations all over California (some Mboned) on the Caching project: NCAR, IETF (Montreal), FIX-west, Ipsilon, Digital-NSL, AOL, Xerox Parc, SDSC (Duane Wessels)
7. invited to give talks at special workshop on caching in Warsaw, Poland. (Duane Wessels)

SDSC NLANR software/web repositories released/supported (http://www.nlanr.net)

1. http://www.nlanr.net/Caidants CAIDA network tool set, used to support NLANR Internet visualzation projects
2. ftp://www.nlanr.net/Traces maintaining current online address-masked trace data (packet traces from FIX-west, cache logs from the NLANR caching system) (e.g,. FIX-west, NLANR caching system), for the research community to use in traffic studies.
3. ftp://www.nlanr.net/Software/Tcpdpriv software to hide raw network address from tcpdump file output, in order that it may be released to the research community safe from violating privacy considerations.
4. http://www.nlanr.net/INFO maintaining a respository of provider-supported Internet statistics and
5. (http://www.nlanr.net/Viz/Deunion) Internet data visualization projects, many of the latter being NLANR-supported

Cooperative Association for Internet Data Analysis (CAIDA)

Examples of statistics visualization that would be useful include:

1. Mbone tunnel topology to illustrate the multicast tunnel infrastrcuture that carries an increasing proportion of network traffic, but unfortunately often with suboptimal configuration.
2. workload flow profiles, including by application, source-destination matrices (by country, although other granularities are possible), and graphic illustrations of how packet, byte, and flow durations correlate for a given protocol. The flow analysis methodology and (gzipped, tarred) previous unix software and OC3MON platform software are publicly available. K Claffy, Joel Apisdorf, Kevin Thompson, and Rick Wilder of MCI/vBNS submitted to and presented a paper, oc3mon: `flexible, affordable, high performance statistics collection', on OC3MON at Usenix LISA '96. Future OC3MON plans include refining the statistics collected, support xterm-based remote control rather than only a PC, porting to unix, supporting AS matrix instead of classful network extraction from IP address, and integrating FPGA design tools.
3. Performance characteristics of a path from a fixed server in the Internet to a specified destination host. Jamshid Mahdavi and Matt Mathis have deployed the Treno server at PSC to perform this function via a form-based web interface, where the user may set parameters according to his performance interest.
4. Lower-cost performance metrics. One drawback of many commonly used performance tools is their somewhat aggressive use of the path to measure the path itself. We seek to minimize the burden imposed on the network, and NLANR seeks to foster the development of other tools and supporting models that will enable estimates of performance statistics with minimal imposition on the network. Inspired by Braun's extended low-impact measurement study (using a modified version of ping to timestamp packets at a microsecond level) and visualization, Claffy and Hoffman have been developing a tool to explore the quality of underlying unicast paths in such end-to-end measurements. Results of initial tests with this tool were the subject of one of the NLANR videos shown at the June IETF.
5. Visualization of caching infrastructure and a daily update of the caching hierarchy. Although these are rough VRML-based visualizations, we are currently developing tools similar to the Mbone tools above to enable greater manipulation of topological information.

Internet infrastructure information repository

Caching system prototype

In line with the goal of using the information gathered from network analysis to improve the performance of the architecture, NLANR continues work on its information caching prototype, and has submitted a proposal to NSF for continued evolution of the system. A key result of Internet workload analysis has been the recognition of the existence of an increasing proportion of web traffic as measured at reachable locations, as well as simulation results that indicate that caching could leverage the high degree of redundancy in web document transmissions.

Participation in the NLANR caching project continues to grow. We have upgraded the machines from 128 to 256 Mbytes of RAM for improved performance. The caches originally used the original Harvest cache software, designed for performance and hierarchical scaling. Since May 1996 we have been using the public domain squid software, a community effort led by Duane Wessels, NLANR technical PI for the caching project. Peter Danzig, architect of the original Harvest cache, is providing commercialization support for the Harvest cache.

international participation is still extraordinary, and dwarfs U.S. interest in caching in general. We began in July to place restrictions on using the caches, to encourage a more sound hierarchical topology. We announced this policy in May in response to visualization analysis of the neighbor database. An example configuration for access control at each site is generated by ranking all NLANR clients for the month of June by number of requests. Low ranking sites from countries with higher ranking caches have been dropped.

NLANR operates a number of mailing lists related to this project:
ircache, ircache-dev, squid-users, squid-bugs, cache-coders. Requests to info@nlanr.net.

Additional Information
Sample Statistics are available, as are Issues, Future Work, Open Research Questions, usage patterns, network configuration, VRML Visualization, Getting Caches into NAPs, as well as code status, published access logs, cache registration service, daily access summaries and statistics, and problems encountered.

Multicast infrastructure

NLANR continues to support multimedia conferencing and supporting multicast communications infrastructure to use and provide a development environment for such tools on the vBNS.

Claffy is also working on developing tools for Mbone visualization, as mentioned earlier, with Tamara Munzner at Stanford, Bill Fenner at Xerox, and Eric Hoffman at Ipsilon. Their paper, Visualizing the Global Topology of the MBONE was accepted to Information Visualization '96, a symposium associated with IEEE Visualization '96, to be held in San Francisco, Oct 27 - Nov 1 1996. The tool outlined in the work enables one to zoom in on logical subsets of the Mbone infrastructure such as the vBNS-related subset of tunnels, illustrated to the right.

Collaboration environments

NLANR still supports text-based collaborative environments, two in particualr: NLANR moo, an environment for NLANR researchers to chat, explore, and build within an online community, and IPNmoo, a collaboration with the Routing Arbiter's IPN project (established and led by Craig Labovitz) to provide an online collaborative environments for use in real-time ISP communication to debug network problems. Both moo's have already proven a useful resource for the IPN community. ( http://www.nlanr.net/Ipnmoo)

Toward development of an undergraduate Internet engineering curriculum

Claffy has hired some students to start building a prototype web repository of links to resources for an an Internet engineering curriculum ( http://iec.nlanr.net/). Intended as resource for instructors at all levels, as well as students, the repository will facilitate sharing of instructional resources (e.g., slides, problem sets, exams, outlines, notes).

SDSC/UCSD FY97 Plans

NLANR agenda for coming year

1. helping NCRI to provide a viable network environment to the R&E community, in order that scientists can take maximum advantage of community computational resources, many of which are also provided by the National Science Foundation (i.e., the supercomputers (vector and mpp), visualization/graphics engines, workstation clusters, data archives, mass storage, and AFS services of the NSF-sponsored supercomputer centers)
2. explore larger issues in evolution of the commodity Internet, including metrics and statistics of workload and performance, cooperation among ISPs, charging and settlement models.

NCRI recognizes that many of the networking as well as computational technologies will find their way outside of the NSF-spawned R&E environment, just as they did with the NSFNET program. NCRI continues to take care to optimize the interaction between the research infrastructure and the market-provided infrastructure, while pushing the technology envelope gracefully, such as with the recently announced NSFNET Connections Program. The neutrality of the constituents of NLANR also make it a good basis for infrastructural and interagency network activities, such as assisting NSF with this program, and continued collaboration among U.S. federal agencies. NLANR facilitates collaboration in conjunction with various other organizations, some of which are actively supporting NLANR activities.

For the remainder of the NLANR cooperative agreement, currently scheduled to end in October 1997, we plan to gear our activities towards supporting the larger NSF high performance connections communnity, leveraging off the experience of the initial vBNS sites over the last two years. Over this time, NLANR has served the vBNS networking community in at least four separate capacities:

1. helping users effectively run their applications on the vBNS
2. providing technical and operational support for vBNS equipment and configuration at each site
3. providing technical feedback relevant to vBNS policy issues
4. pursuing and coordinating forward-looking research and immediately apply to vBNS infrastructure

For the remainder of the duration of the cooperative agreement, as well as for follow on agreements we expect to jointly support, NLANR would like to serve NSF high performance connection awardees in the following capacities:

1. user support: 5.25 (user support, tracking, and training: 4.25 FTE; public relations: 1 FTE)
   Each user support person would be responsible for seeking out and supporting a moderate set (15/20) of vBNS applications. User support would include: providing user education through multicast conferencing as well as on-site workshops; providing written reports; and arranging for demonstrations at fora such as SC97. Since the NLANR user services group will be distributed across the five sites (as well as perhaps at new sites), a specific 0.25 FTE would be responsible for coordinating this distributed organization.
2. technical/operations support coordination: 2.5 FTE.
   PSC will coordinate technical discussions, i.e., planning general meetings of new sites, perhaps co-located with NANOG meetings.
3. technical feedback on policy .6 FTE
   to represent the research community beyond the SC centers. (i.e., PI responsibilities)
4. research cordination and supporting tool development 2.5 FTE

1. Cornell: .5 tech/ops, 1 user services
2. PSC: 1.25 user services/mgt, .5 tech/ops, .25 PI
3. NCAR: .5 tech/ops, 1 user services
4. NCSA: .5 tech/ops, 1 user services , .1 PI
5. SDSC: .5 research cord., 1 PR/admin, .25 PI 2 FTEs stats/tools, 1.5 web pages (students), .5 caching outreach

MCI FY97 Plans

Cornell Theory Center FY97 Plans

The following projects planned at CTC are funded through NLANR and other sources, and are all related to NLANR work:

CTC plans re-architecting of internal networks and connections to the SP2 using routed and switched ethernets and IP over ATM links, integrating a FORE 7000 PowerHub, FORE ASX-1000 ATM switch, ethernet switches, and cells-in-frames (CIF) attachment devices. Cornell will experiment with and evaluate several alternatives for virtual routers between Classical IP and Emulated LANs- including a workstation-based solution, FORE's beta code for the PowerHub, and an IBM 8210 Multiprotocol Switched Services server. Cornell may upgrade the Cisco 7000 routers which connect to NYSERNet to 7500 series routers and connect a second T3 to NYSERNet. A new NYSERNet T3 from Ithaca direct to the Pennsauken NAP will be installed by December 1996.

CTC will continue testing of Cisco's beta RSVP code and will focus on advanced resource reservations. Cornell will also work with other RSVP implementations under Linux and AIX. CTC plans to develop better mechanisms for scheduling bandwidth and to integrate network bandwidth reservation with of distributed computing.

CTC is planning on the installation of an IBM high speed gateway in spring 97, which will allow multi-gigabit connectivity to the High Performance Switch fabric which is internal to the SP2.

CTC is currently considering an increase in staffing to provide additional resources for a joint NLANR project with SDSC, which would include maintenance of network tools (such as OC3mon) and enhancement of user interfaces and visualization capabilities. SDSC and CTC may also work on expanding on SDSC's work on the routing arbiter.

National Center for Atmospheric Research FY97 Plans

1. NCAR NLANR personnel will assist MCI Engineering with the installation, testing, and operation of the vBNS performance monitor at NCAR.
2. NCAR NLANR personnel will assist MCI Engineering with the installation, testing, and operation of the OC-12 network circuit upgrades.
3. NCAR NLANR personnel will assist MCI Engineering with the installation, testing, and operation of Fore Systems ASX-1000 ATM switch hardware and software at NCAR.
4. NCAR NLANR personnel will assist Duane Wessels with the implementation of a Web caching hierarchy including evaluation of traffic patterns, bandwidth saved, and dollars saved at NCAR.
5. NCAR NLANR personnel will assist the NCAR Mesoscale and Microscale Meteorology Division with Internet Data Distribution project over the vBNS.
6. NCAR NLANR personnel will assist the NCAR Mesoscale and Microscale Meteorology Division with a distributed mesoscale modeling application over the vBNS.
7. NCAR NLANR personnel will assist Unidata with the reapplication and implementation of their IDD vBNS application.

National Center for Supercomputing Applications FY97 Plans (NCSA)

Enabling Applications on the OC-12 vBNS

In the upcoming year NCSA plans on continuing its aggressive work to enable more applications across the vBNS. This includes finding new strategic partnerships with other networks, finding and assisting end users in using the vBNS to full advantage, and further pursuit of latency-tolerant applications such as those already started with Boston U. and LCSE/Minnesota. All applications outlined above will continue to be supported and enhanced to take advantage of the OC-12 vBNS upgrade during FY97. NCSA is particularly interested in expanding our activities in assisting applications in using the vBNS during FY97.

Of particular interest during FY97 in the application area will be the support of coupled virtual environments (see CEWES application above), involving CAVEs as well as Immersadesks and Infinity-Walls. In addition, latency tolerant applications such as the LCSE/Minnesota application will be pursued as more high performance resources become available through the NSF high performance connections program.

vBNS Workshop and Support for Expanded vBNS Participation

NCSA will hold a training workshop for users on using the vBNS during the first quarter of 1997. The workshop will cover specifics of the vBNS as well as general coverage of high performance networking technologies and issues such as ATM, RSVP, MPI and achieving high performance over high-speed, high-latency networks.

NCSA worked with many of the new high performance connection sites in their proposal preparation process to help them understand the technical and operational considerations to engineering a high performance connection. Following on this work, NCSA also intends to work with NSF to determine how to promote technical coordination and cooperation among existing vBNS sites and these new sites.

Operational Exchanges with other High Performance Network Operators

Through the GIBN activities outlined earlier, NCSA has established technical ties with the CANARIE network operations teams. A technical exchange between NLANR and CANARIE will take place during SC'96 in November 1996. Work with Interim DREN operators in FY96 was instrumental in facilitating the peering of vBNS and iDREN. Similarly, through collaboration with Argonne National Laboratory NCSA has been establishing technical ties with the ESNET community. NCSA intends to continue to forge technical exchanges with CANARIE, DREN, ESNET, and other high performance network infrastructure groups during FY97.

Organization of Working Groups

In parallel to this working group, NCSA has agreed to organize NLANR efforts to monitor the performance of applications running currently over the vBNS. This effort is concentrated mainly on the large number of un-instrumented applications that have no way to determine what level of network bandwidth they are consuming. This working group is closely tied with NCSA's primary objectives to help users take full advantage of the vBNS for scientific research.

Pittsburgh Supercomputing Center FY97 Plans (PSC)

SDSC/UCSD FY97 Plans

vBNS

Visualization tools

We plan to further develop and enhance the Caida network tool set for drawing large network topologies and visualizing Internet data. This effort will include building conversion tools between our input files and those of other common public graping tools (e.g., graphed). We will also collaborate with Tom Sawyer visualization software to assess the utility of their graphing software package for NLANR and RA-collected data.

Visualization applications

We also plan to collaborate with the Routing Arbiter in developing better visualization methodologies and tools for the vast amounts of BGP data they collect.

Measurement tools

Packet and log file trace analysis

RA data analysis/IPN

Caching system/Squid

1. encourage greater U.S. participation. we have started to hear that the NAPs are finally installing cache hardware and we will be welcoming them to the NLANR hierarchy
2. standardize ICP protocol used for caches to communicate with one another, and develop more sophisticated automatic neighbor selection protocols and algorithms. Configuration of cache parents and neighbors is still quite manual, and only incorporates delay data rather than available bandwidth data in neighbor selection.
3. build a simulator for ICP
4. visualization of workload statistics
5. continue development of cache software (http://squid.nlanr.net/. Items for future releases include support for HTTP/1.1 protocol features, generic redirection to for access controls and automatic mirroring, If-modified-since requests between caches, reviving the Harvest encapsulation protocol for multiplexing cache-cache traffic over a single persistent TCP connection.
6. workshop participation and hosting. NLANR will participate in international as well as domestic workshops in the caching community.
7. investigate further the benefit of mutual-parenting relationships. Intuitively, this makes sense so long as the parent caches are located at ``high spots'' in the network (for example near each end of an international link), but we need empirical data to support the hypothesis.
8. develop a scalable cache discovery system: will likely involve using the DNS domain ircache.net for top-level domain cache location.
9. pushing objects into caches, i.e., server push
10. multicast: collaborate with multicast researchers (at LBL, UCLA) to determine how to best integrate multicast technology into the fielded hierarchy
11. benchmark hardware/software platform performance as high-end caches
12. explore how to accommodate the information provider need to obtain cache-hit feedback on their documents. Obviously providers want to collect demographics on visitors to their sites. From the user-perspective, some organizations prefer the fact that proxies and caches hide some information about the end users. Is there a middle ground? Should the protocol specify exactly what sort of demographics must be available to the information providers?

Other research questions that require attention include:

1. What obstacles will be encountered as we scale up the caches? We find that we are currently limited by the amount of physical memory and seek time of hard disks. Will arrays or clusters of caches be necessary and/or viable?
2. How do we enable providers to retain accurate hit count data without writing cache-unfriendly pages? We fear that a number of information providers make an effort to thwart caching so they can accurately log every access to their site. We would much rather develop a general-purpose access-reporting mechanism to satisfy both requirements: serving objects from a cache to save bandwidth and improve performance, and still provide a mechanism for information providers to obtain hit count data for their objects.
3. Do cache access traces represent an accurate slice of WWW traffic? A few people are using our sanitized log files for simulations and as a way to measure which web sites are popular. Because our caches are at the top levels of a hierarchy, their access logs do not likely represent end user browsing patterns.

Caida

IPNmoo

Iec

• 1996 quarter 2 report (1 apr - 31 june 96)
  
• 1996 quarter 1 report (1 jan - 31 mar 96)
  
• 1995 annual report (1 may - 31 dec 95)
• NLANR/NCRI Meeting 7/96 on Project Directions

Contents

• Cache access control
• Example access control configuration
• Usage Patterns
• Network Configuration
• Code status
• Sanitized access logs made public
• Cache Registration service
• Daily access log summaries and statistics
• Problems Encountered
• Status reports and presentations
• Mailing lists
• VRML Visualization
• Getting caches at the NAPs
• sample statistics
• Issues and Future Work
• Research Questions

In line with the goal of using the information gathered from network analysis to improve the performance of the architecture, NLANR continues work on its information caching prototype, and has submitted a proposal to NSF for continued evolution of the system. A key result of Internet workload analysis has been the recognition of the existence of an increasing proportion of web traffic as measured at reachable locations, as well as simulation results that indicate that caching could leverage the high degree of redundancy in web document transmissions.

Participation in the NLANR caching project continues to grow. We have upgraded the machines from 128 to 256 Mbytes of RAM for improved performance. The caches originally used the original Harvest cache software ( http://harvest.cs.colorado.edu/ ), designed for performance and hierarchical scaling. Since May 1996 we have been using the public domain squid software ( http://www.nlanr.net/Squid/ ), a community effort led by Duane Wessels, NLANR technical lead for the caching project. (Peter Danzig, architect of the original Harvest cache, is providing commercialization support for the harvest cache.

The international participation is still extraordinary, and dwarfs U.S. interest in caching in general. Following is a list of sites using the NLANR caches:

• Africa: iafrica.com
• Austria: univie.ac.at
• Australia: ausom.net.au, bns.com.au, connect.com.au, flex.com.au, geog.unsw.edu.au, mq.edu.au, senet.com.au, starwon.com.au unsw.edu.au, uts.edu.au, xavier.sa.edu.au
• Belgium: hades.sckcen.be, ucl.ac.be
• Canada: gmcc.ab.ca
• China: news.fudan.sh.cn
• Commercial: ange.nuri.net, compuserve.com, cray.com, gdscorp.com, hp.com, hunter.com, mot.com, netcom.com, nttlabs.com, pics.com, sony.com, wmsoft.com
• Czech Republic: cesnet.cz, feld.cvut.ca, liff.jcu.cz, vslib.cz,
• Denmark: netman.dk
• Germany: dkrz.de, fh-rpl.de mvv.de, netman.de, ruhr-uni-bochum.de, tu-chemnitz.de, uni-bonn.de, uni-frankfurt.de, uni-kiel.de, uni-kl.de
• U.S. Educational: binghamton.edu, colorado.edu, colostate.edu, cornell.edu, duke.edu, georgetown.edu, missouri.edu, msu.edu, psc.edu, uogi.edu, sdsc.edu, stanford.edu, sunysb.edu, syr.edu, trincoll.edu, usc.edu, utk.edu
• U.S. Government: lbl.gov
• India: ernet.in
• Israel: tau.ac.il
• Italy: forza-italia.it, lnf.infn.it unipd.it unive.it
• Japan: imasy.or.jp nttlabs.com
• Korea: kaist.ac.kr
• Networks: cic.net, cni.net, fugger.net, iphil.net, jwindow.net
• New Zealand: nzgate.net.nz,
• New Zealand: nzgate.net.nz, waikato.ac.nz
• Non-Profit: hall.org, loop.org, oceana.org, pacbell-sucks.org, park.org, seld.org
• Norway: uninett.no
• Philippines: admu.edu.ph i-manila.com.ph
• Poland: icm.edu.pl, pw.edu.pl, wsub.waw.pl
• Singapore: pacific.net.sg,
• South Africa: momentum.co.za
• (Former) Soviet Union: arcom.spb.su, sovam.com jet.msk.su
• Sweden: nada.kth.se
• United Kingdom: barclays.co.uk, demon.co.uk, gla.ac.uk, ic.ac.uk, ph.ed.ac.uk, mcc.ac.uk, tadpole.co.uk, tcp.co.uk

The sites shown in bold have established ``mutual parent'' relationships with us. That is, we agree to be their parent cache for URLs located in the U.S., and they are our parent for URLs located within their country.

We began in July to place restrictions on using the caches, to encourage a more sound hierarchical topology. accept connections from any address and will process any URL request. We announced this policy in May in response to visualization analysis of the neighbor database.

Cache access control includes the following objectives:

• Each peer cache will be allowed to use two NLANR caches, basically determined by geography/topology. By default European sites will use our East-coast caches, and Asia-Pacific sites will use our West-coast caches. U.S. sites should use whichever NLANR caches are appropriate
• The class A/B/C network of each peer cache is specified in the config file, to allow backup/replacement machines and some address renumbering with minimal config changes.
• We strongly encouraged new sites to join this hierarchy/mesh. We may first ask non-U.S. sites to peer with an existing cache in their country.

Note these are only guidelines, not hard-and-fast rules. We are very open to other arrangements which make sense.

The list below shows an example access configured for each site, generated by ranking all NLANR clients for the month of June by number of requests. Low ranking sites from countries with higher ranking caches have been dropped.

ebone-proxy.univie.ac.at        192.174.65.0/24      it pb
cardrout.card.ucl.ac.be         130.104.0.0/16       it pb
gateway.gmcc.ab.ca              198.161.33.0/24      bo it
news.fudan.sh.cn                202.120.224.0/24     sv sd
tinderbox.netman.dk             193.88.72.0/24       it pb
ababel.ecm.ub.es                161.116.0.0/16       it pb
iitk.ernet.in                   202.141.40.0/24      it pb
light.imasy.or.jp               202.227.24.0/24      sv sd
cosmos.kaist.ac.kr              143.248.0.0/16       sv sd
isis.waikato.ac.nz              140.200.0.0/16       sv sd
sunsite.icm.edu.pl              148.81.0.0/16        it pb
sunsite.nada.kth.se             130.237.0.0/16       it pb
pacific.net.sg                  203.120.90.0/24      sv sd
zaphod.arcom.spb.su             194.135.104.0/24     bo it
alpha.icmp.lviv.ua              193.124.228.0/24     bo it
norse.mcc.ac.uk                 130.88.0.0/16        it pb
phoenix.doc.ic.ac.uk            193.63.255.0/24      it pb
sirius.tadpole.co.uk            192.52.161.0/24      it pb
connect.com.au                  192.189.54.0/24      sv sd
manifera.itd.uts.edu.au         138.25.0.0/16        sv sd
iliad.lib.mq.edu.au             137.111.0.0/16       sv sd
netra.geko.net.au               203.2.239.0/24       sv sd
holly.flex.com.au               203.19.214.0/24      sv sd
starwon.com.au                  203.15.243.0/24      sv sd
geog.unsw.edu.au                149.171.0.0/16       sv sd
topdown.bns.com.au              203.19.43.0/24       sv sd
bbs.ausom.net.au                203.8.161.0/24       sv sd
vweb.access.net.au              203.13.25.0/24       sv sd
eyre.iinet.net.au               203.19.149.0/24      sv sd
alpha.xavier.sa.edu.au          203.17.192.0/24      sv sd
spock.rz.uni-frankfurt.de       141.2.0.0/16         it pb
rz.ruhr-uni-bochum.de           134.147.0.0/16       it pb
uran.informatik.uni-bonn.de     131.220.0.0/16       it pb
doener.unix-ag.uni-kl.de        131.246.0.0/16       it pb
aiken.zw.klautern.fh-rpl.de     143.93.0.0/16        it pb
proxy.germany.eu.net            194.175.173.0/24     it pb
www-proxy.iafrica.com           196.7.0.0/24         sv sd
umomware.momentum.co.za         196.13.225.0/24      sv sd
groa.uct.ac.za                  137.158.128.0/24     sv sd
iphil-communications.ph         204.70.35.0/24       sv sd
crispin.i-manila.com.ph         203.172.14.0/24      sv sd
vultur.vslib.cz                 147.230.0.0/16       it pb
adis.cesnet.cz                  194.50.6.0/24        it pb
digi.feld.cvut.cz               147.32.0.0/16        it pb
ax0rm1.roma1.infn.it            141.108.0.0/16       it pb
server1.infn.it                 131.154.0.0/16       it pb
www.forza-italia.it             194.166.23.0/24      it pb
mail1.fw-bc.sony.com            198.83.177.0/24      it sv
telford.nsa.hp.com              15.0.0.0/8           sv sd
gatekeeper.hunter.com           199.217.148.0/24     uc
thesparc.wmsoft.com             194.64.125.0/24      it uc
ns.nttlabs.com                  204.162.36.0/24      sv
zmark.cts.com                   204.94.75.0/24       sd
ucar.edu                        128.117.0.0/16       bo
colorado.edu                    128.138.0.0/16       bo
msu-buzz.edu                    206.26.77.0/24       bo
syr.edu                         128.230.0.0/16       it sv
cornell.edu                     128.84.0.0/16        it
trincoll.edu                    157.252.0.0/16       bo
bxscience.edu                   204.217.8.0/24       uc bo
colostate.edu                   129.82.0.0/16        bo
missouri.edu                    128.206.0.0/16       it sv
psc.edu                         147.72.0.0/16        pb
sunysb.edu                      129.49.0.0/16        it
diogenes.sedl.org               198.213.9.0/24       sv
valinor.oceana.org              205.233.219.0/24     pb uc
janus.sedl.org                  198.213.9.0/24       uc bo
ren.ee.lbl.gov                  131.243.0.0/16       sv
ange.nuri.net                   203.255.112.0/24     sv
zuse.space.net                  194.59.182.0/24      it uc
isca10.fugger.net               194.77.44.0/24       uc
tm3.cni.net                     194.115.51.0/24      pb
binabik.gatewest.net            198.163.227.0/24     uc
musclewood.hq.cic.net           198.108.58.0/24      uc
shell.wisp.net                  198.67.33.0/24       it

Usage Patterns

A busy cache will typically handle 150,000 requests and serve 3 Gbytes of Web objects. Cache hit rates are usually in the range of 5-10% percent and a couple of percentage points higher when weighted by byte volume.

By far, the largest number of requests are in the .com domain. Often 60% of all requests are for .com URLs. Another 20% are for .edu, .net, and .org. We calculate that the caches are currently only being used to about 15% of their capacity. The cache software should be able to handle one million requests and 16 Gbytes per day. However, at this rate, we would be serving twice as much data per day than we could store on disk; additional disk space would be desirable.

We have also been exploring metrics to indicate how well client caches are using parent caches. One measure is relative number of ICP requests vs HTTP requests (queries vs fetches). We probably do not want to see only one HTTP fetch for every 10 ICP queries. However, for sibling caches we could expect a pretty low HTTP:ICP ratio because a cache only fetches from siblings on a HIT.

So another measurement is hit rate. If a cache gets good hit rates from us, then maybe the HTTP:ICP ratio doesn't matter so much.

So what we are tentatively calling "cache utilization" is

	     HTTP_COUNTS      HTTP_HITS
utilization = ------------- + --------------
	      ICP_COUNTS     HTTP_COUNTS

Here's calculated utilization values for 'sv' on 7 May 1996:

		HIT    H/I
REVERSE FQDN            RATE   RATIO  UTIL
----------------------- ----   ----   ----
edu.georgetown.webcache 0.56 + 1.00 = 1.56
gov.lbl.ee.ren          0.24 + 0.99 = 1.23
org.sedl.diogenes       1.00 + 0.21 = 1.21
edu.syr.rocket          1.00 + 0.17 = 1.17
[206.106.252.226]       1.00 + 0.14 = 1.14
br.com.gns.srv1-bsb     0.33 + 0.80 = 1.13
net.nuri.ange           0.43 + 0.63 = 1.06
[193.175.197.97]        1.00 + 0.04 = 1.04
uk.ac.ic.doc.phoenix    1.00 + 0.04 = 1.04
IT                      0.07 + 0.94 = 1.01
com.hp.access.paloalto  0.20 + 0.80 = 1.00
BO                      0.15 + 0.85 = 1.00
com.sony.fw-sj.user1    0.00 + 1.00 = 1.00
au.com.flex.holly       0.37 + 0.62 = 1.00
SD                      0.08 + 0.91 = 0.99
com.nttlabs.ns          0.03 + 0.95 = 0.98
com.vivid.greedo        0.14 + 0.80 = 0.94
PB                      0.07 + 0.87 = 0.94
dk.netman.tinderbox     0.00 + 0.94 = 0.94
jp.or.imasy.light       0.27 + 0.66 = 0.93
net.mci.sanfrancisco.ip 0.16 + 0.72 = 0.88
nz.ac.waikato.isis      0.26 + 0.62 = 0.87
au.edu.uts.itd.manifera 0.22 + 0.64 = 0.85
cz.vslib.vultur         0.78 + 0.07 = 0.85
net.nlanr.oceana        0.00 + 0.80 = 0.80
jp.or.imasy.luck.yebisu 0.50 + 0.26 = 0.76
au.edu.unsw.geog.electr 0.17 + 0.55 = 0.72
localhost               0.01 + 0.69 = 0.70
edu.ucar.scd.surf       0.20 + 0.50 = 0.70
au.net.access.power4    0.64 + 0.06 = 0.70
au.com.connect.myangup  0.17 + 0.52 = 0.69
it.infn.lnf.pccarboni   0.40 + 0.28 = 0.68
uk.ac.mcc.norse         0.64 + 0.04 = 0.68
de.tu-chemnitz.hrz.hasi 0.60 + 0.03 = 0.63
au.net.geko.netra       0.08 + 0.48 = 0.57
de.uni-frankfurt.rz.spo 0.30 + 0.25 = 0.55
uk.co.tadpole.sirius    0.50 + 0.02 = 0.52
au.edu.mq.lib.iliad     0.18 + 0.34 = 0.52
au.net.access.vweb      0.33 + 0.11 = 0.45
de.uni-bonn.informatik. 0.19 + 0.10 = 0.29
de.uni-kl.unix-ag.doene 0.00 + 0.00 = 0.00
org.sedl.janus          0.00 + 0.00 = 0.00
br.com.gns.pixel-bsb    0.00 + 0.00 = 0.00
de.dkrz.www             0.00 + 0.00 = 0.00

We expect that around 0.80 utilization is a reasonable threshold; we may want to ask any child cache falling below that point to use a different (or fewer) parent cache(s).

A full collection of statistics can be found at http://www.nlanr.net/Cache/Statistics/ .

Network Configuration

Because some of the cache machines are located at supercomputer center sites, they are able to communicate with each other over the vBNS. This is advantegeous because it provides a high-speed cache-to-cache communication channel. It means that in most cases, there is very little penalty to retrieve an object through another NLANR cache. The FIX-West cache is not on the vBNS, but we have established a collaboration with DREN which enables us to leverage the DREN connectivity between NOSC in San Diego and FIX-West.

Because the supercomputer centers are not optimal locations within the Internet topology, we are still trying to locate caches at top-level interconnection points (such as the FIXes and NAPs). Because the supercomputer sites are one or two ISPs ``underneath'' from the backbone providers, U.S. users may find little benefit in using an NLANR cache. The FIX-West cache is actually in a very good location and this makes it more popular than the others, and we continue trying to convince NAP service providers to participate in this project.

Code status

The current version of the caching code, squid has many powerful features: Support for SSL, graceful shutdown without loss of current cache obejects, store.log that logs the actions of the storage manager and swap status/ttl of a requested object. private objects, proper parsing of HTTP reply codes, support for If-Modified-Since GET, improvements to the access log, metadata reloads in the background, flexible access control scheme, using SIGHUP to reconfigure the cache, an ftpget server to avoid forking, and assigning weights to cache neighbors.

We also purchased and installed an NLANR copy of Purify software for use with Squid code development, which has been a great help (no advertisting intended, the software speaks for itself.)

Sanitized access logs made public

We make sanitized versions of our daily cache access log files available for anyone to use. Some organizations are using these logs to analyze trends in WWW access patterns. At least one is using them to determine which Web sites are the most popular. The log files are available via anonymous ftp from ftp://oceana.nlanr.net/Traces/Caching/.

Cache Registration service

The Harvest project included the idea of providing a registration service for caches. This would allow cache users to locate one another and set up local cache hierarchies. The initial registration service was via email, which proved to be problematic.

NLANR now operates a cache registration service based on UDP announcement messages. If configured, a cache will send an announcement at regular intervals (i.e. once per day) or every time the cache daemon starts. The announcement includes hostname and ports, software version, contact information, geographical location (latitude/longitude), and any additional comments the administrator wishes to add. The registration database can be viewed at http://www.nlanr.net/Cache/Tracker/ .

Daily access log summaries and statistics

We provide daily summary reports of cache activity. These reports include total traffic through the caches, hit-rate percent for number of requests and by traffic volume, and a breakdown of the request result types (hit, miss, error, etc). The reports also give the top 20 server sites being accessed, the top clients, the top URLs, and a list of object types (loosely determined from the URL). These are available at http://www.nlanr.net/Cache/Statistics/ .

Problems Encountered

Unfortunately we had a streak of bad luck with Digital's Unix operating system. Our moderately busy, FDDI-attached machines exercised a few different bugs which caused the machines to hang very often (i.e. daily). First, a problem with the FDDI driver would hang a machine if it received too many illegal packets coming from a Cisco AGS+ router. Second, every received (small) packet would be placed into a separate `mbuf' of 8192 bytes. After some time, megabytes of RAM would be consumed by the network. Third, a bug in the TCP code would cause the maximum window size to become huge. This caused megabytes of data to become stored in a socket receive queue and led to a shortage of mbufs described above.

After two months of dealing with Digital technical support we were finally put in touch with kernel developers who provided us with fixes for these problems. Since then, the machines have been operating without any unexpected hangs.

Status reports and presentations

We have written status reports and given a number presentations on this project. Claffy has given presentations at NANOG, IETF the Federal Networking Council (FNC), and numerous ISPs. Wessels has presented the project at NCAR and SDSC. Wessels was an invited speaker to a workshop on caching in Warsaw, Poland. He gave about four talks during the workshop.

Mailing lists

NLANR operates a number of mailing lists related to this project:

• ircache
• ircache-dev
• squid-users
• squid-bugs
• cache-coders

All available by requesting to info@nlanr.net.

VRML Visualization

Claffy, Tamara Munzner (Stanford), and Eric Hoffman (Ipsilon) have produced 3-D VRML representations of the NLANR cache hierarchy. Using a database which translates IP addresses into geographical lat/long coordinates, they are able to generate a global view of which sites are using the NLANR root caches. These visualizations are available at http://www.nlanr.net/Cache/cacheviz.html , with a view updated daily at http://www.nlanr.net/Cache/daily.html .

Getting caches at the NAPs

Claffy has been working with the NAPs, encouraging them to install caches at their sites. So far, two have placed orders for machines. We are excited about getting caches at the NAPs since they will be much more logical locations for many U.S. users.

---

sample statistics

We make current usage statistics publicly available at http://www.nlanr.net/Cache/Statistics in several formats

Amount of data served by the NLANR caches (per day)
since project beginning (27 November 1995)

---

We also provide graphs for each cache separately, at http://www.nlanr.net/Cache/Statistics/Graphs/, and reports and summaries at http://www.nlanr.net/Cache/Statistics/Reports/ and http://www.nlanr.net/Cache/Statistics/Summaries/, respectively.

Each of the cache machines is a high-end Digital Unix server with 10G of disk, 256M of RAM, and FDDI interfaces. The cache software keeps object metadata and hot objects in virtual memory. We found that with 128M of RAM we must limit our cache size to around 200,000 objects or 3.3G bytes. When the cache process grows too large, the system has swap the process to disk, which degrades overall performance. We increased the memory size to avoid this behavior as much as possible. Still, we're working those puppies pretty hard.

As the above graph indicates, a cache will often serve 2G bytes of data per day, more than half of its entire cache size. We typically observe cache hit rates ranging from 10-20%, not so surprising since

• almost all of our clients are other caches which handle the majority of cache hits.
• our cache size is too small for the load it is handling. We need to increase our cache size which will require more physical memory.

Issues and Future Work

Ongoing activities will be in areas to:

1. encourage greater U.S. participation. we have started to hear that the NAPs are finally installing cache hardware and we will be welcoming them to the NLANR hierarchy
2. standardize ICP protocol used for caches to communicate with one another, and develop more sophisticated automatic neighbor selection protocols and algorithms. Configuration of cache parents and neighbors is still quite manual, and only incorporates delay data rather than available bandwidth data in neighbor selection.
3. continue development of cache software (http://squid.nlanr.net/. Items for future releases include support for HTTP/1.1 protocol features, generic redirection to for access controls and automatic mirroring, If-modified-since requests between caches, reviving the Harvest encapsulation protocol for multiplexing cache-cache traffic over a single persistent TCP connection.
4. workshop participation and hosting. NLANR will participate in international as well as domestic workshops in the caching community.
5. investigate further the benefit of mutual-parenting relationships. Intuitively, this makes sense so long as the parent caches are located at ``high spots'' in the network (for example near each end of an international link), but we need empirical data to support the hypothesis.
6. develop a scalable cache discovery system: will likely involve using the DNS domain ircache.net for top-level domain cache location.
7. pushing objects into caches, i.e., server push
8. multicast: collaborate with multicast researchers (at LBL, UCLA) to determine how to best integrate multicast technology into the fielded hierarchy
9. benchmark hardware/software platform performance as high-end caches
10. explore how to accommodate the information provider need to obtain cache-hit feedback on their documents. Obviously providers want to collect demographics on visitors to their sites. From the user-perspective, some organizations prefer the fact that proxies and caches hide some information about the end users. Is there a middle ground? Should the protocol specify exactly what sort of demographics must be available to the information providers?

Research Questions

Other research questions that require attention include:

1. What obstacles will be encountered as we scale up the caches? We find that we are currently limited by the amount of physical memory and seek time of hard disks. Will arrays or clusters of caches be necessary and/or viable?
2. How do we enable providers to retain accurate hit count data without writing cache-unfriendly pages? We fear that a number of information providers make an effort to thwart caching so they can accurately log every access to their site. We would much rather develop a general-purpose access-reporting mechanism to satisfy both requirements: serving objects from a cache to save bandwidth and improve performance, and still provide a mechanism for information providers to obtain hit count data for their objects.
3. Do cache access traces represent an accurate slice of WWW traffic? A few people are using our sanitized log files for simulations and as a way to measure which web sites are popular. Because our caches are at the top levels of a hierarchy, their access logs do not likely represent end user browsing patterns.

---

NLANR quarterly meeting (Tuesday 27 june 1996, IETF, Montreal)

The following meeting summary is included as it outlines NLANR strategies and future emphases.

Goal of meeting: focus NLANR on key areas where we can make an impact.

1. What is NSF getting for its investment in the vBNS
   1. Get information back to NSF on what is going on with the vBNS
   2. Produce quarterly reports modeled on SCC quarterly reports
   3. Demonstrate overall plan, tangible results (milestones) each quarter
   4. Include plans/goals for next quarter in each quarterly report
2. Increase application usage
   1. Better vBNS PR to SCC users via newsletter articles and training
   2. Solicit high profile applications (i.e. GC3, IWAY)
   3. Prepare better documentation for new users of the vBNS
3. Prepare vBNS for the influx of new connections
   1. Work with sites on planning & overall network architecture
   2. Attempt to implement technical solutions for congestion
   3. Help make sense of conflicting requests to MCI
   4. Provide ongoing support for specialized routing needs. Include in this addressing fundamental tensions between IP users and ATM users.
4. Promote research into new networking technologies on the vBNS
5. Specific operational needs
   1. Strong need for network monitoring and real data on reliability and utilization.
   2. Generate a technical plan to implement something by 7/15
   3. Review current routing architecture and discuss any desired changes.
6. Design/Architecture for new sites
7. Applications Working Group
   1. Define goals for an applications working group
   2. Discuss strategies for increasing usage by current apps and bringing on new apps
   3. Select a core group to implement strategies as rapidly as possible.
8. Networking Research Working Groups
   1. Brainstorm possible areas of interest
   2. Select networking research areas which will have maximum impact
   3. Select a core group (or groups) to forge ahead in these areas