Summary of Research Activities - August 2003

• Development and distribution of measurement and analysis tools:
• Activities extending the Network Analysis Infrastructure (NAI) in support of new and developing HPC needs:
• Outreach, application support, utilization improvement, and documentation activities:
• Ongoing measurement and analysis, networked data, and infrastructure support:

Development and distribution of measurement and analysis tools:

Good progress continues on the development of new metrics and real-time analysis for PMA.

• Spent quite some time discussing data link layers and their differences and merits with Chris before moving on from rather educational and philosophical subjects to a really good follow-on meeting to shape the nature of the real-time application we would like to design. One way to look at the Internet is to think about how you drive your car. You listen to the sound of the engine, the noise of the road, the feeling the steering wheel provides for grip on the road, the forces you experience in going around corners, the light conditions, rain, ice, etc. With the Internet, there are no such signals. The users, or network administrators, are flying blind: no sounds, no windows, no radar screen. The first time you notice is when something "doesn't work." The change is rather sudden, like the car falling apart in the middle of the highway at 70 miles an hour. No warnings whatsoever. This engine we are trying to design should provide the kind of "noise" characterizing normal behavior, and put the user into the position to detect abnormal behavior early, labeled neither good nor bad, just different. In time, that is important. [Jörg Micheel]
• Finished a first version of the flow engine, and began making it more sophisticated. Had a very interesting and productive meeting with Jörg discussing the different metrics to pull out of the flows that I am getting. [Chris Gross]
• Advance on the technical front of the real-time analysis project has been going well. With the PC from Jamie Curtis (WAND), Chris began the installation process; we used another Dag3.2E to terminate the connection from the hub, which is our tap into the link. We can continue using the SDA point (SDSC Abilene link monitoring ATM OC12c) for the time being. This will give Chris an opportunity to test his design on a different link layer. [Jörg Micheel]
• Began to look at other link layer technologies that are present in the NAI. To date, my real-time development has been focused solely on Ethernet and now that I have access to the SDA machine in San Diego it can focus on ATM/AAL5 at OC12 speeds. Became stumped with the parsing of the 48 byte ATM payloads. Jörg helped in understanding the ATM AAL5 LLC/SNAP encapsulation for IP packets.  [Chris Gross]
• During discussions with Chris, we also settled on understanding why it is that there is no payload present, as the DAG card filters all but the first cell from the stream on the link. We discussed how this came into being and the reasoning behind it.  [Jörg Micheel]
• We have put together something of a remote sensor. There are presently three different layers. The first layer will look basically at the link and IP headers and produce a total of 7 counters, broken down into packets and bytes, inbound and outbound. Counters are all 32 bit. The second layer looks at applications (TCP/UDP port numbers) and records the top 10 active applications (the limit of 10 is rather arbitrary at this point). The third layer looks at a number of flow (connection) properties. We are still carving those out, but they look a great deal at TCP/IP connection intrinsics, connection symmetry, packet loss, retransmission, reordering, SYN/SYN/ACK, FIN/FIN handshakes, and much more. These parameters would be sampled once per second and fed back to a central collection point (pma.nlanr.net). First estimates indicate we will have a data flow of 1024 Bytes/sec (8 Kbits/sec). Imagine a hundred sensors in the field - this still makes for less than 1 MBit/sec at the central collection point, quite a reasonable design. We would store those parameters in a database per each collection point and age them on an hourly, daily, weekly and monthly basis, using RRD. The RRD tool suite would also be used to display the sampled parameters on a Web page. [Jörg Micheel, Chris Gross]
• Klaus and I touched bases about our TCP flow engines. We have decided to continue writing our own engines then compare them for performance advantages afterwards. Taking each of the "best parts" and combining them should form what will hopefully be an awesome engine. Klaus also sent me a library that helps process erf records, both current ones and legacy ones. [Chris Gross, Klaus Mochalski]
• The next evolution of my real-time program seems to be done; now it can work nicely with ATM and Ethernet, either legacy or otherwise. I have also been further structuring my program to allow of the addition of a second stream of data, be it from another interface on the same card, or from a second physical card.  [Chris Gross]
• Am working on code to merge two streams of traffic for bidirectional statistics. Finished a fair bit of the code to merge the Dag streams. I have actually come up with two different ways to account for the traffic on each link; I have to decide which path to take. Also started working with RRDtools.  [Chris Gross]
• After discussion with Jörg, I changed the instantaneous bandwidth tool to display gaps between packets instead of the actual instantaneous bandwidth of a packet. This avoids the artificial packet size dependency of the previous approach. However, the graph turned out to look much less clean than previous ones. There were just too many spikes in the histogram for different, discreet gap lengths. Thus, I chose to use a cumulative histogram (CDF) instead, which works much better. It also gets rid of some scaling problems on the axes. Preliminary tests with some trace files looked really promising in that they revealed some properties of the traffic which were not visible by just looking at bandwidth averages. [Klaus Mochalski]
• Began evaluating the output of my histogram sequence (instantaneous bandwidth) tool. The sample sequences at http://pma.nlanr.net/~klaus/histo/ have some really interesting features. Comparing the different directions of the same trace shows that this packet gap histogram approach yields some insights into the traffic behavior which are not visible by just looking at average utilization figures. In some cases they actually provide quite opposite ratings for bandwidth availability than the average utilization does. [Klaus Mochalski]
• Klaus has come back with some very interesting updates on his link level packet analysis. We have yet to understand all of the findings, but I honestly think he is onto something and we may well be working on a single measurement point notion of IP packet congestion. [Jörg Micheel]

Additional progress was made on the reimplementation of AMP and development of a new testing architecture.

• Progress was made on the amplet code, primarily doing soak tests and correcting a few bugs. Had some difficulty tracking down another bug that only happened every hour or so. The behavior itself seemed quite random as well. It turned out to be a buffer overflow that was happening as a result of the thread pool overflowing. The thread pool overflow was happening because previously I had not correctly thought through what should happen when a thread terminated because of an error. Fixed this bug, then immediately ran into a deadlock that was also related to abnormal exits. Pthreads has a nice mechanism to deal with that (a stack of exit routines) but the Linux macro implementation of the push includes an unmatched open brace that is matched by the pop. This is problematic because it severely limits how you can use the routines. I eventually worked around this and have cleanup code that works on FreeBSD and Linux; still need to test it on the other systems. Currently progress seems a bit slow; but it is good to find these problems now, rather than after I release the code. Also have been tidying up bits and pieces, like the debug code and making the constants more usable.  [Tony McGregor]
• Will be writing code to add Iperf to the testing architecture of the AMP reimplementation software. Discussed the particulars with Tony and did several things in preparation. Created a development environment, obtained the Iperf source code, and compiled it. Looked through code and started running it. Working with various people regarding aspects of this project, including meeting with Ben to discuss multithreaded programming and how networks function. [Lana Kennedy]

Worked more on the Scamper library, and have been using it to quickly write code that can help analyse data that has been collected from the six Scamper runs that have been taken from various points in the IPv6 Internet. <Details on these analysis programs are in the next section of this report.>   [Matthew Luckie]

Modified the Scamper source slightly to make it compile on MacOSX, which only took three lines of code to do in the end, due to a difference in MacOSX's implementation of recvfrom. I have not tested it yet, as I do not have the ability to open the raw sockets without root privileges; I expect that it would work, though. [Matthew Luckie]

Progress on the reimplementation of the Cichlid 3-D Visualization System continued.

• Began developing a more sophisticated interface for graph objects to publish their unique options (NURBS, render quality, interpolation) to the windowing interface in a graph-independent way. This consists of an interface infrastructure that can enumerate a list of graph characteristics and the available options (multiple choice) for each one, which then gets built into a menu. All of this happens without the window class knowing anything except a dynamically allocated data structure which holds character strings. [Ben Reesman]
• Also spent more time working with the windowing system, regarding the generation of timer events to keep the graph animating properly. One aspect is that I had to be more specific about the particular mode of operation that the player is working in, as the previous animation was not adequate to the task. [Ben Reesman]
• Worked on routines that pack and unpack bar charts and took some advice from Klaus on how to speed up the partial data updates for bar chart graphs. Developed a new serialization method for similar graphs, extending my previous implementation to include a more sophisticated interface which allows multiple units of color per cell. Also wrote a more elaborate serialization method using these unit data structures. [Ben Reesman]
• In preparation for correcting several mistakes in the way that I have designed Cichlid, spent a lot of time studying network and systems programming fundamentals. The two most important regard portability: byte ordering of network traffic and the portability of threading. Wrote 1500 lines of supplementary code for Cichlid which solves some of the problems that I have raised with the work to date. This includes: a small thread library, byte packing code, new network utility code, and a rigorous network specification. Todd Hansen and I also wrote several key routines in the client rendering setup. [Ben Reesman]
• Rewrote (yet again) some of the handshaking code for some of the threads: have eliminated ALL of the crashes that I had been experiencing, and also eliminated all of the mutex destruction bugs that were plaguing me. [Ben Reesman]
• Used some of the new Cichlid drawing code to create three-dimensional pie charts for the Citings/Proceedings Web page. The pie chart format will be a part of the new Cichlid implementation. [Ben Reesman]

Activities extending the Network Analysis Infrastructure (NAI) in support of new and developing HPC needs:

I did some work to profile the length of loops discovered during the Scamper IPv6 runs, which is a fairly important statistic given that 25% of all addresses in the list failed with a loop detected.   http://voodoo.cs.waikato.ac.nz/~mjl12/ipv6-scamper/  [Matthew Luckie]

Wrote a series of programs to do some rudimentary analysis on the data collected from the six Scamper runs that have been taken from various points in the IPv6 Internet. http://voodoo.cs.waikato.ac.nz/~mjl12/ipv6-scamper/   [Matthew Luckie]

• The first is a simple program to test the API by plotting a simple histogram of hop lengths seen for a given Scamper file.  [Matthew Luckie]
• Wrote a program that, given a prefix, will extract all addresses that match the prefix and then draw a directed graph of all ingress and egress hops for that prefix. A diagram of ipv6.aorta.net based on data collected with Scamper at   http://voodoo.cs.waikato.ac.nz/~mjl12/ipv6-scamper/127prefix.aorta.png. The red bits are examples of what could be /127 routing; the green bits indicate the addresses that fit within the prefix. [Matthew Luckie]
• Wrote a program that, given a Scamper file, will produce a graph of the RTT against the hop. The bits that are interesting are the RTTs that are seconds above the mean RTT seen, which might indicate the use of a tunnel. [Matthew Luckie]
• Also wrote a program to generate basic stats of the hit-rate of the various address sources used to compose the IPv6 address list. [Matthew Luckie]
• Wrote a program to fetch out the EUI64 addresses seen in the Scamper traces and obtain counts of the Organizationally Unique Identifiers (OUIs) from them, and convert that to OUI name. [Matthew Luckie]
• Started investigating the presence of /127 routing prefixes - something that I am not sure makes a lot of sense given the traceroute method of data collection. Given two sequential hops, if the first 127 bits matched then I pulled them out and suggested that a /127 might be in use.   http://voodoo.cs.waikato.ac.nz/~mjl12/127prefix.html  As far as I can tell, ICMP responses in IPv6 will come from the interface that the packet with the TTL expiry was received on, so my presumption is that sites are configuring some routes statically, otherwise I am at a loss to understand how I would see these /127s at all (other than the last hop in the traceroute). The following diagram is how I'm picturing the use of the /127. The ??? labels are addresses of the interface used to forward the packet on, of which I do not know the addresses. Later modified my table of /127s to remove those where the second address in the /127 was the last address reported in a traceroute to be sure that the pair was used in routing. [Matthew Luckie]
  

    ---> [2001:730::1:44 : ???] ----> [2001:730::1:45 : ??? ] --->

Work continues on the development of the OC192mon. Working with Kevin Walsh of SDSC NetOps, arranged to obtain direct access to the Spirent Adtech test equipment which we are using to test the monitor. This will enable control over the test values as well as be invaluable to the project in the future. Other collaborative details have also been resolved; Kevin has great enthusiasm in assisting us with this project. I expect to have access to the equipment by Jörg's visit to SDSC in early September. We are also working with Stephen Donnelly of Endace on the issues with the OC192mon. Looked into more powerful equipment to use in testing the new Dag6 cards per Stephen's request.  [Jim Hale]

We now understand that the Adtech supports all three PHY modes and we have configured it to OC192c PoS (as opposed to the other 10GigE LAN and WAN modes). Still seeing RXF errors at the PHY chip (Khatanga). Stephen reckons there is an issue with the two cards in the system, the folks are working on it. I am hoping to resolve this during my visit to SDSC in early September. [Jörg Micheel]

I did some changes to the anonymization tool to make it work for the GigE traces I have captured in Leipzig (thanks to Jesper Peterson of Endace for his help with this). These are part of Leipzig-II, a two-point trace taken at the Internet access router. I anonymized the traces and I just need to do some tweaks to make them available for download. [Klaus Mochalski]

~ New deployments and updates on new (and developing) strategically important measurement sites

Two new PMA monitors which were recently completed and shipped to the sites - the KISTI site in Korea and the AMPATH GigaPop in Miami, Fla. - have been installed and connected. They are expected to be coming online soon as connection issues are being resolved. Work continues getting the MAX GigaPop PMA site online.  [Bud Hale]

A PMA monitor was assembled, configured, and shipped to Internet2 in Ann Arbor, Michigan, and is expected to be connected soon. Working with Matt Zekauskas on this. There had been a possibility of Jörg visiting Ann Arbor, but his travel plans will not allow time to do this. [Jim Hale]

The additional information needed to ship the requested AMP monitor to the amp-cudi, Corporacion Universitaria para el Desarrollo de Internet(CUDI), site was received. The machine was shipped this period and, after some work with Mexican Customs, the machine arrived at CUDI and was installed. It has been connected and is now collecting data. [Bud Hale]

The requested AMP monitor for the Northwestern University end of the NaukaNet was shipped. That monitor has arrived on site, was installed and initiated, and is now collecting and transferring data. [Bud Hale]

The long-awaited startup of the AMP monitor at the Great Plains Network GigaPop, amp-gpng, was finally connected and started this period. [Bud Hale]

Outreach, application support, utilization improvement, and documentation activities:

Attended ACM SIGCOMM 2003 in Karlsruhe. It was not only a data communications festival, but a prime opportunity to talk to a lot of people in the network arena. The list of people is way too long, sufficient here to say that I caught up with Anja Feldman (TU Munich), she is the General Chair for SIGCOMM this year, Christophe Diot, with whom we continue to find a way to install more measurement points and support activities by other research groups, the folks from AT&T Research, and a lot of European research organizations. Met people from Internet2 and the NSF, naturally. [Jörg Micheel]

Attended the Internet2 Security at Line Speed - NSF Sponsored Workshop (Invitation Only, total of 30 attendees), in Chicago, IL, August 12–13, 2003. The outcome was not unexpected, but educational. Most people will treat security as a matter of restricting IP communications by moving "wide open" to "tailored to needs" via firewalls and similar "middleboxen". More interesting was the fact that tools are all there, no surprises expected in the near future, but changes are expected in the way that those mechanisms are operated. Multi-tiered policies, better ways to communicate, multi-campus arrangements and similar communications and cultural changes promise some form of "improvement" on the subject. Overall, reassuring to see that there are no strict technology changes expected.   http://apps.internet2.edu/sals/2003aug/agenda.html  [Jörg Micheel]

Attended the Kansas Joint Techs TEV meeting in Lawrence Kansas: a busy and productive number of days, including meeting with Greg Monaco. [Ronn Ritke]

Participated in the PIPES meeting. Eric Boyd is interested in using the AMP data as a fundamental building block of the PIPES project. [Ronn Ritke]

Attended the JET meeting. There was interest from DOE in the OC192mon. [Ronn Ritke]

Attended the PRAGMA meeting at Peter Arzberger's house. Attendees included some colleagues from Taiwan who were quite interesting, as well as some of the LTER folks. [Ronn Ritke, Hans-Werner Braun]

Gave a presentation with Peter Arzberger of PRAGMA to Dr. Yang from Korea. [Ronn Ritke]

Lana Kennedy and Ben Reesman each prepared and presented a poster at the SDSC sponsored Student Symposium for student researchers on their current work and activities.  

Participated in the SDSC Measurement Working Group meeting. There are plans to have some kind of measurement meeting at SDSC in December. [Ronn Ritke]

Steve Corbato of Internet2 is keen for us to get one of the OC192 systems installed into IPLS. Currently deferred as the system in San Diego is not working yet. We may schedule the installation towards the end of October, piggybacked to IMC in Miami. [Jörg Micheel]

I sent the am_slave code to Warren Matthews and wrote up some installation instructions so he can get the data from the SLAC monitor in real time. [Tony McGregor]

Many email exchanges with Nevil Brownlee, who is working away on NeTraMet based analysis on DAGMONs. [Jörg Micheel]

Met with Rick Summerhill. NLANR/MNA has the opportunity to put AMP machines at each of the 11 Abilene backbone nodes as part of the Observer project. The Front Range GigaPop was shipped by Bud and will soon be connected. [Ronn Ritke]

Met with Paul Schopis; he is supportive of hosting an AMP at the OARnet GigaPop. [Ronn Ritke]

Met with Margaret Murray about the ENDACE card donations and collaboration with CAIDA. [Ronn Ritke]

Received a brief note from Ian Pratt about PAM2004 progress. [Jörg Micheel]

During the Internet2 Security at Line Speed workshop had several interesting chats with Kevin Thompson from NSF.  [Jörg Micheel]

Significant progress was made on the Citings: Data Users Web pages; several were completed sufficiently for posting. The aim of these pages is to know who is doing what with our data, and approximately how many folks are using our publicly available measurements, analyses, and tools, for their own research efforts. Created three separate Citings pages (in addition to index page): a list of Citings in alphabetical order by year, a list of just the meetings and related pie charts without the citation list of papers following, and the complete Citings/Proceedings page with the meetings, the citations, and the pie charts. Worked on a layout redesign of the Citings Proceedings page; the new design accommodates all the necessary features as well as the pie charts in a much better way than the previous layout. Wrote the introduction and other text for the index page and created the html page. Validated all pages across browsers before posting. Lana took the templates and populated them with the information that she had previously compiled. Ben, Lana, and I designed (Ben executed) pie charts to accompany the statistical information for each meeting's proceedings. Example: Of 29 total accepted papers at IMW 2001 (San Francisco, CA, USA November 1-2, 2001), 11 papers referenced our work (37.9% of the total accepted). The results of this work which spanned many weeks of compilation efforts by Lana form an excellent foundation (and huge beginning) which will be extended as new information is added. The pages are posted at:   http://moat.nlanr.net/Data-users/. [Maureen Curran, Lana Kennedy]

Work began on the redesign and restructuring of the PMA Web pages, the primary aim of which is to increase readability and accessibility. The PMA Web page committee - Jörg, Maureen, and Klaus (Lana is a member also) had an extended conference call and Jörg outlined the primary pages that will form the core of the new PMA Web pages. Developed sample pages for Jörg's review during conference call. Worked with Klaus on developing a special page design for use with data (a "Data page template"), such that the researcher is not confined to a specific size when displaying illustrated data (graphs and/or images), but ensuring that text remains readable (i.e., no long text lines). Redrafted the PMA navbar to correspond to the primary pages; began rethinking the design of the directory structure. [Maureen Curran, Klaus Mochalski, Jörg Micheel, Lana Kennedy]

We have defined the substructure below http://pma.nlanr.net/PMA/; and come up with a list of nine pages we would like to see there. Also achieved a clarification on content. Klaus has agreed to publish Leipzig-I and -II and help with the display of data collection/user activity, which I most appreciate. [Jörg Micheel]

Designed a one cell sized mini version (vs. the full sidebar one) of the navbar (just two links) for use with the data pages (where the complete sidebar navbar is not appropriate). [Maureen Curran, Klaus Mochalski]

As a result of working on the PMA pages, refined the design/layout of the MNA template, created a new template for regular MNA pages, changed the navbar to include a link to the Citings/Data-users pages and refined the text size and placement of the navbar. Will also be adapting the data page template for additional uses, including the AMP pages. [Maureen Curran]

An updated "Meet the ANR Team" page was posted,   http://mave.nlanr.net/MTM/. Also, the design for a Web page: Reference List of Collaborations, was completed and the page populated (will not be posted until the full information version is ready for posting). These pages are part of the Citings/Data users suite of pages. Ben created a "more info" button graphic. [Maureen Curran, Lana Kennedy]

Spent time this period adding coordinate information to the AMP database to make my automatic map generation scripts work for AMP international sites, and making several other tweaks to those scripts <for the new AMP Web front pages>. Also started making a map for the PMA sites <ror the new PMA Web front page>. [Ben Reesman]

Ongoing measurement and analysis, networked data, and infrastructure support:

"Writing Applications for Realtime Network Monitoring" was published in the Proceedings of the AUUG 2003 Conference: Open Standards, Open Source, Open Computing, Aug. - Sep. 2002, Sydney, Australia.  [Jörg Micheel]

Made a run of the system manager to distribute the HPC and international lists to all sites since the new startups went online, so all sites will be working with the new sites. [Bud Hale]

The amp server experienced a disk crash in the concatenated disk array; the failed disk was located and replaced. At the end of the period, the amp server was being restored from the volt disk array. The am_slave process that collects the data from the amplets was down on both servers. Due to the ease of recovery on a non-concatenated disk structure, decided to rebuild the disks as they are constructed on the volt server. Expect to restart am_slave on both servers shortly. [Bud Hale]

As this was the first time in quite a while that we have lost a disk (a couple of years), we discussed the fact that in the not too distant future we should upgrade the amp and volt servers. There has not been any major work on them for quite some time. A RAID controller would remove many hours of work for Bud when we lose a drive and new drives could hugely increase the capacity of the array and lengthen the time between backup runs. [Tony McGregor]

The investigation of available motherboards to replace the GigaByte Inc. system board continues. We have been using the current board for about a year; it has gone out of production. In surveying available replacement boards I have found that few manufacturers are continuing to produce boards supporting the Pentium III processors. Boards supporting Pentium IVs are much more available now. This may necessitate a switch of processors to the P-4. Plan to have a candidate board undergoing testing soon. [Bud Hale]

Replaced the PMA Server system disk, had been receiving countless messages during backup DUMP. The damaged file segments were the Library files, as well as others. [Jim Hale, Jörg Micheel]

Existing measurement sites maintenance and troubleshooting:

A total of 18 remote sites in the NAI infrastructure received attention during this period: Six have been resolved and the monitors are again collecting data. 12 were still being investigated, or pending site action, at the end of the period. (Outages are considered "open" until the monitor is again collecting data.)

AMP  -  11 problems:   4 resolved, 7 open
PMA  -    7 problems:   2 resolved, 5 open

~ AMP machines

amp-hutf (Helsinki U. of Tech.), analysis indicated a disk failure. A replacement system disk was shipped, is on site, and will be working soon. [Bud Hale]

amp-jhu (John Hopkins U. in Baltimore) is relocating their AMP monitor. However the network rework is not as extensive as former sites and the monitor should be back online sooner.  [Bud Hale]

Two sites, amp-mit (Mass. Inst. of Tech.) and amp-ncsu (North Carolina State U.) remain down for network changes. Monitors at both sites are to be relocated to new subnets. Continued communication to get them back online following the network reconfiguration at both sites. [Bud Hale]

Site amp-odu (Old Dominion U.) seems to be experiencing network problems. Both the AMP monitor and the PMA monitor at that site are currently unreachable. [Bud Hale]

amp-rpi (Rensselaer Poly. Inst.), a short outage occurred which proved to be an unintentional power removal. [Bud Hale]

The amp-surf (SURFNet in Holland) site had an outage. Able to work with site people to get it restored. The problem appears to have been a switch port that went bad. [Bud Hale]

Sites amp-uic (U. of Illinois at Chicago) and amp-ufl (U. of Florida at Gainesville) had outages. The monitor at UIC appears to have a power supply problem. A replacement power supply has been on site this week but the site people have not been able to do the installation. The monitor at U. of Florida, Gainesville appears to have failed system board. A replacement machine was shipped but site people have not been able to do the installation as yet. [Bud Hale]

The outage at amp-unin (UniNet in Thailand) required the shipment of a replacement monitor. That replacement was shipped, the new system disk was initialized, and the site finally came back online.  [Bud Hale]

Site amp-unm (U. of New Mexico) is currently indicating an outage and is being investigated. [Bud Hale]

~ PMA machines

We received notice that nai-p-aix was going to be relocated and would be receiving a new address. Worked with Alan Bishoff on the transition. After coordinating the elements of the conversion, we had a couple of communications relayed. Overall, the changes went smoothly and the data captures are occurring as they had previously. [Jim Hale]

Another site needing attention was the nai-apan site at UIC (U. of Illinois in Chicago). It needed a reboot to correct a hang-up. [Bud Hale]

Rework of the two PMA monitors for Florida sites nai-p-fla (U. of Fla. at Gainesville) and nai-p-apm (AMPATH at Miami) was completed, and both machines were shipped. The AMPATH machine has been installed in the equipment rack at AMPATH and is connected to the ethernet. Also it is reachable from moat.nlanr.net. The site technician has promised to schedule a short outage at the site soon to install the optical signal taps. That machine should be available to start collecting traces on the AMPATH network very soon. [Jim Hale, Bud Hale]

Other sites receiving attention at this time are nai-odu (Old Dominion U.), nai-txs (Rice U.) and MAX GigaPop in DC. [Bud Hale]

- 30 -