NLANR Measurement and Network Analysis Group (NLANR/MNA)
Passive Measurement and Analysis (PMA)OC192 Measurement and Analysis - In November, we successfully acquired and implemented a platform for an OC192 monitor supporting an Endace Dag6 card. NLANR/MNA now has an OC192 monitor platform in place, connected to the DTF OC192 link to Chicago (located at SDSC). Unfortunately, there were no systems connected to the link, therefore no traces could be taken nor tests run. OC192MON activities continued making excellent progress in early 2003. PMA project leader Jörg Micheel went to Paris in January, where he worked with Christophe Diot of Sprint ATL on the field installation of the first OC192MON, tapping into the link between Paris and London. The work was successful, we managed to collect a few hours of trace files, with GPS synchronization. Link loads were between 420,000 and about 510,000 packets per second, and around 2.5 GBits/sec, equivalent of a full OC48c. This is an average 600 Bytes/packet. Work is continuing with this pilot in Paris; we hope to get SDSC involved very soon in order to have a better testbed with access to test gear. Development of new metrics - New PMA student, Chris Gross, spent time learning about the PMA trace files, specifically the tsh format. He and Jörg discussed plans for how to proceed with some interesting new analysis work which will eventually involve permanent monitoring and a low-volume data stream to be displayed as RRD graphs on the pma data server. The plan is to also develop real-time analysis for PMA. Chris worked on developing a set of passive measurement metrics. Stats analysis of pma.nlanr.net (Web log) - The trace data download log files have been collected and integrated with existing files to create an historical report. A tool to filter httpd and ftpd logs into a common format and merge into a single sorted file was scripted. We are using awstats to generate reports of trace download data; work continues on the interface. ~ Status of PMA sites and new deployments ~Two PMA machines were shipped and are expected to be installed shortly at the Front Range GigaPop and Tel Aviv University. Both machines are replacements due to upgraded connections at the sites. The OC48 monitor for the Mid-Atlantic Crossroads GigaPop was shipped. Bud Hale is working with Dan Magorian at the Mid-Atlantic Exchange GigaPop (MAX) who has the new OC48 monitor in place at the site in downtown DC. (They were experiencing some ethernet problems in the facility, but are expected to have it back online beginning of the next period.) A new passive measurement monitor is being prepared for installation at StarLight. Testing also began on the machine that is in preparation for the connection to APAN at Kisti in Korea; we are awaiting configuration data. We communicated with Brent Sweeny at Indianapolis regarding the OC12MON and the three OC48MONs there, seeking to place a new set of monitors at the new Qwest POP. Another development on the PMA scene is that NCAR (National Center for Atmospheric Research) is requesting a GigE monitor. There are a few questions that need to be answered but that machine should be taking shape early in the next period. More detail on these activities can be found in the monthly reports for this reporting period, available at: Active Measurement Project (AMP)IPMP - A major accomplishment was the taking and analyzing of measurements with IPMP on the CRCnet (Connecting Remote Communities Network) wireless network, with an aim of measuring the bottleneck. The CRCnet wireless network is located in a rural area west of Hamilton, NZ (near the Univ. of Waikato). In order to achieve this, a Linux implementation of IPMP was created so that it could be deployed on CRCnet. The dispersion graph (http://voodoo.cs.waikato.ac.nz/~mjl12/dispersion.png) is very interesting - there is a big gap between 3 and 5 milliseconds in the middle of the network around WTU that is caused by contention on the MCG <-> HSK and HSK <-> WTU half duplex links. Because we are sending two packets, there will be contention at WTU when the first packet is turned around while the second packet is still in transit. The IPMP Internet draft was modified to reflect the current use of the protocol and submitted to IETF; it is posted on the Internet-drafts directory. http://www.ietf.org/internet-drafts/draft-mcgregor-ipmp-01.txt Development and modification work on IPMP included substantial changes to the format of the echo packet, the addition of an IPv6 path record structure, flow-on changes to the information packet, and the addition of a real time reference point structure. The timestamp is now 48 bits. We kept the fraction portion of the timestamp at 32bits, a resolution of 200 picoseconds, as with the rate packets are serializing these days we need that resolution. We now specify that the timestamp must be taken at the time the last bit of the packet is received. Additional work was done regarding IPMP protocol changes to do bandwidth estimation on half duplex networks (the CRCnet wireless network). The crux of this work is to give the protocol the ability to measure in just one direction and return measurement results out of band. Dispersion Analysis of a Packet Train: http://voodoo.cs.waikato.ac.nz/~mjl12/. IPv6 - We began analysis of IPv6 data over the six week period during which we have been collecting data. Graphs were generated which showed how the distribution of IPv6 RTTs changed over time. By looking for sudden shifts, several interesting cases were found. Code was written for the traceroute grapher in order to position it for use with IPv6 / IPv4 comparisons. Several modifications were made to the code initially written by James Spooner.
The general architecture of this traceroute utility is a tool that can conduct multiple traceroutes simultaneously so that if it gets stuck on one hop (a hop that does not generate ICMP destination unreachable messages), it can traceroute other paths until that hop times out. The time between each probe can be configured so the amount of bytes per second this tool generates is configurable. Also, AMP IPv6:IPv4 comparisons can now be generated upon request vs. the previous generation of static pages every 15 minutes. An IPv6 address list to trace the forward path to all the IPv6 prefixes has been started. The IPv6-world trace has 860 addresses (1141 prefixes). Matched each address, where possible, with a prefix and have 645 addresses matched to a prefix. Unfortunately, 311 prefixes have one or more addresses, out of a total of 1141 (or about 27%). We continued to work closely with Joe St Sauver (U. Oregon) and Bill Owens (NYSERnet) regarding the IPv6 project. Discussions have included the possibility of developing plans to try and map the MTUs seen between select AMP paths. To do so, we will need to install GigE cards that have an MTU of approximately 9180 bytes, and get sites to be jumbo clean all the way to Abilene. Bill and Joe can do this in a relatively short time, but other sites might be a problem - if they are even interested. The choice of GigE cards is important because apparently many cards currently in production fall (way) short of the needed MTU of approximately 9180 bytes. The AARNet machines have been shipped with the 3Com 996B-T cards, which are likely 4470 byte MTU cards. Amp-utah has had IPv6 tests added to it per their tech's request. Email discussions ensued because they have around 87% loss for their IPv6 connections to other AMP sites. We will be monitoring that site to assist in finding and fixing the problem. http://watt.nlanr.net/active/cgi-bin/v6_sitesummary.cgi?amplet=amp-utah&date=103.2.14 New testing architecture - We have reached the point where it is difficult to manage the different tests that we perform on the AMP mesh with the ad hoc approach that we currently have. Therefore, we are developing a new testing architecture for AMP which will allow us to add new tests and control which monitor does which tests from the database. The new architecture will add support for a more flexible range of tests, meshes, and test schedules. This is important because we want to extend AMP into a range of new tests, including the IPv6, and regular bandwidth tests, etc. Not all sites will want, or be able, to support all tests. Therefore we need a robust way to manage which site does what. A proposal (guidelines) for how we can extend the AMP testing architecture to accommodate these needs has been developed, and is being used as a guide to implementation. The Web page and database to create new tests and most of the one to assign monitors to tests are complete. An important detail not included in the original design document is how to handle defaults. It is important to be able to enable, disable, and set to the default the ability to do a test. This is mostly for implementation reasons, but may also be useful, for example if we upgrade a test and this requires more abilities from the monitor. An investigation of tools for bandwidth estimation has been performed. Pathchar and nettimer were insufficient for our purposes; neither works very well for paths containing more than a few hops. ~ Other significant AMP activities ~Work restarted on event detection; primarily on establishing the status of the project and documenting more of the code. Investigation of why it is apparently not detecting an event that it should, was started. Currently the algorithm is complicated, but so far nothing simple works. Infrastructure work included the creation of an international mesh with all the non-US sites in it as well as sdsc, psc, and bu. Over time, we would like to migrate the international sites out of the main HPC mesh. A visualization project for AMP to dynamically render sites and serve as a basis for rendering real time performance of AMP sites was begun and significant progress made. A new "day in the life" dataset for AMP was created and posted. The AMP system disk was modified per a decision (in December) to make it such that it has a network card driver installed for the Intel NIC, RealTek NIC, and the 3COM 3C996-T gigabit network card. This system disk version will be shipped with all AMP monitors in the future, including the monitors for Australia. We also resolved an airflow issue with the 1 RU AMP chassis in consultation with the vendor. The decision is to create a vent in the cover at the vendor location by opening a hold in the cover with a 2 inch metal punch and cover it with screening. We have decided that we should change from using mSQL (which is proprietary) to postgresql for the AMP support; there is no timetable for this change as of yet. ~ AMP remote sites: new deployments and status of sites ~New AMP monitors were prepared and shipped this period to the following sites: U. of Idaho, Stanford Linear Accelerator (IPv6 monitor), ELTENET in Hungary, HEANet in Ireland, TCTnet in Finland, UNINet in Thailand, U. of Wisconsin Network, and Great Plains Network GigaPop. Six additional monitors were sent to Australia for the AARNet mesh. With minimal exceptions AMP sites are collecting and transferring very solid. AMP servers are running well. Archiving of the VOLT server disks was successfully performed (due to previous problems, the process was carefully monitored). Even though we have not migrated to the recommended HPSS interface, it appears our archiving issues are mostly resolved. Late in the period, the new AMP server, ohm, was completed and installed in the machine room at SDSC. More detail on these activities can be found in the monthly reports for this reporting period, available at: Additional Performance Measurement and Analysis ActivitiesA new version of the HPWREN topology map was created (http://hpwren.ucsd.edu/topo.html). Several of the measurement and analysis Web pages were improved and updated, including the UPS pages. The Tsunami event detection script was completed and documentation added. There is still work left to do for automatic user notification of events. More progress was made on the network utilization script (for the network reports page and email). The anemometer data graphing tool originally created in the last quarter, was improved. (Note that this tool has since become unusable and as yet, cannot be revived.) More detail on these activities can be found in the monthly reports for this reporting period, available at: Documentation, Networked Data, and ToolsIn conjunction with the major infrastructure work during the previous period regarding mesh creation, a Web page was created to list the AMP sites that are a members of a mesh. In addition, information regarding to which meshes a site belongs was added to the site details pages. This is to help in understanding the mesh system that we have created. xxpointer? All of the yearly data on both the amp and volt servers needed to be regenerated due to a bug in dogenyearly that was causing problems, after which the nightly processing was restarted. (The bug was found and corrected.) The MTM page (Meet the Team) was moved from the moat server to mave (which earlier in the period had been set up to be an additional Web server); it was also updated to reflect our staffing changes. Extensive groundwork and study is being done re the current state of the Cichlid 3-D visualization system, with a view to both reimplementing it, as well as developing new functionality. An aspect of this was a project to research the current state of the art in the rendering of multidimensional datasets. The new Cichlid/ visualizations student, Ben Reesman, performed this research and presented his findings (internally). Ben is in contact with Jeff Brown, the creator and original developer of Cichlid. A new project was begun to manage (and create) dynamic front page content. It has been extended to serve as a "rolling reports" system as well, whereby as items are updated for use on the dynamic front page, all information from the weekly reports will be archived and retrievable from a database. From the database, the archive page can be generated and the monthly reports can also be created in a draft form, in addition to the generation of the dynamic front page content. It is currently in the design development stage. More detail on these activities can be found in the monthly reports for this reporting period, available at: Papers, Publications, Presentations, and Conference/Meeting Participation~ Presentations and Meetings ~SC2002
APAN/PRAGMA Meeting
IMW (Marseilles, France)
10GigE conference (at SDSC)
AMPATH meeting in Florida
Fall I2 Member Meeting.
Matthew Luckie gave a seminar to the computer science department at Waikato. He also gave a presentation on IPMP at the Waikato Univ. student meeting, resulting in much discussion. Tony McGregor went to Auckland to attend a seminar and talk to Cristophe Diot from Sprint about the measurement work he does on Sprint's network. Chris prepared and gave a tutorial on PHP to a group which included the MNA staff as well as some members of the SDSC staff who work on Web pages (very well received). With Ian Graham, Jörg has made a preliminary proposal to have the Internet Measurement Conference 2004 (it will be upgraded from IMW to IMC next year) in Hamilton. ~ Papers and Publications ~ A new issue of the Network Analysis Times (NATimes 3.2) was published (with an international collaboration theme). Print copies were distributed widely, including to SC2002 in Baltimore, the IETF November meeting in Atlanta, the NSF PIs January meeting in Virginia, and throughout SDSC. The look of this issue greatly benefited from the work of Ben, the new student whose area is images and the development of Cichlid. http://moat.nlanr.net/NATimes/NAT.3.2.pdf "NLANR's Active Measurement Program: Network Knowledge Leads to Practical Payoffs" was published in Online, Vol. 7 (3), February 5, 2003. Online is the SDSC/NPACI biweekly newsletter. http://www.npaci.edu/online/v7.3/nlanr.amp.html Matthew registered and submitted a paper to Sigcomm 2003: "Using IPMP to Identify a Bottleneck Router" which will discuss some of the bandwidth estimation work with IPMP on the WAND CRCnet wireless network. http://voodoo.cs.waikato.ac.nz/~mjl12/sigcomm2003.pdf Jörg wrote and submitted to PAM2003 a paper on performance assessment of passive Internet monitors during the weekend of December 6 and the following Monday (NZ time), with coauthors from Endace. Todd Hansen wrote an abstract for PAM2003 submission for a paper about applying active measurement techniques to mobile satellite links. Klaus Mochalski (Univ of Leipzig) has drafted a paper on delay measurements; it is going to be a joint paper with PMA. More detail on these activities can be found in the monthly reports for this reporting period, available at: Collaborations and Student InvolvementBill Owens of NYSERnet, one of the people who is working closely with us on (and encouraged) the AMP IPv6 project, sent an email to the Internet2 IPv6 Working Group with a pointer to the IPv6 Web pages. From comments, it appears that the pages were warmly received. Work on displaying the http/ftp download volume of PMA data has revealed that overall, the old statement holds true that the amount of download activity by users matches the amount of data collected, in the 3-5GB per day range. This is good news, as it reflects that, on average, all our data is useful, is being looked at, and studied. A delegation from the National Center for High Performance Computing of Taiwan visited SDSC in December. Taiwan will soon be upgrading their network infrastructure to 40 Gigabits. Interest was expressed in hosting 1-2 AMP monitors and in developing some local measurement capabilities. We met with them and gave them a tour of SDSC, including the machine room where our local machines are located. Ian Graham from Waikato University and Dave Miller from Endace arrived from New Zealand and participated in the configuration and installation of the OC192 monitor (which has a Dag6 card). Yan Chen from UCP asked for two months of AMP data. Tony sent the first month and is awaiting the request for the second month. They have written one paper about AMP data already. "On the Stability of Network Distance Estimation", Proceeding of ACM SIGMETRICS Practical Aspects of Performance Analysis Workshop (PAPA 2002), which was also selected to be published in ACM SIGMETRICS Performance Evaluation Review (PER), September issue, 2002. While attending the European 10GigE monitoring platform meeting, IST SCAMPI, Jörg met with a number of new folks with whom he was unfamiliar. He found it was quite interesting to see what everyone is doing, and some of the discussion reminded him of things that he had seen and experienced five years ago, e.g., discussions about file formats and the like. He presented our approach to passive monitoring, and it was apparently very well received by most of the folks in attendance. The perception is that only with the Dag technology are they likely to get to where they want to be. We need to explore how we want to create a research collaboration with these folks, there is quite some room and opportunity. Through the strong concerted efforts of several members of our team, significant progress was made towards our hosting of the PAM2003 Passive and Active Measurement Workshop: registration was opened, submitted papers were assigned, reviewed, accepted (or rejected), and authors notified, and the PAM2003 Web pages continued to be developed. The quality of papers submitted for PAM2003 was very strong this year and will likely result in an excellent workshop. In addition, a number of students submitted papers and we will give some support to student authors to assist in their participating. We gave Jeff Olenchek from uwn access to the on-demand throughput tests (AMP); later in the period we wrote to let him know it was fixed after the amplet upgrade. We are working with Manhee Lee (KISTI/Korea), providing some support and information regarding bringing KREONET's new AMP machines into the AMP mesh. We also sent him 20 printed copies of the current NATimes to distribute at a special event of his. (He has an article on KREONET in the issue, and is part of the front page image collage.) We are also working with the following folks:
The Pipefitters Project (Russ Hobby and Eric Boyd)is interested in using AMP data for their E2E project. Matt Zekauskas is looking into how to do authentication for active tests. Once he has that problem solved we might revisit the idea of doing some cross active mesh measurements between AMP and Surveyor. Matt will ship the SC2002 Surveyor to SDSC, so that we will have one at SDSC. Dr. Xiao Chen at UCLA (statistician) is interested in working with us on research for a paper about our current sampling research. Ronn had many discussions with various groups from all over about potential collaborations:
Jörg helped Cedric Westphal of Nokia IPRG (former Ipsilon) with access to trace data to compute distributions of IP address prefixes. Got him in contact with Christophe Diot of Sprint ATL's IPMON group, in the hope he can be helped there. Jörg met with Dr Hank Dardy and Basil Decina of Naval Research Labs in the Washington DC area. He had the opportunity to look at all the interesting projects they are pursuing, including the new ViPR video conferencing system from Marconi, which is, at last, an excellent quality, easy-to-use system. Downside is that this lab is pursuing ATM deployment, even at OC48c and OC192c link rates. While in New York, Jörg had meeting with Carter Bullard from QoSient, who is very active in the IETF IPFIX working group, and the author of a sophisticated commercial flow engine, freeware parts of which are available as Argus. While in the bay area, he called Bruce Penrod of EndRun Technologies and learned that they are just about getting ready to have their Japanese version of the CDMA antenna tested, which would provide another opportunity to get the measurements going that John Hicks started almost a year ago, across the TransPAC. Jim Williams from TranPAC discussed some of the near future plans for TransPAC with Ronn. John Hicks gave a talk about his measurement activities and mentioned the DAG cards and ENDACE. Jörg has been instrumental in the development of the TransPAC passive measurements. They may want to have some additional active measurement on TransPAC. Bud is working on issues related to IBM disk reliability. He contacted the Center for Magnetic Recording Research (CMRR) on campus (UCSD). CMRR conducts studies and research into many areas of magnetic recording. He discussed our concerns with a major researcher there: Gordon Hughes. CMRR has conducted major hard disk reliability studies, some in collaboration with IBM. They are currently involved in some collaboration with IBM and Gordon is working with a IBM storage division vice president: Roger Hoyt. Gordon is very interested in our issues and has consented to take some steps toward opening a study with IBM related to the use we are making of the IBM disks. (This activity was brought about by the high number of disk failures that both AMP and HPWREN were experiencing.) Ronn had meetings with Rozeanne Steckler (SDSC Assoc. Director, Education) about the NSF-Thailand workshop. He also discussed the meeting with Bill Chang at NSF in Virginia. While discussing our current activities and some future plans with our Program Manager from NSF, Greg Monaco, at the Fall I2 Member Meeting, Alan Blatecky (also a Program Manager at NSF) joined the conversation between Ronn and Greg. Alan suggested doing some PR on the international collaboration model for AMP. The model consists of a country hosting an NLANR AMP machine and one or more groups from that country working to create a local measurement infrastructure (mesh) and making that data publicly available. The local measurement infrastructure data adds to the data that can be used to detect and isolate network problems. In the future, Ronn plans to write a paper that might include both the measurement infrastructure meshes and the "people meshes" that help make this model possible. Che-nan Yang (Director, Taiwan) and his deputy director Dr Tsai met with Ronn and expressed interest in the AMP project. Taiwan has some ambitious network upgrade plans and would like to include network measurement as an integral part of that effort. They expressed interest in hosting 2 NLANR AMP machines - one at their Northern NOC and one at their Southern NOC. This group from Taiwan may be the 3rd country to follow the new collaboration model by hosting an NLANR monitor and deploying their own local AMP mesh. They have already started experimenting with the current AMP software. (The first two groups are AARNet [Australia] and KREONET/KISTI [Korea].) At the APAN/PRAGMA meeting, Man Hee Lee (KREONET) gave a talk about an APAN-wide AMP project. This will provide us with the opportunity to leverage the existing NLANR AMPs in APAN countries to help create a larger APAN active measurement infrastructure in collaboration with KREONET/KISTI. We continue to work with Peter Arzberger and Teri Simas of PRAGMA, this quarter helping with the planning for the PRAGMA meeting during this period. Dr. Wanchai Rivepiboon from Chulalongkorn University Thailand asked to continue the dialog on measurements. Jörg has arranged for an OC3MON to be placed on the Internet2 connection to Thailand. Dr. Rivepiboon is interested in hosting an active monitor and asked Ronn to give a talk to a local measurement workshop. Ronn met with Director Yan Baoping and Kai Nan from Beijing. Soon they hope to complete the request to the government of China to get approval to host an NLANR AMP monitor. Another site in Beijing expressed interest in hosting an AMP, if possible. We plan to continue working with Director Yan. Tony talked to Cristophe Diot from Sprint about the measurement work he does on Sprint's network when he gave a talk at the Univ. of Aukland. It was an interesting talk and he suggested that if we needed Sprint data we can use it at their labs, which is an offer to keep in mind. Amongst other things the talk lead Matthew and Tony to a discussion of routers reordering packets which is relevant to the new draft of IPMP (which was submitted to IETF during the reporting period). Jörg also met with Christophe Diot from Sprint ATL at the Univ. of Auckland; Diot included the first analysis data from the pilot OC192MON in his presentation. They had very good talks on ideas on future monitoring work. Christophe shared his plans for the medium and longer term future. The new "day in the life" dataset was sent to Mark Crovell from Boston Uni and Teng Fei (a grad student at the Univ. of Massachusetts). We are working with Joe Abley about the AMP machine at the Auckland Peering Exchange. Finally the machine is live and ready for the AMP software to be loaded onto it. Jörg has been in contact with the following folks during this period:
We are working with Kevin Walsh and others in the SDSC ENS group re several things including a possible tutorial/demo and a testbed proposal for NSF and DTF plans and the need for AMPs at the DTF/ETF sites. Jan-B Gao (Univ. Florida) would like to use our data for modeling research. More detail on these activities can be found in the monthly reports for this reporting period, available at: Other ActivitiesAMP Project Lead, Tony McGregor, began his increased time (to 80%) during this reporting period (first week of February 2003). Tony and Matthew travelled to SDSC for a week in early December. There were many meetings with various people in San Diego, as well as a strategy meeting on some possible NLANR/MNA next steps. Jim Hale became a full-time UCSD/SDSC employee (50% NLANR/MNA, 50% HPWREN), as the AMP/PMA assistant systems administrator. Logistic and administrative aspects of the 10GigE cards were pursued this quarter. A significant amount of time was spent on staffing activities by some members of the group (the PAII sysadmin position, Dave's post graduation job description, follow-up on the student positions). More detail on these activities can be found in the monthly reports for this reporting period, available at: - 30 - Back to the Top |
