NLANR Measurement and Network Analysis Group (NLANR/MNA)
Passive Measurement and Analysis (PMA)Indianapolis (IPLS) router clamp installation (OC48) ~ Abilene - Three OC48 PMA machines were prepared and shipped to IPLS. PMA project lead, Jörg Micheel traveled to Indianapolis for the installation. At the Qwest POP on Kentucky Avenue, he worked with John Hicks (of TransPAC at Indiana University), Caroline Carver, and Grover Browning (both of the Abilene Team at Indiana) to bring the splitters into the links to Cleveland (CLEV), Kansas City (KSCY), and Atlanta (ATLA), (see http://www.ucaid.edu/abilene/assets/images/abilenebackbone.gif). The OC48c link to Chicago was considered too fragile to be touched. They completed the installation and connectivity for the two OC48mons looking at IPLS-CLEV and IPLS-KSCY. There were technical issues with the link to ATLA which made it unsuitable for any serious monitoring. Via emails with Steve Corbato (Internet2), it was determined that a further instrumentation of the IPLS router would not be feasible at this time, and that this level of installation would be all that could be achieved given the various constraints. Trace collection with the pair of OC48 monitors at Indianapolis (IPLS) was accomplished after the MOU with Internet2 was finalized. Work began on providing stratified sampling from these Indianapolis OC48c machines. The plan was to implement a systematic sampling scheme similar to all the other PMA monitors, perhaps slightly longer snapshots, which we ran until the links at IPLS were dismantled. A two-hour snapshot from 9am to 11am CDT was collected. The first of the Abilene OC48c data, Abilene-I, was published after the trace files were post-processed (anonymized, compressed, and copied to San Diego for publishing, the data mirrored onto the HPSS) and The same day of the two-hour snapshot from the OC48 monitors at Indianapolis (IPLS), another snapshot was cut, independently, for CAIDA folks in San Jose (OC48mon pilot), eight hours. All together, the traces collected during this one day comprised a total of 500GB of OC48c packet headers (our first at this speed). What a day for passive Internet measurements! The eight hours from San Jose is probably the largest IP header trace file in the history of passive measurements, with a trace volume of 336GB uncompressed. That's 5.6 billion IP packets, contiguous and CDMA synchronized. The sheer volume raises many questions about the usefulness. Additional Passive Measurement activities Work continued on the PMA sampling research. A summary Web page that displays data for sampling rates of 4, 8 and 16 was created. http://moat.nlanr.net/jfields/sampling.html Phase space analysis on traces: code was written to create a 3-D representation of the analysis of a phase space on the interarrival times of packets in a TSH file (PMA trace data). Having successfully done that, experimented with creating the phase space based on the packet length and adding additional information to the visualization using color to represent size classes. Obtained some interesting results, the meaning of which is not yet known. ~ Status of PMA sites and new deployments ~ Significant progress was made on getting the outstanding PMA monitors back online and collecting data. The Abilene IPLS-CLEV monitor died late in the period and was not revived. Work continued with the KSCY monitor collecting 5 minute samples. Several meetings were held with Jay Dombrowsky and Kevin Walsh (SDSC Network Operations Group) with regard to connecting passive monitors to the Abilene and CalREN connections at SDSC. Worked with Tom Hutton (also SDSC Net Ops) re assigning the IP addresses to the two monitors to be connected with nai-p-sdcr and nai-p-sdab (named for their SDSC connections to CalREN (sdcr) and to Abilene (sdab). The machine for the Abilene connection located at SDSC (nai-p-sda) was completed (after receipt of the latest Endace software), back on the connection and collecting traces. At the end of the period, discussions were continuing with NetOps regarding the connection to the CalREN network. When the latest software CD from Endace arrived and installation was successful on the SDSC Abilene machine, the Tel Aviv (TAU) and Front Range GigaPop (FRGP) monitors followed. Earlier in the period we worked with the Tel Aviv site people to plan the replacement of the PMA monitor there with one that can monitor their new POS connection. Planning to create a GigE PMA monitor at the Google GigaPop in the northwest. Received a GigE PMA monitor interface card from Endace. Working with the manager, Dan Magorian re the OC48 monitor at MAX site (Mid-Atlantic Crossroads, near Wash. D.C.). The possibility of a PMA monitor going to Korea was discussed, a very interesting possibility. Began preparing OC48 PMA monitor for SC2002 convention in Baltimore. Started construction of the machine by creating the system disk from the Endace software CD and configuring the hardware. An inventory of the NLANR PMA equipment and devices currently on hand was performed, so we know what is available for use in future PMA deployments. More detail on PMA activities during this period can be found in the monthly reports, available at: Active Measurement Project (AMP)IPMP - Additional changes and improvements were made to IPMP. We changed the path record structure to contain the TTL value in addition to the receiving IP address and timestamp. The information packets were also updated to supply an error bounds with each real time reference point. This is important for the work we wish to do with NTP on the AMPlets. The quality of the various NTP servers (that sites have advised) was investigated. One-way delay measurements using NTP were conducted with a few sites from sorcerer, the machine located at Waikato University (New Zealand). AARNet's (Australia) path to Waikato (across the Tasman Sea) produces very interesting graphs. http://moat.nlanr.net/mjl/graphs/aarn/ We plan to send a CDMA time receiver to one or two AMP sites to support the one-way delay using NTP activities; the first went to NYSERNet (New York). We do not have plans to equip many AMP machines with time receivers, as they are expensive. A couple will give us a starting point and a good reference with which to compare. Significant work was done on the draft of the IPMP Internet Draft (for IETF). We also conducted a bandwidth estimation exercise using IPMP echo packets on a network setup in the lab to see if an accurate estimation could be obtained. IPv6 - Discussions and planning for the development of an IPv6 mesh within AMP were held during the quarter. An email invitation was sent to all AMP site administrators, asking if they are interested in participating in the IPv6 trials (re assigning their AMPlet an IPv6 address). Many sites seem excited about the IPv6 tests, which is good. The email also invited sites interested in the NTP based one-way delay work to contact us. A number of AMP related projects were done in order to make IPv6 possible, including an IPv6 KAME implementation for IPMP, making ipmp_ping follow the KAME guidelines for writing a dual stacked application, and traceroute6 tasks. A script to go to each AMP site and fetch its rc.conf file, change a few settings, and then write it to the system manager was written. Each AMP site will have the traceroute request software working and IPv6 turned on by default (for autoconf purposes) when this change gets pushed out to the 4.6 sites. Chose to write the compressed files by default (a departure from the ASCII output AMP currently uses) because we will be capturing different data on the IPv6 mesh - namely one-way delays. Joe St Sauver (U. Oregon) and Bill Owens (NYSERNet) were contacted regarding AMP now being in a position to do IPv6 tests. They had originally inquired a few months back; both are still interested. NYSERNet was configured for IPv6. The CDMA time receiver there will allow accurate time delay tagging at that site, as well as providing a reference point for the one-way delay measurements. In September, we had a couple of machine IPv4 address changes as a consequence of people wanting to move their monitors to put them on an IPv6 network. We worked with SDSC Net Ops to acquire an IPv6 IP address for the amp-sdsc machine. Although IPv6 exists at SDSC, it was not easily connectable. However that connection was completed. Also a CDMA time receiver was connected to the SDSC amp monitor to provide for delay measurements. By mid October, we had eight sites with an IPv6 address. Also, in mid October, we started collecting IPv6 measurements; the scripts to collect the data and generate those pages were written and the traceroute grapher was ported (to understand the traces being collected). http://amp.nlanr.net/active/cgi-bin/v6_portal.cgi OS upgrade - A significant project during this period on the AMP system was the updating of the operating system (OS) on each of the AMP site monitors (AMPlets) from FreeBSD3.0 to FreeBSD4.6. After the upgrade the AMPlets will have OpenSSH 3.2, which has been a recent addition to the FreeBSD -stable branch. There were several reasons that the OS upgrade became a high priority. Site administrators were worried by the old system and it was becoming harder to get security fixes and support for the old OS. And, importantly, there are several things we want to do, but cannot, with 3.0, such as IPv6, one-way delay with IPMP and NTP, GigE, as well as other things. The upgrading of the AMPlets was almost a site by site exercise with each process being run interactively. Despite this, it went quite smoothly in general. This huge task included developing, then extensively trialing the process, followed by an extended period of actually upgrading the monitors, and troubleshooting the process. We did not lose a single AMP monitor during the upgrade. However, several machines (approx. 12) did require individual attention. These stragglers were brought up as their individual issues were discovered and corrected. We then updated the system manager to 4.6 (not a trivial process). A list of the seen transfer speeds for downloading a 145MB /usr tgz to the AMPlets from moat was created. http://moat.nlanr.net/mjl/upgradespeeds.html By the end of the reporting period, we had achieved almost 100% completion of the OS upgrade for the AMPlets. An infrastructure-wide system update of all FreeBSD4.6 AMP remote sites was performed after multiple changes were made to the master system disk and successful testing. By the end of October, most of the issues re the master (FreeBSD 4.6) system disk had been resolved, which will allow the preparation of many new machines for shipment to remote sites that have previously requested them. Many discussions were held regarding upgrading AMP and VOLT (the two data collection servers for AMP) to FreeBSD 4.6. At the same time, an update of the Apache server software is planned. Rather than just upgrade the machines, we plan to create a new (third) server and migrate everything to it. When it is operating (with a fresh 4.6 install), the existing servers can be upgraded. This upgrade strategy was chosen in part because AMP and VOLT are no longer well configured (e.g., their slash drive, at 2Gb, is too small). And, as a first step to the repackaging/reimplementation of AMP, a list of everything that is installed on the machine is being maintained. In keeping with the trend to physics parameters and inclusion of the remaining term in Ohm's Law, this temporary server will have the host name of "ohm.nlanr.net." Other significant AMP activities Major infrastructure work during the period included the mesh creation and writing code to create the ability to delete a mesh from the database. In addition to improving the current mesh management, these activities provide tools that will be useful as the number of meshes increases with campus deployment and the development of other meshes which will extend AMP. A new testing architecture is in the pre-planning stage. The challenge is to think up something relatively simple that accommodates what we need, with new types of tests being added. One interesting aspect is the need for some tests that are done only if others can't be (e.g., use ICMP if you can't use IPMP). It has been decided that we will stop collecting temperature and voltage sensor data as it was only available on a few machines and does not seem to have actually been useful in detecting system faults. ~ AMP remote sites, new deployments and status of sites ~ We deployed a new AMP monitor in Tokyo, Japan (amp-apantyo). The new AMP monitor that had been shipped to the Pacific Northwest GigaPop (Seattle, Wash.) was turned on. There had been some difficulty in doing so; continued work with the site people revealed it was a router problem in the GigaPop. That was resolved and it is now working and collecting data. Remote sites are functioning solidly except for some minor outages and some sites in transitional, or stopped, states. At the end of October, work was being done on 13 AMP sites with issues/problems (this is out of more than 120 AMPlets). More detail on AMP activities during this period can be found in the monthly reports, available at: Additional Performance Measurement and Analysis ActivitiesWith the latest installations to HPWREN on San Clemente Island (about 72 miles off coast) and the King Stormwater Bridge near the Salton Sea, HPWREN now literally stretches from the desert, through the mountains, via the beach, and ways into the ocean. ROADNet's link to the R/V Revelle goes out into the high seas. An updated topographic image of HPWREN is available at http://hpwren.ucsd.edu/topo.html. HPWREN measurement and analysis activities this period included radio signal event detection, network event reports and rebuilding the PAM machines to do different measurements. (The HPWREN measurement machines are configured for both passive and active measurements, therefore they are called PAM machines.) More detail on wireless performance measurement activities during this period can be found in the monthly reports, available at:
Documentation, Networked Data, and ToolsAs mentioned previously in this report, the Abilene I OC48c data was published. (http://pma.nlanr.net/Traces/long/ipls1.html) Planning and discussion sessions were held re needs and development of the PMA Web pages. A new distribution of the central data collector software (for AMP) was made and added to the Web page. Also the license and NSF acknowledgment information was added to the Web page (it has always been in the software itself). A problem with the script that updates the AMP splash pages was fixed. A favicon for the AMP Web pages was created (a little meter that looks like a sun). [Tony McGregor] A data banner with the new acknowledgment information (with both cooperative agreement numbers and begin dates) was created for use on the Web pages, especially those with data and/or tools. Work began on the revising MNA front page, including rewriting the acknowledgement section. Also, numerous strategy and planning sessions were held re options for easing the process of making changes to the Web pages (SSIs, tunneling, Perl scripts), using templates and their structure and the templatization of the MNA files. A script was written to perform a task similar to Server Side Includes as we determined this was the cleanest way to obtain the functionality desired. Preliminary discussions, planning, and development of ideas regarding a new Web page highlighting our international collaborations were held. Some preliminary work was performed (directory structure, preparation of photographs, creation of the index page, a header banner, and other artwork). A world map highlighting NLANR/MNA international collaborations for both AMP and PMA was created. This new world map impressively displays the depth of our collaborative efforts; and it will be used for (slide) presentations. Ben Reesman accepted the second student researcher position (will begin in early November). He will work on the Cichlid 3-D visualization system, the development of which has been dormant since Jeff Brown went into the UCSD Computer Science Ph.D. program. More detail on activities during this period can be found in the monthly reports, available at:
Papers, Publications, Presentations, and Conference/Meeting Participation~ ~ ~ Presentations and Meetings Thailand and China trip: Ronn Ritke gave an invited presentation to the UniNet Group in Thailand. This was followed by three meetings and presentations in three cities in China. During the trip, multiple presentations, both formal and informal, were given regarding the NLANR/MNA activities, with special emphasis on collaboration and possible placement of AMP and/or PMA monitors. A report on this extended trip was submitted to NSF with an overview of possible resulting collaborations. Australian UNIX Users Group Annual Winter Conference in Melbourne AUUG2002)
Talk on NLANR/MNA research work while visiting AT&T Research Florham Park; it was well received. [Jörg Micheel] Boulder Joint Techs meeting
IST SCAMPI meeting in Prague (http://www.ist-scampi.org/)
10GigE Workshop at SDSC
Univ. of Waikato Computer Science Dept.
~ ~ ~ Papers and Publications Significant progress was made on the next issue of the Network Analysis Times (the focus is on our international collaborations). The current issue was distributed at the Colorado Joint Techs meeting and throughout SDSC. An article on the recent PRAGMA2 Meeting presentation in Korea was completed and submitted for publication in the Korea Institute of Science and Technology Information (KISTI) Supercomputing Newsletter. Work continued on the IPMP draft (in re IETF Internet Draft). There is a possible paper with the people in Bell Labs that we will submit to a New York local conference on networking if it progresses. The NLANR/MNA STI award press release was written, and after review by NSF, released. > More detail on activities during this period can be found in the monthly reports, available at:
Collaborations and Student InvolvementIn addition to our on-going collaborations, we have begun or are exploring a number of new ones. Our collaborative efforts include continuing to work closely with Internet2 and working to develop projects with the other NLANR groups (Front Range GigaPop [FRGP]). We will be hosting PAM2003 (to be held in April) and efforts began in several areas for this, including beginning the Web site for PAM2003. Briefly summarized here are the activities involving these collaborations during this period.
~ ~ ~ Students New student researcher Chris Gross began work as the new PMA student the last week of October. Chris is a first year student at UCSD. Ben Reesman accepted the second student researcher position (will begin in early November). He will work on the Cichlid 3-D Visualization System, the development of which has been dormant since Jeff Brown went into the UCSD Computer Science Ph.D. program. Ben is also a first year student at UCSD. We are looking forward to working with these new students over a very productive number of years (hopefully four!). More detail on activities during this period can be found in the monthly reports, available at:
Other ActivitiesAMP project leader, Tony McGregor traveled to San Diego in mid-July, staying for a week. He came for the interviews of the final candidates for the AMP FTE (Software Engineer) position, as well as to meet with the rest of the team and to facilitate and prepare for the AMP upgrades. PMA project lead, Jörg Micheel was in San Diego for a short, but fruitful (one day) visit during which some valuable discussions took place, regarding various opportunities and options to proceed with the router clamp and future collaborations with Internet2. This period had a quite a bit of activity with regard to bringing on additional staff, primarily regarding the AMP FTE (full-time employee) Software Engineer opening. We also posted the AMP/PMA Assistant Systems Administrator (as an FTE) and openings for student researchers. As part of the interview and selection process for the AMP Software Engineer opening, a coding skills test was administered to each of the candidates (Perl and C problems of varying difficulty). The programming test turned out to be a very important and useful part of the interview process and we've decided to continue to use that approach for all positions, including the student positions. A large amount of time and effort was spent by several members of the team on staffing activities, in particular for the AMP Software Engineer position (which is in regard to the reimplementation of the AMP software). Have completed the hiring process for the new students and part of the AMP/PMA FTE. Cooper Nelson started half time with MNA in mid-September (having previously worked for HPWREN in combination with Scripps Institute of Oceanography), he will be assuming a technical support role for both the PMA and AMP projects. David Cheney, having graduated from UCSD in June 2002, was rehired on a part-time basis to provide assistance to the new student(s) and work with Maureen on various Web projects (writing the back-end scripts), including continuing the work on creating a system and related scripts to ease changes to the MNA Web pages (for now and in the future). Two new students joined the staff, Chris Gross and Ben Reesman; both are first year UCSD students. We had an outstanding group of student applicants. In addition to the superb three finalists, there were a number of other students who stood out for various reasons and we will be keeping (and sharing with HPWREN/ROADNet) their resumes, etc. Shared some of the best candidates with an SIO research group associated with HPWREN/ROADNet (they chose one of the students). We also brought one of the students to the attention of Ian Graham of WAND (Waikato). More detail on activities during this period can be found in the monthly reports, available at: - 30 - Back to the Top |
