NLANR/MNA Activities Summary - September and October 2002

• Highlights
• Network Analysis Activities (including Measurement Infrastructure Development and Management)
  • Passive Measurement and Analysis (PMA)
  • Active Measurement Project (AMP)
  • Additional Performance Measurement and Analysis Activities
• Networked Data, Documentation, Tools, and Result Presentation
  • Documentation, Networked Data, and Tools
  • Papers, Publications, Presentations, and Conference/Meeting Participation
• Collaborations and Student Involvement
• Other Activities

Highlights

• Published the first of the Abilene OC48c data, Abilene-I. started working on Abilene-II. Made significant progress getting the outstanding PMA monitors back on-line and collecting data.   
• Planning to create a GigE PMA monitor at the Google GigaPop in the northwest.  Received a GigE PMA monitor interface card from Endace. Working with the manager, Dan Magorian  re the OC48 monitor at MAX site (Mid-Atlantic Crossroads, near Wash. D.C.).
• Started collecting IPv6 measurements.
• Spent some time thinking about a new testing architecture for AMP.  The challenge is to think up something relatively simple that accommodates what we need, with new types of test being added.  
• Major infrastructure work included the mesh creation and deletion code.  Have achieved almost 100% completion of the OS upgrade for the AMP mesh.  
• Made extensive efforts towards filling our staff openings:  two open FTE (full time) positions (AMP Software Engineer and AMP/PMA Assistant Systems Administrator) and two student researcher openings.  Have completed the hiring process for the new students and part of the AMP/PMA FTE.  
• In addition to our current collaborations, we have begun or are exploring nearly a dozen new ones.  These include continuing to work closely with Internet2 and working to develop projects with the other NLANR groups (Front Range GigaPop [FRGP]).  We will be hosting PAM2003 (to be held in April) and efforts began in several areas for this.  We participated in and presented at several meetings and conferences world-wide.    

Passive Measurement and Analysis (PMA)

• Published the Abilene OC48c data. Finished with generating all of the standard trace graphs (packet, bandwidth, flows, volumes etc) and the .html pages. Design of the front page for the data set was problematic because of the acknowledgments,  history, and context. Also did run another check, published the DUCK log files and a set of md5 checksums for the files. [Jörg Micheel]
• Have started working on Abilene-II, which is a collection of 5 minute stratified random samples 8 times a day. Almost identical to the rest of the PMA monitors, except that we use different tools and also publish 5 minute samples, which amounts to a good 1GB of data each time. Collection got stuck several times in early September, in part due to the outages of medusa at SDSC.  [Jörg Micheel]
• The PMA machine for the Abilene connection located at SDSC (nai-p-sda) has been completed; back on the connection and can collect traces.  It was necessary to install new software from Endace. (Required obtaining the latest version of the Endace Dag software; see below)  [Bud Hale and Jim Hale]
• When the latest software CD from Endace arrived and installation was successful on the SDSC Abilene machine, the Tel Aviv (TAU) and Front Range GigaPop (FRGP) monitors followed.   [Bud Hale, Jim Hale]
• Earlier in the period worked with the Tel Aviv site people to plan the replacement of the PMA monitor there with one that can monitor their new POS connection. Most issues have been resolved and that is going forward.   [Bud Hale]
• Planning to create a GigE PMA monitor at the Google Pop in the northwest.  We initiated purchase of a dual-port Gigabit Ethernet Dag card via Endace for the Google monitor, and received a 50% rebate on the list price.  [Jörg Micheel]
• Received a GigE PMA monitor interface card from Endace. Worked on plans to assemble a machine for the interface and install the Endace software with it. [Jim Hale]
• PMA updating and planning/strategy sessions, while Jörg was in town.  [Jörg Micheel and Hans-Werner Braun; Jörg Micheel and Bud Hale]
• Continued work on the PMA sampling research.  Updated the page a few times; set up cron jobs to keep it updated every day.   [Justin Fields]
• Improved the scripts that contribute to the sampling analysis Web page. Added a small summary at the top of the page, and made a few other cosmetic changes.  Also started working on a new type of graph for the page, which will plot accuracy of a sample versus bytes per second. [Justin Fields]
• Created a summary Web page that displays data for sampling rates of 4, 8 and 16.  http://moat.nlanr.net/~jfields/sampling.html   [Justin Fields]
• Continuing discussions with NetOps at SDSC on the connection to the CalREN network.  [Bud Hale]
• Possibility of a PMA monitor going to Korea was discussed, a very interesting possibility.  [Jörg Micheel]
• Began preparing OC48 PMA monitor for SC2002 convention in Baltimore. Started construction of the machine by creating the system disk from the Endace software CD and configuring the hardware.  [Jim Hale and Bud Hale]
• PMA existing sites, maintenance and troubleshooting:  
  • The Abilene IPLS-CLEV monitor died and is unreachable, continue to work with the KSCY monitor collecting 5 minute samples. Have implemented a script to mirror those traces to the HPSS as well, we had run out of disk space the other week. Am not reviving CLEV at this point. [Jörg Micheel]
  • Ohio State - OC3 monitor had been hung for a while and it was rebooted; back up and collecting traces. [Bud Hale]
  • Texas GigaPop - more site help working now (provided with a good contact) and will be getting some power signal level measurements done soon.   [Bud Hale]
  • U. of Fla. at Gainesville - PMA monitor is now fully operational with the Dag3.5 cards (disk and Ethernet problems). Delayed for some time due to the unavailability of people. The optical interface cards in this machine were replaced with the more sensitive Dag3.5 cards attempting to resolve a low signal level problem. [Bud Hale]
  • Special configuration for U. of Fla. monitor to enable Dag3.5 support with runcoral. [Jörg Micheel]
  • MAX site (Mid-Atlantic Crossroads, near Wash. D.C.)  - working with the manager, Dan Magorian  re the OC48 monitor there.  It appears that there was some damage to the machine in shipment. (Curiously it was shipped back and forth to New Zealand with no damage but it appears it was damaged in the shipment to College Park, Md.) [Bud Hale]
  • Old Dominion University  - turned off both the AMP and PMA monitors pending the resolution of router problems caused by the installation of a new router.  Work towards a resolution continued at the end of the reporting period.  [Bud Hale]
  • Columbia University, the nai-p-bwy site - previously it was detecting data on one of the two channels, it is now failing to detect data on either of the two channels. Continued work with the site people is expected to resolve this problem. [Bud Hale]
  • U. of Memphis - site outage, campus connection problem. (Both the AMPlet and the PMA monitors not connecting.) Resolved. [Bud Hale and Jim Hale]
  • U. of Buffalo - worked with the PMA site personnel to resolve their optical connection problem.   [Bud Hale, Jim Hale]
  • Argonne National Lab.  - needed to move their monitor; monitor was shut down and the move proceeded.   [Bud Hale]
  • Resolved an issue regarding Host Access Restriction (examples of previous standard practices). Will also have for future machines. [Jim Hale]
  • Worked on machine access issue on the PMA machines. [Jim Hale]
  • Inventory of the NLANR PMA equipment and devices currently on hand.  Included in the inventory was the passive monitor computers, chassis, and both Fore Systems and Dag optical interface cards. Prepared a report for Ronn Ritke on what was available for use in future PMA deployments.  [Jim Hale]
  • Inclusive list of sysadmin tasks

Active Measurement Project (AMP)

• Spent some time thinking about a new testing architecture.  The challenge is to think up something relatively simple that accommodates what we need, with new types of test being added.  One interesting thing is the need for some tests that are done only if others can't be (e.g., use ICMP if you can't use IPMP).  [Tony McGregor and Matthew Luckie]
  ~ ~ ~
  IPv6 and IPMP
• Things continue to progress with the IPv6 work that Matthew is leading. We had a couple of machine IPv4 address changes this period as a consequence of people wanting to move their monitors to put them on an IPv6 network.  [Tony McGregor]
• Started collecting IPv6 measurements. Wrote the scripts to collect the data and generate those pages, and ported the traceroute grapher to understand the traces being collected.   http://amp.nlanr.net/~mjl/ipv6/   [Matthew Luckie]
• Received an email from Michigan Tech - they've just installed a Juniper that does IPv6 and allocated AMP an address.  8th site with an IPv6 address (mid October). [Matthew Luckie]
• It would appear that a router is dropping IPMP packets encapsulated with IPv6 headers in the path between the two.  Will modify the code to do a  traceroute style probe so I can see which hop is dropping the packets. [Matthew Luckie]
• Wrote a script to go to each AMP site and fetch its rc.conf file, change a few settings, and then write it to the system manager.  Each AMP site will have the traceroute request software working and IPv6 turned on by default (for autoconf purposes) when this change gets pushed out to the 4.6 sites. [Matthew Luckie]
• Made the measurement daemon write compressed files by default and made the config file format spiffy. Decided to write the compressed files by default (a departure from the ASCII output AMP currently uses) because I'm going to be capturing different data on the IPv6 mesh - namely one-way delays. [Matthew Luckie]
• Working on creating an IPv6 network connection for the SDSC amp site amp-sdsc. Although IPv6 exists at SDSC it was not easily connectable. However that connection is complete and seems to be working well. Also a CDMA time receiver has been connected to the SDSC amp monitor to provide for delay measurements. [Bud Hale, Jim Hale]
• Troublesome issues arose with the CDMA receiver and the connection using the time distribution unit from Endace. It developed that there are undetermined problems with this connection and the CDMA receiver had to be connected directly to amp-sdsc. Additional work will be needed to resolve these issues.  It is important because other applications also depend on this receiver.  [Bud Hale, Jim Hale ]
• The CDMA receiver for the NYSERNet installation was shipped to Bill Owens.  The IPv6 at SDSC will be testing with the NYSERNet machine in New York.  [Bud Hale]
• Added code to the IPMP kernel module to only use ip6_output if it is available in the kernel.  The module used to not link against a kernel that did not have IPv6 support.  Now it does. [Matthew Luckie]
• Using some spare machines in the WAND lab, installed Linux and implemented IPMP path record support for Linux 2.4.19. [Matthew Luckie]
• Conducted a bandwidth estimation exercise using IPMP echo packets on a network I setup in the lab to see if I could get an accurate estimation.   [Matthew Luckie]
• Some work on incremental updates to the IPMP checksum under Linux to make sure am getting the best performance possible.  [Matthew Luckie]
• Investigating the quality of the various NTP servers (that sites have advised).  Also been conducting one-way delay measurements with a few sites from sorcerer, the machine at Waikato. AARNet's path to Waikato (across the Tasman Sea) produces very interesting graphs.   http://moat.nlanr.net/~mjl/graphs/aarn/  [Matthew Luckie]
  ~ ~ ~
  
• Major infrastructure work included the mesh creation and deletion code. Completed the code to add a mesh.  The main thing that was missing was code to make it fork from the crontab.  Wrote that and testing has begun on SDSC.  [Tony McGregor]
• Distribution everywhere to follow if tests are successful.  Eventually the cron stuff will go away when we add the measurement daemon, but that's waiting on the data interface library (which will allow us to change the data file format).  [Tony McGregor]
• Removed the HPWREN mesh from the database so that the HPC.list file won't have them any more.  Also added the ability to delete a mesh from the database.  That was needed to remove the HPWREN mesh and will, no doubt, be needed again as the number of meshes increases with campus deployment.  Zipped up all the data from the HPWREN mesh and saved it on the HPSS (and deleted it from AMP and VOLT).  [Tony McGregor]
• Went over the list of machines that were sending data to HPWREN machines and checked out each one.  A bunch are not accessible, others are 3.0 (which were updated manually).  [Tony McGregor]
  ~ ~ ~
  
• By the end of October, most of the issues re the master (FreeBSD 4.6) system disk had been resolved, including issues with the system disk kernel (a kernel parameter connected with the keyboard flag).  Also connected with that was the need to have a single binary kernel that would be on all machines of this type. We are currently using the binary kernel from the amp-sdsc machine on the disk being used to develop a master disk. Tests are continuing. This will allow preparation of many machines to be shipped. [Tony McGregor, Bud Hale, and Jim Hale]
• An infrastructure-wide system update of all FreeBSD4.6 AMP remote sites was performed after additional changes were made to the master system disk and successful testing.  At first, following the update, most sites were reporting a malformed line in the HPC.list, (report was in the data status page). (Persisted only on the amp-csupomona site, see below.)  Following the infrastructure-wide update many sites indicate connection to the amp-apantyo (Asian Pacific Advanced Network in Tokyo) site. The upgrade to FreeBSD4.6 can now be considered fairly complete.  [Tony McGregor, Bud Hale, and Jim Hale]
• As part of the update all system manger run, made a new script to update a given list of sites in parallel. (Needed because of the network failure at SDSC.)  Corrected a couple of IP addresses that had changed during the upgrade to 4.6 and sent them out using system manager.  Also a new key for Matthew.  Passed on to Bud some information about the decision Matthew and I made in resolving the discrepancies between sdsc2 and sdsc and the system manager state. [Tony McGregor]
• Worked on getting the system manager back into sync, ensuring the update only touches files that need to be updated.  Wrote a perl script to contact each host and get any IPv6 addresses that it has so I can see if other AMPlets have picked up an IPv6 address with autoconfiguration that I did not know about. [Matthew Luckie]
• Early work on the 4.6 system master disk during the reporting period included reseting the amp-sdsc2 pseudo site to test the system manager to update sites and perform the initial turn on sites. At which point it was discovered disk image created as the FreeBSD4.6 AMP system disk had some problems. This was resolved; additional system manager update tests were run, followed by an update of the local SDSC site. It was necessary to work with Todd on troubleshooting the problems which resulted from creating the AMP FreeBSD 4.6 system master disk image from an uninitiated AMP FreeBSD 3.0 system disk.  [Bud Hale, Tony McGregor, and Jim Hale]
• Fixed (I hope) a small bug with AMP booting IPAS that showed up after the SDSC power failure. [Tony McGregor]
  ~ ~ ~
  
• The am-slave process crashed on the volt server late in October. The cause of the crash was not obvious. After some study it appears the compression software may have contributed to the crash. Consulting with Todd to develop some good conclusions.  Disk balancing was done to hold off an archiving that will be needed soon.  [Tony McGregor and Bud Hale]
• A previous examination of the archive process on the VOLT machine revealed a problem. When it was run last it appeared to have completed but did not take the disk fill down to the expected point.   Earlier, AMP had run out of disk space, the archiver was started and apparently ran fine.  [Tony McGregor and Bud Hale]
• As the VOLT server disks were reaching the upper capacity limit in early October, Tony wanted to let the fill run to near capacity to check the inode allocation. After allowing the disk fill to reach near the limit Tony determined the inode allocation was not being exceeded. This was followed by running the archive process. This was completed but some anomalies occurred which bear investigation.  [Tony McGregor and Bud Hale, Jim Hale]
  ~ ~ ~
  
• Further work on the data transfer to Fiano. Been held up for some weeks because our scp won't talk to theirs. However, ssh does work so I rewrote the scripts to do the copy with ssh. It's a bit clumsy, but does seem to be working. Project turned over to Cooper later in this period. [Tony McGregor]
• Started work on a solution to the ssh connectivity problems (causing data transfer problems to Simone Triglia, Fiano).    [Cooper Nelson]
• John Sherwood at Dalhousie University wrote enquiring if it was possible for them to get a monitor.  After discussion with Ronn we decided to place a monitor there, mostly because they are advanced with IPv6.  [Tony McGregor]
• The new amp monitor for the Pacific Northwest GigaPop Seattle was turned on.  (After some difficulty configuring it caused by a rogue file that used up all this disk space.)  However, upon being turned on, it immediately appeared with a problem. Earlier it seemed that there were two hosts on the same IP. However continued work with the site people revealed it was a router problem in the GigaPop. That was resolved and it is now working.   However as with the new amp-apantyo site it is not fully meshed in the infrastructure due to the fact that the system manager has not fully distributed the HPC.list.  [Bud Hale, Jim Hale, and Tony McGregor]
• AMP existing remote sites maintenance and troubleshooting:  
  
  Remote sites are functionally solid except for some minor outages and some sites in transitional states. At the end of October, work was being done on 13 AMP sites with issues/problems. [Bud Hale, et al.]
  • Worked on testing and compiling a list of machines still to be scrutinized for FreeBSD upgrade problems.  [Jim Hale]
  • Three sites, amp-korea, amp-uwyo and amp-okstate are not upgraded but can be. Go ahead is in place. [Bud Hale]
  • Three other sites: amp-wpi, amp-odu (router issue), and amp-bcm have connectivity problems. They are collecting and transferring data but cannot be logged into by ssh. Working on those issues (possible that port 22 is being blocked at a border router at each site). [Bud Hale]
  • Note:  Old Dominion University (amp-odu) turned off both the AMP and PMA monitors pending the resolution of router problems caused by the installation of a new router.
  • amp-ucsb - after much time and possible resolutions, the site was recently started but the system_manager-3.0 update/startup process caused a connection problem. Am working on determining why it lost connectivity to be careful not to cause it again. [Bud Hale]
  • amp-apantyo - <in October> an issue still exists such that all other sites are not collecting from it. That indicates that the HPC.list in all sites still does not include that site. That will be checked out.  [Bud Hale]
  • amp-apantyo -  <in September>  The machine was shipped during the effort to update all AMPlets to the FreeBSD4.6 version. For that reason it was not immediately initiated.  Entered the parameters for the Asian Pacific Advanced Network Tokyo into the amp/volt servers and the system manager system. Then initiated the system manager update system to initiate the monitor. After some corrections the site came on-line collecting data. Following this, updated the site to FreeBSD 4.6.  [Jim Hale]
  • amp-csupomona - has been determined to need a replacement system disk. Site people have determined that the machine will not boot from the disk. Either the disk has failed or the boot record is corrupted, as a result of or during the system_manager-4.6 update run. [Bud Hale]
  • Other sites with problems still being pursued as of the end of October - amp-msstate and amp-uah (outage following system update). [Bud Hale]
  • Other sites that were in a transitional or stopped state (resolved and back up) during the reporting period included amp-ucf (outage following system update), amp-lbnl (moving their AMPlet to new network), amp-uoregon (also moving their AMPlet to new network), amp-gatech (switch port problem), amp-rpi (similar to gatech's network problem), amp-memphis (some router blocking campus-wide connection problem), amp-slac (several issues inc. port 22 block, now removed), and amp-cwru (moved their AMP monitor to another network).  [Bud Hale, Jim Hale]
  • A power outage at SDSC in mid-October caused a shutdown of all the NLANR machines in the SDSC machine room. However all the machines were brought back on-line without major problems.   [Bud Hale]
  • Sites in the southeastern part of the country experienced outages the last week of September.  Regional routing problem.  [Bud Hale]
  • Some equipment problems surfaced that had to be resolved. It appears that 160 MByte SCSI connections are cable sensitive. Also some "out of the box" disk failures surfaced that needed to be handled.  [Bud Hale]
  • Inclusive list of sysadmin tasks

Additional Performance Measurement and Analysis Activities

• Did a new topographic image of HPWREN:   http://hpwren.ucsd.edu/topo.html  [Hans-Werner Braun]
• Added an application where a 3-D ultrasonic anemometer is multicasting a new set of measurement data ten times a second; fairly low data volume. [Todd Hansen]
• Some analysis/graphing of the anemometer data. [Hans-Werner Braun]
• Wrote an article for ACM Crossroads (several drafts before completing the final one). Plan to incorporate much of what I've learned in writing the article into writing Web page content. [Jeff Baker]
• Formulating the slides for the coming student presentation on Nov. 6. Finalized changes to the wireless_mon script that produces the status.html page on stat.[Jeff Baker]
• Coding on the Netflow: Port flow page to display each port's usage.  Continued adding additional code to the portflow.cgi page.  [Jeff Baker]
  ~ ~ ~
  
• Designed and wrote two network report scripts: 1. network utilization and 2. outages.  They are in a state ready to put on the main page. This including implementing the new query page and output design on the old script and working out some problems. They are available on the main page at http://stat.hpwren.ucsd.edu/ under "Network Reports." [Bill Gahr]
• Added the html form and html output to the network utilization script. also made some of the code more modular, for ease of reuse of some routines in generating other results. The page is available at:  http://stat.hpwren.ucsd.edu/wgahr/ifutil2/ifutil.cgi  [Bill Gahr]
• Added the peak utilization tracking at the interface level, so each interface's peaks are tracked separately. This will allow much more flexibility including options like restricting results to one per interface, showing addition results from interfaces in the result, querying particular interfaces, or the same sorts of things at the per router level.  [Bill Gahr]
  ~ ~ ~
  
• Worked on scripts for analysis of wireless data, including:  Updated Tsunami script to allow selection of which Tsunami MIB fields to graph.  Wrote, tested, and added code to handle two types of data with which the script will work.  [Zhao Li]
• Wrote a script to plot other data field such as ber, error seconds, and severely error seconds. Modified the scripts to allow use of overlapping windows in calculation.  [Zhao Li]
• Wrote and ran a script to scan all MIB data file on Perf directory for errors.  [Zhao Li]
• Wrote an explanatory Web page about how to use the Tsunami RSL statistics script, and posted some examples.   http://stat.hpwren.ucsd.edu/HPWREN/New/Tsu_vs_Weather_MIB_page.cgi  [Zhao Li]
  ~ ~ ~
  
• HPWREN data analysis project - narrowed the focus to a "typical" week (Saturday through Sunday) in August of 2001 and 2002.  Graphs can be seen at http://stat.hpwren.ucsd.edu/~coop/draft2.htm.  Began writing up an analysis of this data.  Worked on netflow analysis of the same period and generated some graphs of shorter time periods to get a better understanding of minimum utilization during daily cycles.  [Cooper Nelson]

Documentation, Networked Data, and Tools

• Published the Abilene OC48c data.  http://pma.nlanr.net/Traces/long/ipls1.html  [Jörg Micheel]
• Planning and discussions re needs/updates and development of the PMA Web pages (several).  [Jörg Micheel, Cooper Nelson, Maureen Curran]
• Installed link checking software to find broken links on the PMA Web pages. [Cooper Nelson]
• Made a favicon for the AMP Web pages (a little meter that looks like a sun). [Tony McGregor]
• Created a data banner with the new acknowledgment info with both cooperative agreement numbers and begin dates (per email w/Greg Monaco). Set up the directory structure for the International Web pages.  [Maureen Curran]
• Working on the International Collaboration Web pages, including preparing photos, creating the index page, and creating artwork.  [Jim Hale; Ronn Ritke (rev.) Maureen Curran (ed.)]
• Following a suggestion from Hans-Werner, created a world map that highlights NLANR/MNA International Collaborations for both AMP and PMA.  
  [Mike Gannis]
• This world map is an impressive way to display the collaboration information.  Will use this new slide for the presentations.  [Ronn Ritke]  
• Created a couple of updated AMP and PMA maps for an upcoming presentation.  [Mike Gannis]
  ~ ~ ~
  Cichlid 3-D Visualization System
• Ben Reesman accepted the second student researcher position (will begin in early November).  He will work on the Cichlid 3-D visualization system, the development of which has been dormant since Jeff Brown went into the UCSD Computer Science Ph.D. program.  
• Put together a list of things that need to be done with Cichlid. Helped Ben a bit with Cichlid.  [Justin Fields]  
• Worked on producing images for a new Cichlid poster; working with Gail on this. [Justin Fields]
• Working on new Cichlid poster. [Gail Bamber, Ronn Ritke, Justin Fields, Mike Gannis]  

Papers, Publications, Presentations, and Conference/Meeting Participation

• Thailand and China trip, multiple presentations, both formal and informal, regarding the NLANR/MNA activities, with special emphasis on collaboration and possible placement of AMP and/or PMA monitors.  [Ronn Ritke]
• Report on travel to China and overview of possible collaborations; sent to NSF.  [Ronn Ritke, Mike Gannis, Maureen Curran (ed.), Hans-Werner Braun (rev.)]
• Attended the IST SCAMPI meeting in Prague (http://www.ist-scampi.org/). SCAMPI is the European counterpart of HPC activities in the U.S., aiming at developing a 10 Gigabit network analysis platform. This coincides with our OC192MON activities. [Jörg Micheel]
• Presented at the Australian UNIX Users Group Annual Winter Conference in Melbourne (AUUG2002). We had an accepted paper with Ian Graham and two students from the University of Waikato Comp. Science on duplicated packets in an IP trace taken in 2001 at the University of Auckland access link. [Jörg Micheel]
• Talk on NLANR/MNA research work while visiting AT&T Research Florham Park; it was well received.  [Jörg Micheel]
• Presentation at the 10Gig Workshop at SDSC. [Ronn Ritke]
• Completed the slides for the Fall I2 Meeting in LA presentation. [Ronn Ritke]
• Presentation at the weekly computer science seminars (Univ. of Waikato CS dept.).  [Matthew Luckie]
• Conference call/ discussion about MNA public presentations.   [Hans-Werner Braun, Ronn Ritke, Tony McGregor, Jörg Micheel]
• Sent Ronn some AMP presentations.  [Tony McGregor]
• Wrote a bit about the international collaborations for use in the NATimes (and on the main International Collaborations Web page). [Tony McGregor; Maureen Curran (ed.)]
• Completed full editing of several articles for the next issue of the NATimes, which is focused on our international collaborations.  [Maureen Curran]

Collaborations and Student Involvement

• Contacted by Casey O'Leary from PNL regarding the possibility of supplying an OC48MON for SC2002. We've decided to support the idea via spares from the original IPLS installation.  [Jörg Micheel]
• Spent three days at the Australian UNIX Users Group Annual Winter Conference in Melbourne. The conference was quite useful as I got to know a number of interesting folks in this part of the world, in particular some Australian ISPs, Itochon (IPv6 hacker from Japan) and some folks from CISCO in Sydney, who were extremely interested in the talk, which I focused more on current PMA research, packet delays and loss across routers and long distance. [Jörg Micheel]
• Visited AT&T Research Florham Park. No question that their primary interest remains focused on the cards, but I can see that there are opportunities to work together on the research front and we need to explore those. [Jörg Micheel]
• Request from Ilze Ziedens for more details on Auckland-I, a data set I had collected a mere three years ago. I undug traces and tools and could supply her the details, discussion kept going until the end of the week when we finally had clarified all the misunderstandings. Ilze is a statistician at the University of Auckland, also part time doing research with the WAND group. [Jörg Micheel]
• Wed Meeting with Ian Graham. We spoke about the OC192 development.  [Ronn Ritke]
• Brief email exchange with Anja Feldmann (Uni Saarbruecken, on the move to TU Munich), she was doing analysis work on Bell Labs I and found a number of problem. Shocking for me to learn that the data has problems with timestamp errors, I am not exactly sure how they sneaked in. It looks like you cannot correct for the ones I found. [Jörg Micheel]
• Have worked with Klaus Mochalski to produce the delay figures, but there is some nasty bug in his tools.  While things appear to have worked fine for Auckland-VI, they absolutely do not work for Abilene-I, with 0.01% of packet matches. We are looking for the problem. [Jörg Micheel]
• Also got the new set of four monitors in Korea, Germany and Norway approved, on the plate is to contact the sites and negotiate on shipment and installation.  [Jörg Micheel]
• Joe St Sauver (Oregon) and Bill Owens (NYSERNET)have passed the URL for the IPv6 preliminary measurements to a few people, so it seems to be getting a little bit of publicity.   positive feedback/laudatory email [Matthew Luckie]
• Talked with k claffy and Nevil Brownlee at CAIDA about IPv6 AMP.  We talked about doing an IPv6 skitter.  Sent an email to Bill Owens at NYSERnet asking about IPv6 address allocations so we know where to look to start. I got an extremely useful response from Bill so hopefully that project will go ahead. [Matthew Luckie and Tony McGregor]
• Multiple discussions with Jay Dombrowsky and Tom Hutton of the SDSC NetOps group to implement an IPv6 connection for the AMP machine in the SDSC machine room. It looks as though the machine amp-sdsc will require a second NIC that will be setup as the IPv6 interface. Some attention will need to be paid to the system manager to insure it is compatible.  [Bud Hale, Jim Hale]
• After the monitor had the second NIC installed, it was expected to be testing with the NYSERNet machine in New York. The CDMA receiver for the NYSERNet installation was shipped and Bill Owens is expected to get it installed.  [Bud Hale]
• I spoke by phone with Wendy Huntoon about AMPs at the GigaPOPs. We will continue the discussion at the Fall I2 Member Meeting in Los Angeles in just over a week. [Ronn Ritke]
• Met with Todd Hansen for further details about a possible NLANR/MNA and DAST project idea.  [Ronn Ritke]
• I spoke with Peter Oneil during the NLANR coordinating call and he will follow up on having someone from the Front Range GigaPop fill out the on-line AMP monitor form. This is in order for an AMP can be configured and delivered to the Front Range GigaPop.   [Ronn Ritke]
• Efforts to put AMP machines at the major GigaPops resulted in a request for an AMP machine Internet2 group for the Pacific Northwest GigaPop at Seattle. A machine was prepared and sent and is on-line.  Working with John Hicks.  http://watt.nlanr.net//active/amp-pngs/HPC/body.html  [Bud Hale, Tony McGregor, Jim Hale]
• Received a couple of European AMP requests (Switzerland and Amsterdam) as a result of Jörg getting in touch with his contacts. We're also considering an AMP in Mexico.  Followed up on some of the international AMP possibilities that Jörg pinged; discussions continuing.  [Tony McGregor]
• Participated in the E2E TAG call on Wed. AMP may be providing data for the E2E Pipefitters project, spoke with Eric Boyd to pass on Tony's positive response.  [Ronn Ritke]
• Sent updated versions of the AMP daily and weekly CGIs to Manhee Lee (Kreonet), along with the traceroute graphic code.  [Tony McGregor]
• John Sherwood at Dalhousie University wrote enquiring if it was possible for them to get a monitor.  After discussion with Ronn we decided to place a monitor there, mostly because they are advanced with IPv6.  [Tony McGregor]
• Phone call with Ana Preston about AMPATH Chile Meeting, the CUDI project in Mexico, and the scheduling of the Mon presentation at the I2 Fall Member Meeting in Los Angeles. [Ronn Ritke]
• Participated in a conference call with Don Mitchell and Roseanne Steckler. Bill Chang from NSF, returned from his trip to China and mentioned to Don the NLANR/MNA measurement activities in Thailand. Don is working on a project in the North of Thailand and is interested in NLANR/MNA participating and seeing where we might be able to help with this project. [Ronn Ritke]
• Meeting with Kevin Walsh about upcoming 10GigE workshop at SDSC and the ATEAM Special Projects Proposal that will be submitted.  [Ronn Ritke]
• Meetings with Bud and Jim Hale about both AMP and PMA. Seattle AMP for one end of TransPAC, the OC48 mon at MAX, and the current status of loaned cards to John Duggan.  [Ronn Ritke]
• Met with Bud Hale about Nauka Grid AMP at STARLIGHT.  [Ronn Ritke]
• NLANR coordinating call (Oct. 3). [Ronn Ritke, Hans-Werner Braun, Mike Gannis]
• Mark Sheddon and Susan Rathburn worked with me to schedule a timeslot at the next SDSC General Staff Meeting (GSM) to announce the new NLANR/MNA NSF STI award. (Presentation at the GSM meeting was Wed Oct 16th.) [Ronn Ritke]
• Emails with Jianbo Gao about using OC48 and wireless traces for research. [Ronn Ritke]
• Talked with Kevin Walsh about giving a short talk on NLANR/MNA activities (OC192/10Gigbit Monitor and the Gigabit interface AMPs at the DTF sites) at the upcoming 10Gig Workshop at SDSC. He will give me a timeslot on the second day.  [Ronn Ritke]
• Discussed the UCLA measurement proposal with Mario Gerla; further discussions with Hans-Werner. Participated in a UCLA conference call re the proposal. Went to UCLA for further discussions. This is the new NSF funded measurement proposal from UCLA, of which NASA is a part.  [Ronn Ritke]
• Xiao Chen (statistician) from UCLA is interested in being a part of our sampling research.  [Ronn Ritke]
• Made arrangements in advance for a number meetings at the Fall I2 Meeting.   [Ronn Ritke]
  ~ ~ ~
  PAM2003 NLANR/MNA hosting
  • We appear to be moving on the PAM2003 front, Endace has committed to sponsorship with a certain amount of dollars, to support lower conference costs for participants, and sponsor some student travel grants. [Jörg Micheel]
  • Some PAM2003 work, to do with the CFP and process of reviewing papers.    [Tony McGregor, Ronn Ritke (rev.), Maureen Curran (ed.)]
  • Working closely with Nancy Jensen, Donna Turner, and Rochell Bernsdorf (of SDSC, who are our PAM2003 helpers).  [Ronn Ritke]
  • Prepared to order the beach towel giveaway for PAM2003. Completed the hotel contract for PAM2003. [Ronn Ritke]
  • Accessory graphics for PAM2003.  [Ronn Ritke, Gail Bamber, Mike Gannis]
  • Modified the PAM2003 logo for print and sent it to the printer.   [Gail Bamber]
  • PAM2003 Web pages:  
  • Designed and created the PAM2003 navbar (using a client side image map for the links); added a special notification line that can be changed as necessary. (Went through a couple of iterations) [Maureen Curran]
  • Designed and made the PAM2003 home page (started with Tony's original borrowed from 2002), as well as a template for subsequent PAM pages. Created the Call For Papers Web page (complete), and "holding" pages for most all of the other pages.  Wrote text for these pages (Program, Registration, Proceedings) that will be fully developed at a later date.  Posted all, and sent an email to anrstaff and the Program Committee. Created and sent out a text file version of the CFP for fanning out; arranged for posting on several IT/CS sites. [Maureen Curran]
  • Worked on configuring the moat server for the PAM2003 Web site.  [Jim Hale, Hans-Werner Braun]
  
  ~ ~ ~
  
• New student researcher Chris Gross began work as the new PMA student the last week of October.  Chris is a first year student at UCSD.  Ben Reesman accepted the second student researcher position (will begin in early November).  He will work on the Cichlid 3-D Visualization System, the development of which has been dormant since Jeff Brown went into the UCSD Computer Science Ph.D. program.  Ben is also a first year student at UCSD.  We are looking forward to working with these new students over a very productive number of years (hopefully four!).  
• Meeting with Justin Fields to go over the current sampling research. Justin is off to a good start. We talked about some possible paths to explore. Key parts are, what can you tell about the original complete trace when using sampling ?, compare results in taking every "ith" sample as to taking one random sample in every set of "i" samples?, when does sampling break down - not give reliable results?, which applications do better at lower sampling rates? [Ronn Ritke]
• Met with Justin to discuss his trace sampling work (that he is doing with Ronn) and a possible paper (Todd had brought this to my attention); decided to meet twice a week to develop the paper (in order to move towards a possible submission to PAM2003).  Also worked with him re the Cichlid HPC/California images for Cenic.  [Maureen Curran]
• Working with the new MNA students, Chris Gross and Ben Reesman; meeting weekly (often more).  [Maureen Curran]
• Had a student <at Waikato University> write a Web-based task manager for AMP related tasks.  It looks like it'll be very helpful.  [Tony McGregor]

Other Activities

• Cooper Nelson started half time with MNA in mid-September (having previously worked for HPWREN), assuming a technical support role for both the PMA and AMP projects.  Has been working with Jörg, Tony, Bud, and Maureen.  
• Two new students joined the staff, Chris Gross and Ben Reesman; both are first year UCSD students.  We had an outstanding group of student applicants. In addition to the superb three chosen for Jörg to interview, there were a number of other students who stood out for various reasons and we will be keeping (and sharing with HPWREN/ROADNet) their resumes, etc.  Shared some of the best candidates with an SIO research group associated with HPWREN/ROADNet (they chose one of the students).  We also brought one of the students to the attention of Ian Graham of WAND (Waikato).   [Maureen Curran]
• For the student openings: preliminary screening and administration of coding skills test to more than a dozen student applicants( half of the more than 30 applicants took the coding skills test); running their code, compiling the info on each of those applicants still interested, arranging interviews with three of the most promising students.  Prepared and posted (emailed to masg) a package with all the info on each of the three to be interviewed, including coding skills test results. Also consulted with Todd about several of them.  [Maureen Curran]
• Interviews with top three student candidates. [Jörg Micheel and Maureen Curran]
• David Cheney, having graduated from UCSD in June 2002, was rehired on a part-time basis to provide assistance to the new student(s) and work with Maureen on various Web projects, including continuing the work on creating a system and related scripts to ease changes to the MNA Web pages (for now and in the future).  
• Made a list of Dave's projects for Ronn/Human Resource (all the various back-end projects).  [Maureen Curran]
• Working with Human Resources re multiple open positions, changes in the General Atomics subcontract, converting Dave Cheney from student researcher to a regular casual/short-term exception position, etc. Involved many, many meetings and discussions.  Writing job descriptions and detailed UC employment cards for each position.  [Ronn Ritke]
• Multiple staff openings included two FTE (full time employees): AMP Software Engineer and the AMP/PMA assistant systems administrator (opened twice with different candidate pool). These positions took a great deal of effort and time from several folks.  
• Extensive review and discussions re the two open positions:  AMP Software Engineer and the AMP/PMA assistant systems administrator. [Tony McGregor, Maureen Curran, Ronn Ritke, and Hans-Werner]
• Preliminary ranking for the sysadmin job (second time around) and screening of candidate pool; chose three for interviews.  Developed a rough draft of interview questions to use.  [Ronn Ritke and Maureen Curran]
• Fleshed out the interview questions for the sysadmin position, incorporating suggestions from masg.  Interviewed the three finalists.  [Maureen Curran]
• Made some changes to the AMP Software Engineer job description, further emphasizing the coding skills aspect, sent to Tony for comments.  Prescreening and ranking of applicants. Discussed the need to widen our search in order to get more applicants who are qualified; developed some ideas and resources.   [Tony McGregor, Maureen Curran]
  ~ ~ ~
  
• Wrote and edited, formatted, htmled, posted for review, and submitted the first monthly reports covering mid July to August. [Maureen Curran]
• Pursued getting additional NLANR shirts. [Ronn Ritke and Bud Hale]
• Payment authorizations, MNA budget issues, REU payment authorizations, purchase orders, meetings with Ronn, travel, summary reports, Waikato funding/subcontract, Tony McGregor Univ Waikato deob, and other budgetary matters.  [Mary DiMeglio]
• Waikato subcontract revisions [Tony McGregor, Ronn Ritke]

  ---

  Inclusive list of sysadmin tasks

  The following systems administration tasks apply to both the PMA and AMP projects.  They are performed primarily by Bud Hale, with the assistance of Jim Hale.

  1. Maintain an on-going and continuous rapport with all NLANR/MNA remote site people. Contact site people on a regular basis and inquire as to the activities at each site. This is necessary to keep up with personnel changes and network configuration changes at each and every site.

  2. Provide educational and informational materials as need by remote site personnel to garner the motivation for support of the equipment on site and the needed connectivity.

  3. Maintain the remote site database and enter any and all changes to equipment, connections and contact personnel.

  4. Maintain records of the equipment deployed as well as the NLANR/MNA equipment at SDSC.

  5. Maintain an on-going relationship with vendors of equipment and parts needed to support the entire NLANR/MNA infrastructure.

  6. Continuously monitor the performance of all NLANR/MNA equipment on a 7 day, 24 hour basis. Perform test and analysis to anticipate equipment failure. Monitor equipment to determine the fill rate of data storage.

  7. Maintain a rapport with SDSC personnel in all departments to garner support needed for NLANR/MNA equipment installation, NLANR/MNA network connections, work stations and spaces.

  8. Continuously monitor the operational status of SDSC equipment such as the HPSS to determine the usability by NLANR/MNA functions.

  9. Continuously monitor the accumulation of NLANR/MNA data and perform NLANR/MNA data archival as needed.

  10. Maintain a supply of spare parts and/or maintain sources of spare parts supply.

  11. Perform on-going equipment repair on a 7 day, 24 hour basis as needed.

  12. Run the procedures to accomplish remote site system updates on a regular basis.

  13. Monitor NLANR/MNA server operation and performance to anticipate equipment failure and/or software performance problems.

  14. Maintain an awareness of industry progress to anticipate obsolescence. Perform tests of new components as necessary to determine operability with NLANR/MNA systems.

  15. Analyze connection anomalies to develop corrections as needed. Maintain records of corrections applied to aid in future problem resolution.

  ---

  Laudatory email re IPv6

  Date: Mon, 14 Oct 2002 22:40:37 -0400 (EDT) From: owens@nysernet.org To: Matthew Luckie <mjl@nlanr.net> Cc: owens@nysernet.org, Joe St Sauver <JOE@OREGON.UOREGON.EDU> Subject: Re: IPv6 AMP

    On Mon, 14 Oct 2002, Matthew Luckie wrote: > http://amp.nlanr.net/~mjl/ipv6/> > it is pretty basic, let me know if you want stuff added

    I love it - it already tells me a bunch of interesting things. Like the fact that the only low-loss v6 path is the one inside the own backbone, but that still has jitter problems (the three Ciscos involved, probably, since they're process-switching). Oh, and I like the ratio stuff that you just added while I was looking at it ;)
    Bill.