Summary of Research Activities - September 2004
Passive Measurement and Analysis (PMA) Project~ Continuing development of new metrics and real-time analysis for PMA We finished the flow statistics for the IPLS traces. Then started a real-time analysis session on ipls-chic to see if the software was working and able to keep up with the link rate. The link turned out to be much busier than during a trace capturing just two weeks previous -- up to 10GB/s with nearly 7GB/s UDP traffic. Thanks to this high load we uncovered and fixed some performance problems with the real-time software which had not been apparent before. It turns out that we managed to be involved while the group around Harvey Newman (Caltech) was doing experiments. We shared these extremely interesting results with Matt Zekauskas of Internet2. We are trying to coordinate a packet capture with Harvey's activities some time soon. Two more statistics were implemented on the real-time engine to help with performance profiling, showing Dag packet losses and CPU load average. After running this for more than 24 hours, three singular loss events showed up. A new delay analysis tool with RRD support which we plan to use to illustrate single-hop packet delays at the IPLS location was developed. It has been tested only on a short piece of trace and it still has a problem showing a gap between consecutive trace files. We had dialogs on how to improve the real-time interface, resulting in displays of CPU load and DAG packet loss, for debugging. We investigated in detail the present Leipzig source code tree (daglib2) and in follow-on discussions regarding adapting it, we determined that while the RRD database is the nexus (for our methodology), it is also an obstacle to move in this direction. There are fixes and also solutions; the tradeoffs are between quick-and-dirty and longer term solid designs. To be continued. ~ Special Traces The first delay data, IPLS Packet Delay Chicago -> Indianapolis, have some remarkable bits. The data is fairly consistent with simi liar data from Sprint ATL collecting measurements on a CISCO 12000 across OC48c and OC12c link ports about two years ago. The lower base line is about 17 microseconds, the maximum never exceeds 50 microseconds, and typically sits around 35 microseconds or less. This seems to confirm an important long term trend, routers (in the core?) and queuing adding very little to the bottom line of packet delay and jitter, and the delivery times being dominated by the speed of light and link level distance between points of origin and destination. This in turn has implications for the expected outcome of such large scale experiments of optical/TDM packet infrastructures such as Internet2's HOPI. There may be a no loss/no gain situation between switched/routed networks and pure optical ones in terms of the Quality-of-Service, except flexibilities for sharing may be different. This obviously requires further investigation and research. http://mars.nlanr.net/cgi-bin/delay.cgi HPWREN traces ~ We wrote a BPF-based 'tshdump' program that provides two functionalities to improve and streamline the data collection process. It is able to simultaneously collect and collate packets from multiple interfaces (not previously possible), and also write in TSH format directly. This avoids a CPU/time-costly data conversion. As a result, tshdump is now simultaneously monitoring the two interfaces of the wireless links, and hence collects the traffic before being NAT translated, for more interesting analysis. Documentation on the behavior of the tool was written. We fixed data inconsistencies in the Leipzig-I and -II data sets (found by a teaching assistant in Bill Cleveland's group at Purdue). The bug was located in the copy on the pma server, but not in the original data. It appeared to be a trivial problem at first, but since we only had HTTP access and no md5 checksums, initially time/bandwidth was wasted by having to copy all the data, before we took a more targeted approach to mirroring the exact files that were needed. Problems included issues with files larger than 4GB, which we solved by splitting the data. We resolved it and mirrored copies to the HPSS, as well as published the MD5 checksums, so user communities can address simi liar problems at their end. We discovered irregularities with the latest IPLS data set (five of the IPLS-KSCY trace files are shorter than they should be). We are in the debugging process to figure out why. The IPLS data set now involves all three measurement points and delivers delay data across the T640 router in Indy. ~ New (and developing) strategically important measurements and deployments Development of passive monitoring for a lambda network (first stage prototyping of lambdaMON) ~ CISCO responded with support to our efforts on building a prototype lambdaMON, and we can use all the gear at CENIC to build such a system. The only thing we lack is a tunable 50-GHz channel filter. Research into vendors and products for DWDM equipment suitable for the development of the lambdaMON prototypes and field installation has been a bit tedious. We have done a good market overview with nearly 50 vendors and their products surveyed. We have been in contact with roughly half of them and have a good idea about pricing and capabilities tradeoff. We found the first commercial solution amplifier/tunable channel filter that is suited to the same environment that we are considering to operate, the TCL-10/OACL-10 setup for the Acterna ONT-30. After repeated attempts we finally made contact with the sales engineer to receive a quote on tunable channel filters and optical signal amplifiers. Acterna believes they have provided a viable solution, despite the fact that their TCL-10 device has been replaced with a newer model (and may in fact have never been on the market!). However, from the quote/offer we received, it does not contain the particular module we require. We also spoke with a firm called IOLON which may have a solution that more closely matches our needs, at least regarding the tunable channel filter. We are hoping they have a good amplification solution as well. Instrumentation research with all these firms is daunting: they are very hard to make contact with, there is always a great deal of red tape, and furthermore, they often cease and/or seriously modify devices that they previously manufactured (or turn them into a module available only within a larger device). We have responded to the recent email conversation with Pere Barlet at UPC Barcelona regarding joint OC48MON measurement opportunities in Spain. ~ Upgrades, troubleshooting, and maintenance on the PMA servers and infrastructure To improve security, as the pma.nlanr.net server is becoming a more general purpose analysis machine with HTTP access from the outside, CGI scripts, etc. we prepared, configured, and took online a new PMA Sphinx machine, "strangler." It will be the access gateway server to PMAMONs in the field. We will also be restructuring the security/access to PMA systems a little bit in the next period, which will include support for some of the real-time work that is already in progress. We continued to be plagued by failures on the PMA server until approximately mid-month. Our search for remedies revealed other users describing the same problem. From the usenet sites, it appears that upgrading to FreeBSD 5.3 is routinely becoming the solution to their RAID problems, even in the beta version. (Jörg had also suggested this as a possible solution.) Preparations including the development of a list of needed equipment for Indianapolis and Pittsburgh (SC2004) have begun. Active Measurement Project (AMP)~ Progress on the reimplementation of AMP and the development of a new testing architecture The Perl AMP data interface was completed (to the same extent as the PHP one). The C code was rebuilt to pass a set of generic data types that can be used by PHP or Perl and have the PHP recoded as a thin interface that uses that. The Perl interface was coded to open the AMP "database" and to get successive items from it. Worked on the ampcentral code and have it nearly set up for autoconf/automake. The capability to do a general test with the software is being added (AMP generaltest); it is mostly finished. A few mods were made to the code to support biscuit PCs (used by CRCnet), the most significant one being an error and warning post processor that summarizes repeated errors rather than filling logs with them when something goes wrong. Continued work on the AMP IPv6 code, including work on the traceTest and icmpTest code. Also got the code to compile on Solaris. Connected an AMP machine to the IPv6 network which allowed for testing of the recently written IPv6 reimplimentation code. This showed a couple of problems with the traceroute test which were corrected. Both the ping and trace tests now work with IPv6. This code needs to be tested on a real IPv6 network, but it is in good working order between two machines. Fixed numerous bugs and split out constants etc. into configuration files. Did a couple of minor fixes/changes to the dgraph code, which ended up being a bit tricky to do. Worked on some spreadsheet testing to determine if there exists a spreadsheet that can handle six months worth of round trip time (RTT) data, which apparently, they can. After a bit of struggle setting up the test-bed, we did some testing between two fully functional IPMP machines (unfortunately we were not running the latest version of IPMP). Additional testing of the IPMPtest code will be performed, after updating the machines. We added exponential backoff (to a limit) on failure to resolve the DNS name of a collector because of boot sequence problems. We changed the code to usr /dev/urandom, if it exists, to seed the random number generator. We started the new ampz- (New Zealand AMP) mesh this period. It currently has 5 operation machines with another two shipped and several others waiting until this initial deployment is working. It is running the new code and we have an experimental (and somewhat raw) version of the new central code and Web pages running. This is also running on CRCnet. Several small fixes of the amplet code were done for the CRCnet wireless network. One particularly interesting issue is that the tests were having a big impact on the network because they were all running at the same time. We have always known this was a potential issue and the existing system randomizes the timing of the test somewhat. We had not put that code into the new system until we hit this problem. However, what is particularly interesting is the extent of the problem on a wireless network (100s of ms). In this case, many of the links are long point to point 802.11b links; we believe the problem was worse here because they are half duplex and tests were turning up at both sites of the link at just the same time. So, we added some simple linear perturbation code. In the future, we plan to also add Poisson distributed delays. ~ New (and developing) strategically important measurements and deployments University of Nevada at Reno, amp-unr, request received, prepared and shipped, installed, initialized, and collecting data. At the end of the month, two additional AMP monitor requests were received, one from Iowa State and another from U. of Washington. 10GigE AMP ~ The procurement (loan from the vendor) and assembly of the S2io 10GigE NIC card test platforms was completed. After which, we began performance testing the 10GigE interfaces, slowly increasing the speed, tuning the test and the OS, as required. The amp-cnic (China Computer Network in Beijing) continues to perform well. IPMP ~ The IETF IMRG working group released a review of IPMP (posted to the IMRG mailing list), early this month. We wrote a detailed response; it is posted at: http://www1.ietf.org/mail-archive/web/imrg/current/msg01105.html ~ IPv6 and IPv6 Scamper Scamper was publicly released in early September. Matthew was quite happy with the state of the code. http://www.wand.net.nz/scamper/ We moved all the code related to select out of scamper.c itself, which tidied the main loop quite a bit. Also added support to kqueue for systems that support that. (kqueue is basically a scalable version of select.) We found that the interaction between kqueue and BPF BIOCIMMEDIATE file descriptors is different across the BSDs, and so disabled kqueue on Mac OS X. Also finally added code so that Scamper will abort tracerouting a path to a non-responsive target. With Jamie Curtis (WAND), we discussed using Scamper to discover the topology of the NZ Internet before (and after) the Telstra de-peering takes place. Finished writing the code for Scamper to obtain transmit timestamps using BPF. Added code for Scamper to use PF_PACKET under Linux to obtain the same thing. Looked briefly at how the transmit timestamps differed between BPF and gettimeofday in userland. Some potential improvements were seen on slower systems in the order of a millisecond or two, but the actual difference for most timestamps on a FreeBSD 4.10 Athlon 1.2 with an Intel pro100 was in the order of 10-15 microseconds, and 150 microseconds on a FreeBSD 4.10 p3-800 laptop with an Orinoco wireless card. We anticipate that the importance of the code might increase when Scamper's PPS rate is above 100 and/or the local network is busy. Note that the network interface on all of these systems was otherwise idle when doing these tests. In the process of writing that code, we noticed that FreeBSD's BPF generates a timestamp when each packet passes the filter, so if Scamper gets a packet from BPF after tcpdump does, Scamper's transmit timestamp will be slightly after tcpdump's timestamp. We sent a patch to FreeBSD to generate a BPF timestamp just once before BPF does any filtering. http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/71711 A bug in libpcap/bpf during the implementation of similar code in Scamper was also noticed; we submitted a small patch, which got accepted. http://cvs.tcpdump.org/cgi-bin/cvsweb/libpcap/pcap-bpf.c There was some talk on NZNOG about PMTUD blackholes. We used that to point NZNOGers at Scamper. Received a few useful emails back, including someone who tried to compile it on an ancient version of Linux without some IPv6 headers. We managed to get it compiling/working. The WAND Web site took about 90 hits from the link. http://list.waikato.ac.nz/pipermail/nznog/2004-September/008729.html Code was written to enable Scamper to read packets from live pcap devices. The idea is that RTT values will be more accurate. ~ Upgrades, troubleshooting, and maintenance on the AMP servers and infrastructure Testing and transition to the new AMP servers (AMP2 and VOLT2) ~ At the end of the period, the AMP data collection/server function continues to be the VOLT server as the primary and the AMP2 server as a backup. There is a plan underway to return the AMP monitor to service and reconfigure the AMP2 and VOLT2 servers to a different RAID scheme. The new scheme to be used is the RAID10. That is a combination of RAID1 and RAID0. This is a combination of disk mirroring and stripping. The following work regarding the transition was performed this period.
Support and troubleshooting, existing AMP measurement sites: Site outage remains at a very low level. A total of 10 remote sites in the AMP infrastructure received attention during this period. "Open" means that the site is still being investigated, or pending action by site technicians. Outages are considered "open" until the monitor is again collecting data. Details follow. 10 problem sites: 6 resolved, 4 open - at the end of the period. amp-uab (U. of Alabama, Birmingham) had a short outage due to a misplaced ethernet cable. That was corrected shortly after contacting the site. amp-dartmouth (Dartmouth U) ~ had a short outage caused by a bad switch port and was corrected shortly after the site technician was asked to check it; collecting data. amp-fsu (Florida State U.) ~ site technicians became concerned by indications from flow measurements they had made. But after some discussion, we have resolved their questions/issues. amp-korea site (KISTI in Korea) ~ confusion about the machine following some personnel changes at the site has caused some problems. Early in the month, we were able to get the site back up collecting data after it was taken offline. The new site technician reports now that the machine has a hardware failure. The corrective action plan now is to ship a replacement unit. amp-odu (Old Dominion U.) ~ short outage caused by a shutdown of all machine room equipment due to air conditioning system breakdown; back collecting data. amp-ou site (U. of Oklahoma) ~ The monitor was collecting and transferring data but could not be updated. We prepared and shipped a replacement monitor after working with the site technician and failing to resolve the problem. This demonstrated to the site technician that the problem is in the router configuration. We anticipate that the correction will be made shortly. amp-psc (Pittsburgh Supercomputer Center) ~ contacted site technician, power outage, equipment was switched off to ease the load when the power was restored; back collecting data. amp-rnpb (RNPnet, Brazil) ~ working with Sidney Lucina of the RNPnet in Brazil to get the failed AMP monitor there returned. Sidney indicated the Brazilian bureaucracy is giving him some trouble. amp-surf (SURFnet, Amsterdam) ~ continues to lose connectivity to the local network; correctable by logging in through the SURFnet "out-of-band" connection and commanding the NIC down and back up. A replacement monitor is on site waiting for installation. Site personnel have promised to open a ticket to replace the monitor soon. amp-ucsb (U. of Calif., Santa Barbara) ~ power switch failure which shutdown the computer science department machine room. The switch was repaired and the facility powered up again; back collecting data.
Outreach, Collaborations, and Activities supporting Network Research~ Papers, Presentations, and Conference/Meeting Participation Matthew Luckie and Tony McGregor. User Level Path Diagnosis with IPMP. Proceedings of the SIGCOMM Network Troubleshooting Workshop (NetTs), Portland, OR, Aug 30-Sep 3. Kenjiro Cho, Matthew Luckie, and Brad Huffaker. Identifying IPv6 Network Problems in the DualStack World. Proceedings of the SIGCOMM Network Troubleshooting Workshop (NetTs), Portland, OR, Aug 30-Sep 3. SIGCOMM Network Troubleshooting Workshop (NetTs), Portland, OR, Aug 30 - Sep 3 ~ Matthew attended and presented. His talk went quite well. The troubleshooting papers prior to his paper seemed to build motivation for his work which was very nice. Matthew is an author of two papers (primary/first author on one and a coauthor on another) accepted;they only accepted only 13 papers total (from 36 submissions. http://www.acm.org/sigs/sigcomm/sigcomm2004/netts.html PRAGMA7, SDSC, La Jolla, CA, September 15-17, 2004 ~ Ronn attended and also participated in prepations beforehand. The NLANR/MNA logo was featured prominently on the participants page. http://www.pragma-grid.net/pragma7/participants.htm We worked on a strategy and preparations for SC2004. We provided proposal briefs for presentations to be given at the SDSC and NCSA/NLANR booths. Jörg will be giving the presentations/demos. The ordering process for the necessary machines began. ~ Collaborations, Research Partners, and Activities Supporting Network Research With regard to collecting interesting data at the Indianapolis router node, Jörg started a longer email dialog with Harvey Newman's group (CalTech) on joint measurement opportunities, assisted by Matt Zekauskas (for which we are grateful). The purpose really is to have a chance to collect complimentary data to the existing sets when one of the bandwidth challenges is in progress, and to be in a position to study the impact of such giant flows to the overall performance of the router. (Matt also followed another lead with Sylvain Ravot, at CERN and CalTech, we have not yet received a reply.) Xun Su (who is also working on the recent AMPATH data with FIU colleagues) replied on behalf of CalTech, as follows:
Kim Claffy, CAIDA ~ David Moore, CAIDA ~ Greg Cole, GLORIAD ~ Lawrence Wong, National Grid Office, Singapore ~ John Towns, NLANR/DAST ~ NREN folks (RRZE), Erlangen, Germany ~ Bill Chang, NSF ~ Dave Hart, NSF/OLPA ~ Kevin Thompson, NSF ~ Bill Owens, NYSERNET ~ Vijay Samalam, SDSC ~ Charlie Knezevich, SDSC ~ Saverio Niccolini, Telecommunications Institute of Ecole d'Ingnieurs du Canton de Vaud (EVID), Switzerland ~ John Hicks, TransPAC ~ Malathi Veeraraghavan, University of Virginia ~ Jamie Curtis, WAND ~ Ronn Ritke traveled to New Zealand and Australia to meet with several of our research partners, as well as with Tony, Jörg, and Matthew; they discussed AMP software, AMPs in New Zealand and Singapore, AMP and PMA future planning, as well as other topics. He met with the head of Waikato University's Computer Science Department, Steve Reeves, regarding the measurement projects (past, current, and future) and the many successful collaborations between Waikato and NLANR/MNA. He also had a meeting with the Deputy Vice-Chancellor (for Research) to discuss past and future collaborations between NLANR, SDSC, UCSD and Waikato University. He toured ENDACE and met with Ian Graham regarding future projects. While there, he followed up with Claire in Auckland regarding the second FedEx charge to ENDACE. In Australia, he visited the Royal Melbourne Institute of Technology (RMIT) in Melbourne and gave a presentation on NLANR/MNA's work at Melbourne University. Mark Sice of RMIT attended and Bruce Morgan suggested that an AMP at RMIT could be part of a local Australian AMP mesh; Mark is interested in pursuing this. Bruce Morgan of AARNET has long been interested in AMP and developing an AMP mesh for AARNET. He is now in charge of the 10Gig backbone that is being installed and lit around Australia and has not had any time to begin implementation since the code became available. As a result of recent discussions between Bruce and Tony, who offered to help get the Australian AMPs up and running, and Ronn's meeting with him, we have a plan to get the Australian AMP Mesh online. Bruce will send us an email with a local contact to assist Tony.
Documentation, Web Work, Utilization Improvement, PublicationsA new issue of the Network Analysis Times was completed, it is a really good looking issue with the primary focus on the new AMP machine in Beijing, which is now capturing data. We used the press release for the primary article and an abbreviated version of the international white paper. As reported previously, Lana created a great new NATimes "banner" which sets everything off well. Thanks to some stupendous work by Gail who had the printing day from heck, but finally got it to print, to Imprints (campus Graphics) who did a very quick turnaround on the folding, and to Bud, who picked them up and delivered them, we got the issues to Teri Simas by her deadline for inclusion in the attendee packets for the PRAGMA7 meeting which was held at SDSC. The press release regarding the AMP monitor now collecting data was completed and released. http://www.sdsc.edu/Press/2004/09/091504_NLANR.html We finished a paragraph about PMA for inclusion on the Abilene Observatory Project Web pages. http://abilene.internet2.edu/observatory/research-projects.html PMA Collection and Use Statistics (Web logs) project ~ Worked on an issue regarding the scaling of Y-axis values on the Web logs graphs. We want the actual number, but with larger numbers this is problematic given the limited space available. Decided to change the internal scaling and subsequent external representation of the numbers a bit, leaving most of the min/max values to the discretion of RRDtools, rather than forcing the parameter. This appeared to be a viable solution so we began integrating it across the scripts. Refinements will need to be made. Work was done to ensure that gaps are being properly handled by RRDtools and displayed correctly. Work continued on the rollover script which was debugged after the September rollover (Aug 31 -> Sep 1). Graphs for the monitor data that we have back to Oct 2001 were completed. The Sites index page was updated, including rotating retired machines to the historic section. The last column which had been a status column, was changed to Collection Stats, with links to the current month's graphs for each monitor. We created a monitors data collection index page and linked it to the Sites index and the Download summaries index pages ( http://pma.nlanr.net/Stats/monitorsIndex.html). The Download Summaries page was redesigned; it is the index/front page for the Data Collection and Use Stats. We updated all the text bringing it up to date with what we have available. Also added a sample image to give users an idea of what the graphs look like. http://pma.nlanr.net/Stats/ Citings/Collaborations database project ~ We created an ERD (Entity Relationship Diagram) for the database. We updated it per meetings with Klaus who has been helping us understand and learn more about database design. The goal now is to fully develop the ERD before proceeding any further. Once the Citings and Collabs info is accounted for, it appears that it will not be difficult to add completed text to be used for the reports and have it linked to both a date and subproject/project (and perhaps for other uses as well, such as boilerplate language) We are excited about the possibilities of this tool, once it's developed with good database design. Very detailed comments were added to the database scripts. The goal was for Lana (who is leaving) to make them sufficiently clear for Maureen to edit them without assistance. Lana also created files which explain how to run the scripts and started compiling some basic database commands.
Activities of each individual on the projectAMP team
PMA team
MNA, AMP, and PMA Outreach, Documentation, Web work
Management and Administrative Hans-Werner Braun, Ronn Ritke, Tony McGregor, Jörg Micheel ~
- 30 -  Top 2004 October 15 NLANR/MNA home page 
|