Quarterly Report on Research Activities
3Q 2004 (July - September)
~ Continuing development of new metrics and real-time analysis for PMA
Klaus Mochalski, a visiting scholar from the University of Leipzig, arrived in mid July for a two month visit to work with us on real-time passive measurements, in particular a library of routines we intend to reuse in a number of different applications. We also extended the work we began last summer on packet delays and gathered some data from the IPLS instrumentation for this purpose.
- He worked on the TCP packet loss detection engine, getting it to work properly. While it performed to the original specs he had in mind, it was insufficient for the end goal he wanted to achieve: detecting network congestion. The algorithm worked well in finding two duplicate ACKs as a sign of a packet loss. However, not every packet loss is a sign of serious congestion. It is necessary to differentiate Fast Retransmission (of single packets) from an actual Slow Start caused by a retransmission timeout. However slow start is much harder to detect, not the least due to vantage point problems. Apart from this, complete stateful tracking of all TCP connections is probably unfeasible in real-time. Therefore a new plan which involved looking for sudden drops in a TCP's rate in conjunction with packet retransmission was developed.
- To look at packet delay and loss data across the Juniper T640 at Indianapolis, a new dataset was collected : simultaneous traces at all IPLS monitors (approximately 4 hours, 40 minutes). RRD databases were generated for the basic statistics and put on a preliminary RRD-based Web site with graphs that can be generated on request. Inconsistencies in the NTP support were discovered and resolved (another artifact of our recent changes to the new NLANR IP address space). To solve the problem of obtaining continuous flow statistics with the trace being split up into 10 minute chunks, a utility to merge a whole set of traces was written.
- After finishing the flow statistics for the IPLS traces, a real-time analysis session on ipls-chic was performed, to see if the software was working and able to keep up with the link rate. The link turned out to be much busier than during a trace capturing just two weeks previous -- up to 10GB/s with nearly 7GB/s UDP traffic. Thanks to this high load we uncovered and fixed some performance problems with the real-time software which had not been apparent before. It turns out that the session occurred while the group around Harvey Newman (Caltech) was doing experiments. We shared these extremely interesting results with Matt Zekauskas of Internet2. We are trying to coordinate a packet capture with Harvey's activities some time soon.
- Two more statistics were implemented on the real-time engine to help with performance profiling, showing Dag packet losses and CPU load average. After running this for more than 24 hours, three singular loss events showed up.
- A new delay analysis tool with RRD support which we plan to use to illustrate single-hop packet delays at the IPLS location was developed. It has been tested only on a short piece of trace and it still has a problem showing a gap between consecutive trace files.
- We had dialogs on how to improve the real-time interface, resulting in displays of CPU load and DAG packet loss, for debugging.
- We investigated in detail the present Leipzig source code tree (daglib2) and in follow-on discussions regarding possibly adopting it, we determined that while the RRD database is the nexus (for our methodology), it is also an obstacle to move in this direction. There are fixes and also solutions; the tradeoffs are between quick-and-dirty and longer term solid designs. To be continued.
Chris worked on maintaining and adding improvements to his existing real-time code. Much of that time was spent debugging his next_packet function which facilitates the walk through the Dag's memory window. Among other things, he added some reporting functionality and cleaned up many of his command line arguments, keeping in mind the need to be able to open, keep track of, and report about multiple streams from multiple Dag cards. The plan is to make the software the most compatible and portable with any type of Dag card or any configuration of them.
Klaus Degner (an undergraduate student from the University of Leipzig) will be coming here in October for a few months to work on real-time applications with us. Emails were exchanged discussing ideas for real-time one-way delay calculation.
~ Special Traces
A new data set was captured: simultaneous traces at all Indianapolis monitors. The disk space lasted from 13:40 to 18:20 PDT. These were used in the real-time application work detailed above.
The first delay data, IPLS Packet Delay Chicago -> Indianapolis, has some remarkable bits. The data is fairly consistent with similar data from Sprint ATL collecting measurements on a CISCO 12000 across OC48c and OC12c link ports about two years ago. The lower base line is about 17 microseconds, the maximum never exceeds 50 microseconds, and typically sits around 35 microseconds or less. This seems to confirm an important long term trend, routers (in the core?) and queuing adding very little to the bottom line of packet delay and jitter, and the delivery times being dominated by the speed of light and link level distance between points of origin and destination. This in turn has implications for the expected outcome of such large scale experiments of optical/TDM packet infrastructures such as Internet2's HOPI. There may be a no loss/no gain situation between switched/routed networks and pure optical ones in terms of the Quality-of-Service, except flexibilities for sharing may be different. This obviously requires further investigation and research. http://mars.nlanr.net/cgi-bin/delay.cgi
The IPLS data set now involves all three measurement points and delivers delay data across the T640 router in Indy. We discovered irregularities with the latest IPLS data set (five of the IPLS-KSCY trace files are shorter than they should be). We are in the debugging process to figure out why.
We fixed data inconsistencies in the Leipzig-I and -II data sets. The bug was located in the copy on the pma server, but not in the original data. It appeared to be a trivial problem at first, but since we only had HTTP access and no md5 checksums, we initially wasted our efforts, before taking a more targeted approach to mirroring the exact files that were needed. Problems included issues with files larger than 4GB, which we solved by splitting the data. We resolved it and mirrored copies to the HPSS, as well as published the MD5 checksums, so user communities can address simi liar problems at their end.
~ New (and developing) strategically important measurements and deployments
nai-p-amp ~ We worked to bring the new AMPATH (Florida International University) passive monitoring system live with Ernie Rubi in Miami. We enabled data collection on the monitor, which took a few hours since the configuration is rather different from what we have in the field so far. The system was turned into 8x90 collection for now, to have some data for AMPATH with which to work.
nai-p-pur ~ Further dialog with Scott Ballew at Purdue regarding the GIGEMON there. We are still blind in one direction; discussed his using a light meter to attack the problem.
We finished the description of the installation process at the Abilene IPLS location and have shared it with a number of people for comments. The PMA team discussed and began preparations for the upgrade of the IPLS installation. We also began preparations for our deployment at SC2004 in Pittsburgh.
At the Joint Techs Meeting, Jörg had a very long and fruitful technical talk with Bill Owens from NYSERnet, in which they discussed our plans for putting an OC3MON back into Buffalo, NY, and the GIGEMON into the access link to Abilene. We made arrangements to have an OC12 POS to Abilene installed for (NYSERNet Buffalo -CHINng) as well as a GigE Monitor to Abilene (NYSERNet NYC - NYCMng).
We are in discussions regarding an invitation from Pere Barlet at UPC Barcelona, Catalonia, Spain, to continue the collaboration via placement of an OC48MON inside the Spanish R&D network.
We began an email conversation with Malathi Veeraraghavan, Professor at the University of Virginia, regarding the placement of an OC3MON at VOA. She responded positively. (This was our first positive response from the "cold call" emails.)
Linda Winkler of the Univ. of Illinois, Chicago wants to place a GigE monitor on the connection to UIC and asked if we could convert the previously live, now decommissioned, OC3 monitor to GigE. We told her that the hardware of the OC3 monitor would not support the GigE cards, but that we would discuss the possibility of installing a GigE monitor.
At the JET meeting, Jörg approached Bill Johnston from ESNET to propose using some of the spare NLANR OC48MONs to instrument the ESnet/Abilene peering points. His response sounded positive, as did those from Matt Z and Guy Almes, with whom the idea was shared in a follow-up.
Development of passive monitoring for a lambda network (first stage prototyping of lambdaMON) ~
We have been working on questions of how to passively instrument emerging DWDM-based optical networks. We began researching the specifications, price, and availability of products for the optical networking space, with some 40-50 inquiries to various vendors. We are determining the various DWDM components needed for a prototype lambdaMON, as well as for a larger scale roll out. In the process, we are making some strategic contacts as well as developing questions which will need to be tackled.
Research into vendors and products for DWDM equipment suitable for the development of the lambdaMON prototypes and field installation has been a bit tedious. We have done a good market overview with nearly 50 vendors and their products surveyed. We have been in contact with roughly half of them and have a good idea about pricing and capabilities tradeoff. Instrumentation research with all these firms is daunting: they are very hard to make contact with, there is always a great deal of red tape, and furthermore, they often cease and/or seriously modify devices that they previously manufactured (or turn them into a module available only within a larger device).
We found the first commercial solution optical signal amplifier/tunable channel filter that is suited to the same environment that we are considering to operate, the TCL-10/OACL-10 setup for the Acterna ONT-30. Acterna believes they have a viable solution, despite the fact that their TCL-10 device has been replaced with a newer model. However, from the quote/offer we received, it does not contain the particular module we require. We also spoke with a firm called IOLON which may have a solution that more closely matches our needs, at least regarding the tunable channel filter. We are hoping they have a good amplification solution as well.
CISCO responded with support to our efforts on building a prototype lambdaMON, and we can use all the gear at CENIC to build such a system. The only thing we lack is a tunable 50-GHz channel filter.
~ Upgrades, troubleshooting, and maintenance on the PMA servers and infrastructure
We resolved the IPMI remote control issue with the SuperMicro servers after an email exchange with the company. We now have a prototype that will work for other SuperMicro systems, which will be used for the Purdue monitor and the NYSERNet GigE monitor.
We are very pleased with the new machine configuration which we now have in place. It has lots of horsepower and disk space at a low price point, is 1U, and also has full CDMA time support for PC and DAG cards. nai-p-amp in Miami is the first of these to be deployed.
The new pma server was made operational. As requested by Victor Hazelwood, the SDSC security chief, we worked with the SDSC folks to resolve the problem of secure access to the HPSS from the new pma.nlanr.net machine, which is FreeBSD AMD64 for which no HSI client is presently available. After discussions with Mike Gleicher, the IBM consultant assigned to SDSC (who is the developer of the HSI HPSS interface), he volunteered to compile the code on the new server to create support for the HSI on the AMD64 platform. We made arrangements to retain the access to our backup infrastructure before we transition fully and until the HSI issue is fully resolved. The interface, which Mike helped to fine-tune, was put in place, but the first trial produced errors. We need to have a closer look at all the configurations, especially Kerberos.
Early in August, the new pma.nlanr.net data server began suffering from some unusual characteristics causing it to stop working for no apparent reason. This occurred twice. We added an additional drive (40GB) for backup purposes and reworked the nightly procedures. Later in August, the server began crashing repeatedly and inexplicably. The problem cleared and then reappeared and cleared again in September. A highly likely explanation is that there are issues with SMP and locking support in FreeBSD 5.2, which should be overcome with FreeBSD 5.3. We are watching it closely.
To improve security, as the pma.nlanr.net server is becoming a more general purpose analysis machine with HTTP access from the outside, CGI scripts, etc. we prepared, configured, and took online a new PMA Sphinx machine, "strangler." It will be the access gateway server to PMAMONs in the field. We will also be restructuring the security/access to PMA systems a bit in the next period, which will include support for some of the real-time work that is already in progress.
We are now using a symbolic alias ntp.nlanr.net due to the NTP support problem discovered during the collection of the new data set from all monitors at IPLS. We began spreading the configuration among the rest of the PMA infrastructure.
SDSC asked us to relocate the NLANR hosts at SDSC to a different address space. This included IP address migration for the PMA server SDA monitor, and all the clients in the field.
Support and troubleshooting, existing PMA measurement sites:
Both Tel Aviv Univ. (TAU) and APAN (APN) are decommissioned sites, now listed with the historic deployments, on the PMA Sites page. The PMA monitor from the Tel Aviv University was returned this period. It is available for deployment to a new site.
Linda Winkler and Alan Verlo of the Univ. of Illinois, Chicago informed us that the connection at the SBC site in Chicago was no longer there. Alan is returning the OC3 monitor from the APAN site.
Please see the Appendix (last page) of this report for an information table on PMA individual site activity for the quarter - Daily Web log summaries for the last day of each month (July, August, and September).
~ Progress on the reimplementation of AMP and the development of a new testing architecture
A Perl AMP data interface was begun and completed (to the same extent as the PHP one). The C code was rebuilt to pass a set of generic data types that can be used by PHP or Perl, and have the PHP recoded as a thin interface which uses that. The Perl interface was coded to open the AMP "database" and to get successive items from it. Currently the main use of this is for Warren Matthews' Webservices interface, but we anticipate that there will be others.
We began and made excellent progress on the IPv6 code in amp. An annoying inconsistency between how IPv4 and IPv6 are handled from the programming end was corrected. This actually made things easier in IPv6, but also increased the differentiation between the code handling IPv4 and IPv6. We implemented the icmpTest and the traceTest, and tested them using the loopback interface. Then we connected an AMP machine to the IPv6 network in order to test the code. This showed a couple of problems with the traceroute test which were corrected. Both the ping and trace tests now work with IPv6. It still needs to be tested on a real IPv6 network, but the code is in good working order between two machines, and now compiles on Solaris. Fixed numerous bugs and split out constants etc. into configuration files.
After a bit of struggle setting up the test-bed, we did some testing between two fully functional IPMP machines (unfortunately we were not running the latest version of IPMP). Additional testing of the IPMPtest code will be performed, after updating the machines.
Good progress was made on adding most of the front page and destination selection page. Worked on the core of the weekly page and a structure that supports adding display modules for different test types.
The core of a generic map system was done that accepts arbitrary maps (could be anything from the world to a campus). It was tested with a couple of NZ maps and the existing AMP world map.
The code for dealing with sub-tests was finished, including debugging a tricky little PHP extension problem related to the PHP symbol table.
Great progress with dgraph (finished with it for now). We added the ability to do histograms and the IP hop count graph, as well as their associated html map code. Some of the code was reworked for better portability, which allows Tony to compile it on his machine as well. Memory leaks and pointer issues were also cleaned up. While adding an events graph type to the code, a design flaw was discovered; fixing it required rearrangement of some of the code. Time Testing of both workability and graph accuracy took place. Some major restructuring was performed, which yielded a little bit better performance. Comments and documentation were further developed. Testing was done to make sure that it will compile on various machines; it compiles on Linux, FreeBSD, and Solaris. We started using this new version of the graphing tool.
The graphic display code for Amp PathVis was added to dgraph. The eventual other part of this program is currently in dtrace. We need to determine how we are going to implement this functionality. Amp PathVis calculation and path identification will be done in PHP.
A database management schema (for all site, contact, test meshes, scheduling information, etc.) for the AMP machines was developed. http://byerley.cs.waikato.ac.nz/~tonym/tmp/schema
Work continued on the ampcentral code and it is nearly set up for autoconf/automake.
The capability to do a general test with the software is being added (AMP generaltest); it is mostly finished.
We added exponential backoff (to a limit) on failure to resolve the DNS name of a collector because of boot sequence problems. We changed the code to usr /dev/urandom, if it exists, to seed the random number generator.
We investigated a problem with the AMPlet code which the CRCnet people discovered when they ran it on their small solid state nodes. It seemed to be running the machine out of memory. After some initial experiments it appeared that the code may indeed have a slow memory leak. We fixed the leak, and also made some other changes to the configuration the CRCnet people were using for their machines. After testing, we did a new release of the AMPlet code - just for CRCnet -with these fixes.
The IPMP code was updated to the latest IETF Internet draft status and was included in the above release. This is not a public release because after feedback from other users, there are additional things that we want to look at (and potentially change) before a public release.
A few mods were made to the code to support biscuit PCs (used by CRCnet), the most significant one being an error and warning post processor that summarizes repeated errors rather than filling logs with them when something goes wrong.
The new ampz- (New Zealand AMP) mesh is running the new code and we have an experimental (and somewhat raw) version of the new central code and Web pages running. This is also running on CRCnet.
Several additional small fixes of the AMPlet code were done for the CRCnet wireless network. One particularly interesting issue is that the tests were having a big impact on the network because they were all running at the same time. We have always known this was a potential issue and the existing system randomizes the timing of the test somewhat. We had not put that code into the new system until we hit this problem. However, what is particularly interesting is the extent of the problem on a wireless network (100s of ms). In this case, many of the links are long point to point 802.11b links; we believe the problem was worse here because they are half duplex and tests were turning up at both sites of the link at just the same time. So, we added some simple linear perturbation code. In the future, we plan to also add Poisson distributed delays.
A problem with email on the AMPlets was sorted out. Sendmail was unable to resolve DNS names and was sending thousands of syslog messages per AMPlet to amp. It was an odd situation because everything else could resolve names properly. Matthew suggested a reboot which, to our surprise did clear the problem. Therefore, an expect script was written and run to clear the mailq and reboot all the AMPlets. This was sad in a way, as many of them had close to two years uptime. They had not been down since we last upgraded the OS.
Worked on some spreadsheet testing to determine if there exists a spreadsheet that can handle six months worth of round trip time (RTT) data, which apparently, they can.
~ New (and developing) strategically important measurements and deployments
Site amp-cnic (China Computer Network, Beijing) was connected this period. The machine has been on site for several months, but the site was undergoing changes and the machine could not be connected. When site people connected the monitor, we walked them through the edits to bring it online, and finalized the connection of the AMP monitor. We followed-up with runs of the system manager process to initialize the monitor, distribute the international list file to all AMP monitors, and start data collection. The monitor is now up and collecting data on the international mesh.
The new ampz- (New Zealand AMP) mesh was started this period. It currently has 5 operation machines with another two shipped and several others waiting until this initial deployment is working.
University of Nevada at Reno, amp-unr, request received, prepared and shipped, installed, initialized, and collecting data.
The four remaining AMP monitors for the Internet2 backbone locations prepared and shipped. These were for the Internet2 sites at Houston, Kansas City, Seattle, and Los Angeles. At this time, Chris Small has a total of seven monitors to be installed on the Internet2 backbone.
With some assistance from us, AMP was deployed on the CRCnet (NZ wireless network). Please see section above for details.
We communicated with Brian Court of CENIC. Brian is preparing to submit requests for seven monitors for the CENIC backbone nodes. We will be working with Darrell Newcomb to implement this installation.
Chen-en from the HPNC and TAWREN in Taiwan has plans to deploy 11 AMPs as part of a local mesh.
At the end of the month, two additional AMP monitor requests were received, one from Iowa State and another from U. of Washington.
10GigE AMP ~
Progress in the development of a 10GigE AMP continued, albeit a bit slowly. There appear to be only three manufacturers of 10GigE NICs: Intel, S2io, and Nortel, each of which had been having design an/or manufacturing issues, making the cards very difficult to obtain.
The AMP team decided on evaluating the S2io 10GigE network interface cards. We worked with Dan Hatton from S2io on getting the 10GigE xFrame interface cards. S2io is willing to loan cards, so far for a 30 day trial period. We are in talks with them, including regarding price, and have explained our plans and the goals we wish to accomplish. We were told that all available cards are the Short Range 850nm versions. Tony believes these will be fine for testing.
In preparation for testing the S2io 10Gb cards, new SuperMicro 6013 P-Ts machines were ordered and prepared. (They can also be used later for Gigabit PMA monitors.) The cards arrived and were installed, however there were problems. Little information about the S2io 10Gb cards was disclosed before their release, so configuration plans were not accurate. After receiving further instructions and information, assembly of the S2io 10GigE NIC card test platform was completed. We began performance testing the 10GigE interfaces, slowly increasing the speed, tuning the test and the OS, as required.
~ IPMP
Work was done on the BSD kernel implementation, removing unnecessary timers from IPMP flow records. Instead of each flow having a timer associated with it, there is a timer recording when the next flow will expire, and a list of flows in order of expiry. Issues with OpenBSD/NetBSD/Linux portability were also handled.
The IETF IMRG working group released a review of IPMP, posted to the IMRG mailing list. We wrote a detailed response; it is posted at:
http://www1.ietf.org/mail-archive/web/imrg/current/msg01105.html
~ IPv6 and IPv6 Scamper
Scamper was publicly released in early September. http://www.wand.net.nz/scamper/
Many activities relating to Scamper took place this quarter:
A new set of file-writing routines were created that record much more detail on what Scamper saw. A part of the problem is how to deal with PMTU data. Briefly, Scamper remembers MTU data it has seen to specific points given a path and does not re-probe, so a file format is needed that reflects this. Also, Scamper will try to discover PMTU when a router in the path does not send a fragmentation required message, so the file format has to record what MTUs it tried and how often. This becomes very complex when paths are load balanced.
Integrating the new file format and trace structure into Scamper involved untangling the old structure that kept both data and state into a structure with just the data; the state is kept elsewhere. Time was spent optimising the code a bit. Instead of storing all trace objects in a single linked list, they are stored into a splay tree (a self balancing tree that brings commonly referenced objects to the top of the tree) for fast searches, and stored into applicable probe/wait/pause/done queues for scheduling.
Scamper can now solicit more than one response per hop, which allows Scamper to collect more RTT measurements to each hop. Scamper also collects the source address used to transmit probes towards a destination, and the TTL received in a response packet, which can be used to guess at the reverse path's length. All address list details were moved into a separate file, and some code added to allow for per-trace parameters, such as the number of attempts to be made for each hop, the trace method (udp/tcp/whatever) etc.
The new Scamper_file API was imported for writing traces, and it was decided to use an API to record PMTU discovery alongside a traceroute. We also have a solution to the problem with Linux netlink returning the PMTU of the path rather than the MTU of the outgoing interface.
A bug was found in Scamper's logic that caused it to send probes at about half the specified rate. By breaking down large source files into units and generally tidying up the code, it is now easier to navigate and reuse.
The bottleneck that caused Scamper to be limited to sending HZ packets per second was removed. HZ is 100 in most operating systems by default, which meant Scamper was limited to sending 100 packets per second.
Used dmalloc to track down abuses of malloc in Scamper. dmalloc is a very useful library. One instance was found where there was writing beyond the end of a piece of memory, and there were a handful of mallocs that were not freed. Additional work was done on the file writing code.
One of the highlights of the period was adding MTU searching to the PMTU phase. If Scamper does not get a reply for a large packet, then it uses a table of L2 media types and their MTUs to decide on a smaller probe size to attempt. When it does get an answer using one of the known media MTUs, it tries a packet one byte bigger. If it does not get an answer, it has inferred the PMTU to the end host [or the next PMTU bottleneck]. If it does get an answer for the packet, then it does a binary chop to probe the PMTU.
Code was added to do a PMTU search to figure out the largest packet that can be sent along a path when an ICMP Fragmentation required message is not returned. A table of fairly common media types to aid Scamper in quickly finding the PMTU was created. This makes it more dynamic, in that it will not try lesser used media types, as learnt by Scamper. For example, when probing from a machine with a 4470 byte interface MTU, Scamper tries 3 or 4 other media MTUs before probing the 1500 byte Ethernet MTU, which is the most likely media case. Code was also written to allow a list of IP addresses to be passed on the command line, so that it can be used more like traceroute on the command line.
Code to do a TTL limited binary chop of the segments in question in order to isolate the node that is not returning the ICMP Fragmentation required message was written. Also added was a TTL limited search to isolate the hop[s] where packets are silently dropped. Matthew is quite pleased with the code, as it was not trivial to add, and is something we feel is reasonably original.
traceroute from 199.109.33.1 to 129.82.201.9
1 199.109.33.254 0.744 ms [mtu: 4470]
2 199.109.5.86 14.041 ms [mtu: 4470]
3 199.109.5.53 26.454 ms [mtu: 4470]
4 199.109.6.2 22.627 ms [mtu: 4470]
5 199.109.2.2 35.079 ms [mtu: 4470]
6 198.32.8.77 38.690 ms [mtu: 4470]
7 198.32.8.81 47.683 ms [mtu: 4470]
8 XX 59.337 ms [mtu: 4470]
9 XX 63.797 ms [mtu: 4470]
10 XX 61.082 ms [*mtu: 1514]
11 XX 60.909 ms [mtu: 1500]
12 129.82.201.9 61.541 ms [mtu: 1500]
The above traceroute with Scamper shows that the path is 4470 bytes all the way to hop 9. Hop 9 should send an ICMP fragmentation required message for 1514 bytes [Ethernet Max] but does not. Hop 10 sends an ICMP fragmentation required message for 1500 bytes, However, Hop 9 shields that message from being sent until a 1501 byte packet is sent.
We moved all the code related to select out of scamper.c itself, which tidied the main loop quite a bit. Also added support to kqueue for systems that support that. (kqueue is basically a scalable version of select.) We found that the interaction between kqueue and BPF BIOCIMMEDIATE file descriptors is different across the BSDs, and so disabled kqueue on Mac OS X. Also finally added code so that Scamper will abort tracerouting a path to a non-responsive target.
With Jamie Curtis (WAND), we discussed using Scamper to discover the topology of the NZ Internet before (and after) the Telstra de-peering takes place.
Finished writing the code for Scamper to obtain transmit timestamps using BPF. Added code for Scamper to use PF_PACKET under Linux to obtain the same thing. Looked briefly at how the transmit timestamps differed between BPF and gettimeofday in userland.
Some potential improvements were seen on slower systems in the order of a millisecond or two, but the actual difference for most timestamps on a FreeBSD 4.10 Athlon 1.2 with an Intel pro100 was in the order of 10-15 microseconds, and 150 microseconds on a FreeBSD 4.10 p3-800 laptop with an Orinoco wireless card. We anticipate that the importance of the code might increase when Scamper's PPS rate is above 100 and/or the local network is busy. Note that the network interface on all of these systems was otherwise idle when doing these tests.
In the process of writing that code, we noticed that FreeBSD's BPF generates a timestamp when each packet passes the filter, so if Scamper gets a packet from BPF after tcpdump does, Scamper's transmit timestamp will be slightly after tcpdump's timestamp. We sent a patch to FreeBSD to generate a BPF timestamp just once before BPF does any filtering. http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/71711
A bug in libpcap/bpf during the implementation of similar code in Scamper was also noticed; we submitted a small patch, which got accepted. http://cvs.tcpdump.org/cgi-bin/cvsweb/libpcap/pcap-bpf.c
There was some talk on NZNOG about PMTUD blackholes. We used that to point NZNOGers at Scamper. Received a few useful emails back, including someone who tried to compile it on an ancient version of Linux without some IPv6 headers. We managed to get it compiling/working. The WAND Web site took about 90 hits from the link. http://list.waikato.ac.nz/pipermail/nznog/2004-September/008729.html
Code was written to enable Scamper to read packets from live pcap devices. The idea is that RTT values will be more accurate.
~ Upgrades, troubleshooting, and maintenance on the AMP servers and infrastructure
The failed disk, disk7 in the AMP server disk array, which had to be replaced at the end of the quarter, failed again. The data restoration to the replaced disk was completed and data collection was restored on both AMP and VOLT. A disconcerting element of this is that the am_slave process for collecting data did not halt. That event normally triggers a message warning that am_slave down message. Shortly afterward, the same disk failed in the AMP Server Disk array (for the second time in two weeks). It was decided that instead of installing another 36GB disk, a return to the 18GB disks might have a positive effect on this chronic problem. We had previously shifted to using 36GB drives for the added capacity.
In light of the data disk failures there was an even greater need to phase out the old AMP and VOLT servers. Following his attendance at the Joint Techs meeting, Tony visited SDSC. During his visit much progress was made in planning the transition to the new AMP and VOLT servers. Tony and the AMP team discussed his plan to store AMP and VOLT data on the RAID array in the new servers using a NFS mounted system.
Testing and transition to the new AMP servers (AMP2 and VOLT2) ~
During preparations, a major problem was discovered with one of the machine's system boards. Following vendor repair and a recompile of the Linux kernel, it was reinstalled and placed online. Both the new and old AMP/VOLT servers were interconnected on the 10.28.5 network; thus allowing the RAID array in the new servers to be NFS mounted to the old AMP/VOLT servers.
Unfortunately, the transition to the new server array did not go well. We copied AMP's data to the new server and got it mounted back onto AMP so we can continue to use AMP to run the old software, while using the new server for the data, waiting until the new software is ready to go 100% live. After which, NFS was performing very badly. We think that this is due to the large number of processes writing to the file system (one for each remote site, plus others for the Web pages etc.). To improve the performance and resolve this problem, we tried many things over the quarter.
- We increased the number of nfsd processes which resulted in the server performing well, but the client was still badly bottlenecked. (A directory listing that is instant on another client takes 20-60s on AMP.)
- We fired up nfsiod with the maximum number of processes (20) and that help a bit.
- We hacked the source to increase the limit, without much apparent success, leading us to believe that the performance is being limited by other resources. Looked at the kernel, tuned the OS, and changed the startup profile of the workload.
- To deal with the additional load, we doubled the RAM in the machine from 2 GB to 4 GB, then increased that again to 6 GB.
- By the end of August, server AMP2 was collecting data for the AMP server and the modifications had improved the performance somewhat on the machine itself, unfortunately however, NFS performance to AMP was still too poor for it to be used for Web serving. The VOLT server which was already acting as the main data collector, also provided the Web serving.
- Replaced the failed disk in AMP and copied the VOLT data disks to AMP. However, during this copy process we encountered connectivity problems with the AMP server. At the end of the quarter, we were pursuing these anomalies.
- Since the performance of AMP2, running Debian Linux, was not as good as we had hoped for, we prepared to test the VOLT2 server running FreeBSD4.10. To support the test we installed an additional Intel GigE NIC in the AMP2 machine; and we installed 6 Gig of RAM and an additional GigE NIC in the VOLT2 server.
- During this transition, the continued functioning of the primary data collector remains critical. Therefore, the data disk fill on the VOLT server is being watched carefully to assure that archiving is started in a timely fashion in order to prevent a crash of the collector process, am_slave.
This very frustrating time where we tried a whole range of things without real success, brought us to the point where it appeared that an IDE RAID could not handle the load we are putting on it. The main problem is the number of random accesses that are happening. Of note, if we can make it work well: the eventual capacity of the new RAID arrays are expected to be such that archiving to the HPSS should not be needed for perhaps a year.
At the end of the quarter, the AMP data collection/server function continues to be the VOLT server as the primary and the AMP2 server as a backup. The latest solution (arrived at with input from Jamie Curtis of WAND) is to return the AMP monitor to service and reconfigure the AMP2 and VOLT2 servers to a different RAID scheme. The new scheme to be used is the RAID10, a combination of RAID1 and RAID0. This scheme will involve both disk mirroring and stripping.
IP address migration ~
- SDSC asked us to relocate the NLANR hosts at SDSC to a different address space. For AMP, this involved IP address migration for several servers and updating the 150, or so, active monitors. Additional network interfaces in the AMP, VOLT, and calorie machines were installed to facilitate the network transition.
- During the transition stage the machines were on both the .123 and .74 networks. In discussions with Jay Dombrowsky and Lyle Carlson, SDSC Netops, we arranged to be able to keep one of the machines on both networks through October. Or, until such time that all the AMP monitors shipped but not installed, are connected. This will provide for connecting to those machines from the .74 network to edit the /etc/hosts.allow file to allow connection from the new .123 network.
- The network transition of AMP and VOLT caused some anomaly with the connectivity to the HPSS. It appeared to be a TCP-wrapper issue within the HPSS. We worked with the HPSS people on this.
We also tried to install mysql on the calorie server for Warren Matthews, who thinks it will solve his performance problems. Unfortunately, the current ports will not build on calorie and the original ports from 4.7 refer to sources that are not easily available any more. We tried to debug the problem without success and tried several different versions also without success. We may have to upgrade calorie, but would rather not as we hope to retire calorie (and put that functionality on the new servers) in the coming few months. We finally got it to install, but it is not running yet.
We acquired ten additional AMP monitors late in the month. These units are 1 RU sixteen inch deep chassis. This chassis is two inches deeper that the last group. However that chassis has become unavailable. These units will be used to fill the existing requests and future needs for overseas sites, as well as for the CENIC system instrumentation.
Support and troubleshooting, existing AMP measurement sites:
Site outage remains at a very low level. A total of 33 remote sites in the AMP Network meshes received attention during this period; most were resolved and the monitors are again collecting data. Only four were still being investigated, or pending site action, at the end of the period. (Outages are considered "open" until the monitor is again collecting data.) Site details are available in the July, August, and September monthly reports. http://moat.nlanr.net/Reports/
33 problem sites: 29 resolved, 4 open - at the end of the quarter.
HPWREN uses a dedicated 24/7 monitor to assess the data exchange between the HPWREN network and the Internet on a machine called ittrack (Internet Traffic TRACKing). This is in addition to the monitors that are able to collect packet traces at all HPWREN backbone and some user sites. Initial data collection commencing June 23, 2004 was monitoring the Ethernet interface on the SDSC site of the router. This provided suboptimal data, as it is located on the Internet-translated site of the NAT function in the router. It was also exposed to SDSC multicast traffic, which showed up in the daily traffic analyses, until it was explicitly filtered out.
The present collection system (since September 20th) monitors the HPWREN site of the NAT router at SDSC, which connects the two 45 MBits/second backbone links towards Mount Woodson and Mount Soledad. The ittrack data collector uses a pair of Intel Pro/100B and Pro/1000 NIC cards on FreeBSD to gather data via the Berkeley Packet Filter (BPF). BPF is programmed to filter IP data only and is also responsible for packet arrival time stamping. The tshdump program labels the Mt Woodson and Mt Soledad connection as interface #0 and #1, respectively, in the TSH trace file.
For the daily analyses, the tcpdump trace was translated to TSH format, with various analysis programs generating summaries. Afterwards an address-encoded version was transferred to the pma server (long term storage for the data), for integration into publicly available traces. http://pma.nlanr.net/Special/hpwren.html
A BPF-based 'tshdump' program was written that provides two functionalities to improve and streamline the data collection process. It is able to simultaneously collect and collate packets from multiple interfaces (not previously possible), and also write in TSH format directly. This avoids a CPU/time-costly data conversion. As a result, tshdump is now simultaneously monitoring the two interfaces of the wireless links, and hence collects the traffic before being NAT translated, for more interesting analysis. Documentation on the behavior of the tool was written.
Enhancements were made to the daily network analysis software for the 24/7 HPWREN traces. It now reports applications and flow distribution, and also gives some timing information about how long the analysis took.
The HPWREN Cisco MIB statistics collection was changed to only get interface data every minute (plus collection time), and the whole tree once per day when it rolls over into a new file, done for both the Cisco routers and Cisco switches. Previously, it collected the whole trees every 1800 seconds plus collection time. This will give us a much finer time granularity while not collecting a lot of redundant an/or unnecessary data. However, this more than doubled our daily data collection volume for the routers, and it went up by more than a quarter for the switches.
A statistical analysis script for basic statistics for the top 10 ports in a trace file was written. A flow graphing tool was created which displays graphs of packet size, interarrival time, and window size, for different flows. The purpose of this analysis was to be able to profile a certain flow as being bursty, or not, so that when the QOS is implemented we will be better able to route different flows based on their determined profiles.
The weather pages were revamped and made to work with the new graphing and data collection systems. It was also made relatively dynamic. http://sensors.hpwren.ucsd.edu/
There was a problem with the HPWREN traces no longer arriving at the pma server in August. Eventually the HPWREN machine's log file showed EEPROM checksum failures for the GigE interface we were using. After some reluctance (trace disruption) we rebooted the machine, which fixed things for now. We are keeping on eye on the problem.
We suffered serious network problems due to a network firmware upgrades on some routers this period. We worked with Todd Hansen in resolving (and/or excluding) NLANR or HPWREN issues to be the cause. As a result, Todd arranged that direct fiber connections be implemented to bypass some unnecessary routing; this will avoid these same problems in the future.
Ronn Ritke traveled to New Zealand and Australia to meet with several of our research partners, as well as with Tony, Jörg, and Matthew; they discussed AMP software, AMPs in New Zealand and Singapore, AMP and PMA future planning, as well as other topics. He met with the head of Waikato University's Computer Science Department, Steve Reeves, regarding the measurement projects (past, current, and future) and the many successful collaborations between Waikato and NLANR/MNA. He also had a meeting with the Deputy Vice-Chancellor (for Research) to discuss past and future collaborations between NLANR, SDSC, UCSD and Waikato University. He toured ENDACE and met with Ian Graham regarding future projects. While there, he followed up with Claire in Auckland regarding the second FedEx charge to ENDACE.
In Australia, he visited the Royal Melbourne Institute of Technology (RMIT) in Melbourne and gave a presentation on NLANR/MNA's work at Melbourne University. Mark Sice of RMIT attended and Bruce Morgan suggested that an AMP at RMIT could be part of a local Australian AMP mesh; Mark is interested in pursuing this.
Bruce Morgan of AARNET has long been interested in AMP and developing an AMP mesh for AARNET. He is now in charge of the 10Gig backbone that is being installed and lit around Australia and has not had any time to begin implementation since the code became available. As a result of recent discussions between Bruce and Tony, who offered to help get the Australian AMPs up and running, and Ronn's meeting with him, we have a plan to get the Australian AMP Mesh online. Bruce will send us an email with a local contact to assist Tony.
With regard to collecting interesting data at the Indianapolis router node, Jörg started a longer email dialog with Harvey Newman's group (CalTech) on joint measurement opportunities, assisted by Matt Zekauskas (for which we are grateful). The purpose really is to have a chance to collect complimentary data to the existing sets when one of the bandwidth challenges is in progress, and to be in a position to study the impact of such giant flows to the overall performance of the router. (Matt also followed another lead with Sylvain Ravot, at CERN and CalTech, we have not yet received a reply.) Xun Su (who is also working on the recent AMPATH data with FIU colleagues) replied on behalf of CalTech, as follows:
-
I think having real-time packet capture/analysis capability is very valuable to our high speed transcontinental networks. Note that due to the extremely large amount of data coming out of a OC192Mon or OC48Mon, it is currently unlikely to instrument the network with 24/7 real-time packet trace available for analysis. This means as indicated in Jörg's earlier email that one has to coordinate in advance with a "big event" to capture just the right set of data. It is going to be very interesting, and in fact this is part of my work now with FIU students, to argument a network measurement/monitoring infrastructure such as MonALISA, with 24/7 "sampled" traffic information such as NetFlow records, and real-time "undiluted" packet tracing and analysis capability such as the PMA from NLANR. I think how to coordinate/instrument these somewhat complementary system is a worthy challenge.
W.H. Carlisle, Auburn University ~
Matthew received an email from him asking for the tools used in the IPv6 network troubleshooting paper he coauthored with Kenjiro Cho and Brad Huffaker. He is in the process of changing to IPv6 connectivity and wanted to use the tools we created to compare the performance relative to IPv4 before and after the change. Kenjiro supplied the scripts and visualization tools that drive Scamper, and Matthew supplied Scamper itself.
Andre Broido, CAIDA ~
He is interested in the 24/7 HPWREN packet traces; Hans-Werner and Jörg had a few email good exchanges with him and have given him access. We received good and useful feedback from him, as a result.
kc claffy, CAIDA ~
Tony and Maureen sent her some info about the AMPViz (divide and conquer tool). Hans-Werner met with her, discussed some NLANR questions.
David Moore, CAIDA ~
Suggested to Matthew that he enable Scamper to read packets from live pcap devices; they feel that as a result, the RTT values will be more accurate.
Margaret Murray, CAIDA ~
Ronn and Jörg had phone conversations with her about some possible future measurement collaborations and various project matters.
Xun Su, CalTech ~
An REU student (at CalTech) working with Jose Fernandez and Julio Ibarra at FIU (Miami, FL), looking at the PMA data. We are in dialog to collect some specific longer trace files for them to dive much deeper, and they are also interested in our upcoming real-time work. Jörg can see synergies developing here, reconfirming the positive impression that he already gathered during his visit to Miami, FL in May/June.
Brian Court, CENIC ~
Bud communicated with Brian, who is preparing to submit requests for seven monitors for the CENIC backbone nodes. Brian has assigned the task to Darrell Newcomb. Bud will be working with Darrell to implement this installation.
Jim Dolgonas, CENIC ~
Jörg and Ronn had a very good conference call with Jim; he is reviewing information on the lambdaMon and then will respond. Jim is the new CEO of CENIC, now that Tom West has moved full time to his position as the head of NLR. CENIC is very keen to have a stronger PMA involvement, but they only have 10 Gigabit equipment. Interestingly, our new architecture for DWDM monitoring may be an exact match to help solve this problem. We have quite a few opportunities during the coming months to strengthen this relationship.
Aaron Greusel, CENIC ~
Jörg had a brief exchange with Aaron as a follow-up to his recent visit.
Dave Reese (and colleagues), CENIC ~
Jörg spoke with him regarding Internet2's HOPI project and various updates on regional optical networks (RON) at the July Joint Techs Meeting, for details on the CISCO 15808 and 15454 used in NLR.
Bob Aiken, CISCO ~
Ronn discussed our active and passive measurements projects.
Chris Bruja, CISCO ~
Ronn had ongoing discussions about the 10GigE Amp and some future planning.
CRCnet (NZ wireless network) ~
Tony supported them with their deployment of AMP on CRCnet.
Nicolas Simar, DANTE and RUN, Universite de Liege ~
Jörg spoke with him at the July Joint Techs Meeting, on possible pan-European passive measurement activities and encouraged him to promote joint US-EU measurements, such as instrumenting the GEANT/Abilene peering points.
Ian Graham, Endace ~
Jörg met with him to discuss instrumentation for the emerging DWDM-based optical networks.
Jason Hurd, Endace ~
Jörg had final phone call with him and a very attractive offer for DAG equipment to use during the 3rd year of PMA. Jörg and Ronn are trying to determine with Alma how quickly we can move forward with a PO, which will ultimately gate how long it will take us to get the gear into the field.
Jose Fernandez and Julio Ibarra, Florida International University ~
We are in dialog with Jose Fernandez and Julio Ibarra at FIU to collect some specific longer trace files for them.
Warren Matthews, Georgia Institute of Technology ~
Ronn talked with him about recent changes in the GGF measurement standards and Ronn also thanked him in his Joint Techs presentation. Tony worked with him on some performance problems with the AMP WebServices Interface.
Greg Cole, NCSA, GLORIAD ~
Jörg and Ronn had discussions with him on the instrumentation of the OC3c links to Russia and China. He expressed interest in purchasing some PMA monitors for deployment and measurement on GLORIAD links. Ronn had a follow up phone call with Greg, who will be meeting the group from Russia in September and will work to move forward the installation of the AMP in Moscow, Russia. Ronn discussed potential measurements projects with him, and learned about their future plans.
Luke Fowler, Indiana University ~
Jörg spoke with him regarding Internet2's HOPI project and various updates on regional optical networks (RON) at the July Joint Techs Meeting, for details on the CISCO 15808 and 15454 used in NLR.
Rick Summerhill, Internet2 ~
Ronn spoke with him about the OC192 monitors at IND and sent text to him on OC192mons and the Observatory project.
Paul Love, Internet2 ~
He responded positively to our requests for an AMP presentation and PMA tutorial at the next Joint Techs meeting.
Matt Zekauskas, Internet2 ~
We have finished the description of the installation process at the Abilene IPLS location and have shared it with a number of people for proof reading and comments. Jörg exchanged emails with Matt Zekauskas on various topics, who has been extremely helpful to provide more details on the HOPI design process, technical specifications and further contacts to follow up on with questions, for which we are grateful. Ronn also followed up with Matt on the status of the Surveyor active measurement project.
Ivan Koga ~
Tony helped him to compile AMP.
Lawrence Wong, National Grid Office, Singapore ~
Larry Ang from Singapore introduced Ronn to Prof. Wong, who is the director of the National Grid Office. There is interest in hosting an AMP monitor at the National Grid Office site.
Debbie Montano, National LambdaRail (NLR) ~
At the Joint Techs Meeting, Jörg had several discussions with Debbie Montano of National LambdaRail (NLR), regarding the optical network infrastructures and the actual implementation of NLR to date. We have been working on questions of how to passively instrument emerging DWDM-based optical networks.
Jim Ferguson, NLANR/DAST ~
Ronn arranged demonstration times for SC2004 in November. They will give us a slot at the NCSA booth. Jörg will give a presentation on 10GigE application software.
John Towns, NLANR/DAST ~
Hans-Werner and Ronn had a conference meeting on future planning for NLANR.
Jon Dugan, NCSA ~
Jörg and Ronn have been chasing demo setups for SC2004 in Pittsburgh. Jon is in charge of the bandwidth challenge, and we had an email exchange with some detailed ideas about what we could arrange.
NREN folks (RRZE), Erlangen, Germany ~
Klaus and Jörg restarted the dialog with them regarding a trace captured at their end by Klaus last February. The data were thought to be lost due to disk failure but this was only a false alarm. We agreed to publish this trace on the PMA server. The trace was simultaneously captured in Erlangen and Leipzig and would hence allow for a one way delay analysis.
Bill Chang, NSF; Peter Arzberger, PRAGMA ~
Ronn had conversations with Bill and Peter on PRAGMA and international collaborations. The three of us will meet at the next PRAGMA meeting at SDSC. Bill asked me to let him know when the China AMP is online. Ronn met with him re the press release for the NLANR AMP in China.
Don Mitchell, NSF [retired] ~
Hans-Werner, Ronn, and Bud met with Don while he was in San Diego; giving him an overview of the project status and discussing various aspects of the project.
Dave Hart, NSF/OLPA ~
Mike worked with him to generate publicity about the AMP in China.
Bill Owens, NYSERNET ~
At Bill's prompting, Matthew sent an email to the Internet2 IPv6 WG mailing list outlining the AMP IPv6 project and the paper authored with Kenjiro and Brad. He gave Matthew sudo on a Mac OS X machine which enabled him to fix a small Scamper bug on that platform. He also posted a link to Scamper on the Apple IPv6 mailing list in response to a request for an IPv6 Path MTU Discovery tool on Mac OS X. http://lists.apple.com/mhonarc/ipv6/msg00282.html
Ville Aikas, Pacific Northwest Gigapop ~
He has been asking for permission to run throughput tests from their management subnet. Tony gave him permission on the condition he contact the sites concerned before doing large tests. Ville is now interested in seeing if we can upgrade the monitors at the set of sites he is interested in to GigE. The largest impediment to that is probably getting the sites to provide GigE, and hopefully 9kMTU, capable ports. Tony suggested that he make the initial approach.
Eric Boyd, Pipes Project ~
He will contact the Mona Lisa project about using the AMP data.
Teri Simas, PRAGMA ~
Ronn made plans and assisted her with PRAGMA meeting preparations at SDSC.
Bill Cleveland, Purdue ~
Jörg had a long phone conversation with Bill. They are both keen to get the PMA system going at Purdue, but appear to have no luck in encouraging the local sysadmin to tackle the problem with an optical power meter. Bill is looking for a good data source to start teaching in about two weeks, and after some searching we have decided to focus on the Leipzig-I and Leipzig-II data sets.
Charlie Knezevich, SDSC ~
Charlie has applications involving distributed site processing and data access. He is keenly interested in involving NLANR/MNA to help with network measurements to support his distributed processing applications. This is in the very early talking and ground breaking stages but there may be considerable potential.
Susan Rathburn, SDSC ~
Requested some text on international collaborations for Fran Berman's meeting with new UCSD Chancellor Marye Anne Fox. Ronn put together and emailed to Susan two different-length summaries of NLANR/MNA international collaborations.
Vijay Samalam, SDSC Network Director ~
During meetings with Ronn, they discussed the lambda passive monitor and the OptiPuter project. Hans-Werner met with him, discussed some NLANR questions.
Taiwanese R&D network ~
Jörg had a longer talk and writeup with a representative regarding options for passive measurements and possible collaborations with us.
Saverio Niccolini, Telecommunications Institute of Ecole d'Ingnieurs du Canton de Vaud (EVID), Switzerland ~
Jörg answered a PMA-related Web inquiry from him. He works with Nick Duffield at AT&T Research and Marcello Molina at NEC Research in Heidelberg, Germany.
John Hicks, Chris Robb, and Caroline Carver, TransPac, Indiana University ~
In a discussion with John, by chance, Jörg mentioned that he had questions about details on the CISCO gear used at NLR. As a result, John invited Jörg to have a look at the 15454s Cisco gear that they are putting in place for the IU campus network and also the ETF link between IU and Chicago. John quickly arranged it; they received great support from Caroline and Chris. As a result of this meeting Jörg has relatively good confidence about opportunities to instrument modern DWDM networks, specifically using the recent CISCO gear.
John Hicks, TransPac, Indiana University ~
Ronn discussed potential measurements projects with him, and learned about their future plans. Jörg had a short follow-up with John regarding some results from the CISCO 15454 inspection.
Felix Hernandez-Campos, UNC ~
Approached Jörg about PMA anonymization procedures; he was pointed to the most up-to-date toolset.
Nevil Brownlee and Ilze Ziedins, University of Auckland ~
Tony went to Auckland to meet with them and some other folks from the Computer Science department. They talked about a range of things including some work they have been doing on calculating a metric called t-entropy that may be useful for clustering and event detection.
Klaus Degner, University of Leipzig ~
The Klauses (Mochalski and Degner) exchanged emails discussing ideas for real-time one way delay calculation in anticipation of Degner coming to work with us for four months.
Srinivas Kota, University of Massachusetts Lowell ~
He had read our "Network Performance Visualization: Insight Through Animation" paper in IEEE Communication Magazine and asked for a sample Cichlid server. Tony sent him the code for one of the AMP servers.
Pere Barlet, UPC Barcelona ~
Jörg received an email to continue our collaboration via placement of an OC48MON inside the Spanish R&D network.
Malathi Veeraraghavan, University of Virginia ~
Jörg started an email conversation on the placement of an OC3MON. She responded positively. Jörg and she are continuing efforts to find a good time for a phone call to discuss things.
HCI, University of Waikato ~
The group has expressed to Tony an interest in working on visual representations of network data after seeing one of our Cichlid visualizations.
UniLink, Waikato ~
Ronn's meeting with this group, which handles external funding at Waikato, was very productive.
Jamie Curtis, WAND ~
Matthew and he discussed using Scamper to discover the topology of the NZ Internet before (and after) the Telstra de-peering takes place.
Phil Dykstra, WareOnEarth Communications ~
Phil is a long-time research partner of ours. Currently, we are sharing experiences developing 10GigE AMP machines.
David Wetherall, Washington University ~
Matthew received an email from him asking for geographic location (latitude and longitude) of AMP machines. He wants a ground-truth database of IP addresses. We have the coordinates on our Web pages, but not the individual IP addresses of monitors. Matthew will write a script to get the data out of the system_info database and send it to Tony to pass on to David.
~ Papers, Presentations, and Conference/Meeting Participation
- Matthew Luckie and Tony McGregor. User Level Path Diagnosis with IPMP. Proceedings of the SIGCOMM Network Troubleshooting Workshop (NetTs), Portland, OR, Aug 30-Sep 3.
- Kenjiro Cho, Matthew Luckie, and Brad Huffaker. Identifying IPv6 Network Problems in the DualStack World. Proceedings of the SIGCOMM Network Troubleshooting Workshop (NetTs), Portland, OR, Aug 30-Sep 3.
SIGCOMM Network Troubleshooting Workshop (NetTs), Portland, OR, Aug 30 -Sep 3 ~ Matthew attended and presented. His talk went quite well. The troubleshooting papers prior to his paper seemed to build motivation for his work. Matthew is an author of two papers (primary/first author on one and a coauthor on another) accepted; they only accepted 13 papers total (from 36 submissions). http://www.acm.org/sigs/sigcomm/sigcomm2004/netts.html
PRAGMA7, SDSC, La Jolla, CA, September 15-17, 2004 ~ Ronn attended and also participated in preparations beforehand. http://www.pragma-grid.net/pragma7/participants.htm
July Joint Techs Meeting ~
- Ronn gave a presentation which covered AMP achievements and outlined 10GigE monitor "firsts" as a segue to Jörg's talk on PMA.
- Jörg presented a talk on "Monitoring the 10 Gigabit Abilene Network," which included an insight into the actual hardware used and some of the first results of our instrumentation at IPLS. The feedback proved it was well received. Jörg used the opportunity to talk to Steve Corbato, I2 and NLR, to discuss various ideas on further network instrumentation. He also approached Rick Summerhill about the idea of possibly instrumenting the MAN LAN (Manhattan Landing) exchange point with a 10GIGEMON. There are some technical hurdles, but it appears like a real opportunity. For most of the dialogs, Matt Zekauskas was either present, or was briefed later.
Klaus gave us a presentation on his real-time work during one of our weekly meetings to great interest and continued comment afterward, including some long email conversations with Jörg.
We worked on a strategy and preparations for SC2004. Initial planning and arrangements were made for Jörg to give a presentation on 10GigE application software. We arranged with DAST/NCSA to have a slot at the Alliance booth. We provided proposal briefs for presentations to be given at the SDSC and NCSA/NLANR booths.
Matthew is planning an IPv6 paper with Bill Owens of NYSERNET. They plan to submit to the CCR special issue on Internet Vital Statistics. They discussed exactly the data to collect, the tables to present, etc. Bill also suggested we link the IPv6 NetTs paper to the Internet2 IPv6 WG. He thinks it would provoke some interesting discussion on that list. So, we sent an email to the Internet2 IPv6 WG mailing list outlining the AMP IPv6 project and the paper ("Identifying IPv6 Network Problems in the DualStack World").
~ Documentation, Web Work, and Publications
A new issue of the Network Analysis Times was completed, it is a really good looking issue with the primary focus on the new AMP machine in Beijing, which is now capturing data. We used the press release for the primary article and an abbreviated version of the international white paper. As reported previously, Lana created a great new NATimes head banner that retains the previous look (recognition factor), as well as updates it. Copies were included in the attendee packets for the PRAGMA7 meeting which was held at SDSC. http://mna.nlanr.net/NATimes/NAT.5.1.pdf
The press release regarding the AMP monitor now collecting data was completed and released. http://www.sdsc.edu/Press/2004/09/091504_NLANR.html
We finished a paragraph about PMA for inclusion on the Abilene Observatory Project Web pages. http://abilene.internet2.edu/observatory/research-projects.html
PMA Collection and Use Statistics (Web logs) project ~
Enhancements, refinements, and additions continued, including:
- After RRDtools was successfully installed on the new PMA server, the existing scripts were fitted onto the new box, tested, and are now working.
- We have generated graphs for the HTTP/FTP/Combined Download traffic dating back to September 2001. We also have a running graph of up-to-the-day totals for the current month. In addition there is a script which updates graphs for the amount of data and the number of files each remote monitor transfers in a day; this script also updates nightly.
- Auto-detection of the monitors has been implemented and creation of the monitors pages has taken place. The script can detect if one of the current monitors has not returned any traffic for the current day, if so, it's graph is not displayed on the Web page. Work was done to ensure that gaps are being properly handled by RRDtools and displayed correctly. The absence of data is now shown as a gap on the graphs as opposed to a zero (as was the case before).
- A test script to make a current 2004 yearly graph was run, though this has not yet been made into a cron job to update nightly.
- Changed the layout on the monitors page so that each of the monitors' graphs are side by side (there are two for each site (daily number of files and total traffic volume).
- Created roll-over cron scripts to update the makefile/create the new pages. They were debugged after the September rollover (Aug 31 -> Sep 1).
- Timing issues were finally resolved, and the updates appear on the graphs at midnight local (San Diego) time.
- The Reports index page was updated and the layout redesigned. http://moat.nlanr.net/Reports/.
- Worked on an issue regarding the scaling of Y-axis values on the Web logs graphs. We want the actual number, but with larger numbers this is problematic given the limited space available. Decided to change the internal scaling and subsequent external representation of the numbers a bit, leaving most of the min/max values to the discretion of RRDtools, rather than forcing the parameter. This appeared to be a viable solution so we began integrating it across the scripts. Refinements may be needed.
- Data collection graphs for the monitor data that we have dating back to Oct 2001 were completed. The Sites index page was updated, including rotating retired machines to the historic section. The last column which had been a definite status column, was changed to Collection Stats, with links to the current month's graphs for each monitor. http://pma.nlanr.net/Sites
- We created a monitors data collection index page and linked it to the Sites index and the Download summaries index pages. http://pma.nlanr.net/Stats/monitorsIndex.html
- The Download Summaries page was redesigned; it is the index/front page for the Data Collection and Use Stats. We updated all the text bringing it up to date with what we have available. Also added a sample image to give users an idea of what the graphs look like. http://pma.nlanr.net/Stats/
Citings/Collaborations database project ~
Work began on a database to store and sort information regarding our Citings and Collaborations/Partners. This will serve as a backend content source for the related Web pages and for print needs as well. In-depth planning meetings were held regarding the approach. Originally, we explored a combination flat file, Perl, and m4 solution. After a conference call with Tony to get his input, we decided to change from the flat file and instead use PostgreSQL to manage all our information in one big database (a big improvement). We are focusing first on the Citings as that is less complex than the Collabs. We developed a list of parameters which each citing/citation will have (in order to create the several different Web pages). Lana began learning about PostgreSQL databases and Perl query scripts. One of the benefits that we expect from this project is that updating these pages (Citings and Collaborations) will become quite easy, especially to differentiate between AMP and PMA related activities.
We accomplished the initial goal of creating a prototype of each type of Citings page (there are three) and now have three scripts which query two tables. The Perl scripts create the htmlm4 content file (which will then be processed like all Web pages). We created a simplified sample dataset (10 citations, three meetings, 2 years, or so) of the Citings info to use for testing. Using this and the scripts, we can generate pages which look just like the current ones.
We created an ERD (Entity Relationship Diagram) for the database. We updated it per meetings with Klaus who has been helping us understand and learn more about database design. The goal now is to fully develop the ERD before proceeding any further. Once the Citings and Collabs data is accounted for, it appears that it will not be difficult to add completed text to be used for the reports and have it linked to both a date and subproject/project (and perhaps for other uses as well, such as boilerplate language) We are excited about the possibilities of this tool, once it is fully developed with good database design.
~~
Minor link error in the AMP template was corrected and pages updated. This process served as a great reminder of how well the m4/Makefile system is working; am very happy with it.
Jay Dombrowski (SDSC NetOps) asked that NLANR switch to a new subnet to allow for the expansion of additional SDSC assets. A strategy for the IP address migration was developed and implemented for the MNA servers (a new VLAN). There was quite a bit of manual configuration necessary. Information for the AMP and PMA machines is available in their respective sections.
A "media pitch" briefing was created for use by Greg Lund (SDSC External Relations) on our AMP and PMA, 10-gigabit, and international activities.
Matthew Luckie continues to take the lead on the AMP IPMP, AMP IPv6, and Scamper efforts. He attended SIGCOMM 2004, and presented a paper at the Network Troubleshooting Workshop (NetTs). In fact, he was an author on two papers (primary/first author on one and a coauthor on another) accepted by the SIGCOMM workshop, which accepted only 13 papers total (from 36 submissions). He has continued Scamper development (he is funded by WIDE to build upon his earlier work). Scamper was publicly released before the Network Troubleshooting workshop - http://www.wand.net.nz/scamper/. These days he is also working very hard on writing his thesis and finishing up his post-graduate study.
This quarter Chris Gross spent the majority of his time creating and implementing an automated Web logging system for the central PMA Server and for the remote monitors in the field. This system automatically updates and maintains itself, using RRDtools and Web pages for display, and PERL for the backend. The Web logs display various metrics about the amount of data downloaded and the amount of data collected for a given month dating back Oct 2001. http://pma.nlanr.net/Stats/
Lana Kennedy continued working with Maureen on a number of projects. In addition to assisting with reports and helping with tasks for Web pages, she worked with Maureen on the recent issue of the Network Analysis Times. For which, she created a great new NATimes head banner that retains the previous look (recognition factor), as well as updates it. She learned the InDesign program and did the layout, as well as choosing and modifying, as necessary, the accompanying images for the issue. She also created more 100-pixel navbar images for the new PMA Special Traces.
Her primary focus this quarter was the Citings and Collaborations database project. She and Maureen worked on developing a database system to update and generate Citings pages, Collaborations pages, and also to create a repository for the Latest Pings, and possibly boilerplate language. Lana learned PostgreSQL and Perl (to write scripts to retrieve the information). She was an active participant early in the process helping to shape the direction eventually taken. She accomplished the initial goal of creating a prototype of the Citings pages and writing the appropriate Perl scripts to generate the htmlm4 content files. Using these, we were able to generate pages which look just like the current ones. She also worked with Maureen to create an ERD (Entity Relationship Diagram) for the database.
At the beginning of the academic year, Lana resigned her position as student writer, in order to concentrate fully on her UCSD studies. We wish Lana great luck in school and in the future. She is a very quick learner and a pleasure to work with. We (all) will miss her excellent, thorough, work. [MCC]
~~~~~~~~~~~~
Staff notes:
Jeremy Kallstrom, our new AMP Software Engineer, began work with us mid July.
Klaus Mochalski, a visiting scholar from the University of Leipzig, came (back) to work with us from July 21 until September 15. He was continuing his work on real-time measurements and analyses (from his visit late last summer). An important part of this was the improvement and evaluation of a collection of real-time trace processing tools, which had been developed in Leipzig. He implemented a component to track TCP packet loss, which, along with routing information, can be used to detect points of congestion along an Internet path. He conducted some early experiments to prove the feasibility of this approach.
In August, he captured, anonymized, and analyzed a synchronized packet header trace at three different 10-Gigabit-Ethernet interfaces of Abilene's Indianapolis core router. Due to storage space constraints, these traces cover only 4.5 hours. He installed his real-time analysis tool on the measurement system to prove its ability to keep up with very high link loads. The tool continuously generates various statistics based on the observed traffic and stores them in a centralized database. A Web interface is used to retrieve these statistics and generate graphs. During such a measurement, the link rate increased from the commonly observed 3-4 GB/s up to 10 GB/s showing the tool was able to handle such a load. Through this Web interface, researchers can easily access current and historical statistical data at various timescales.
At the end of his visit, he started working on a real-time packet delay analysis algorithm. This is a problem much different from the statistics already implemented, where only data from one data collector needs to be processed. For delay calculation, packet headers recorded at two different - possibly geographically distributed -locations have to be correlated.
The MNA team hopes that Klaus returns again next summer.
- 30 -
Appendix
Daily Web log summaries for the last day of each month - 3Q 2004
| Sites
|
July 2004
|
August 2004
|
September 2004
|
|
|
Total from 15 sites: |
Total from 12 sites: |
Total from 11 sites: |
| |
136
Files |
5837.4
Size (MB) |
96
Files |
2659.6
Size (MB) |
88
Files |
3789.7
Size (MB) |
| AIX |
8 |
0.1 |
8 |
0.4 |
8 |
0.3 |
| AMP |
|
|
8 |
191.9 |
8 |
165.6 |
| ANL |
8 |
26.0 |
8 |
12.3 |
8 |
13.6 |
| BWY |
8 |
127.5 |
8 |
171.9 |
8 |
367.1 |
| COS |
8 |
123.8 |
8 |
378.9 |
8 |
416.0 |
| FRG |
|
|
|
|
|
|
| I2A |
8 |
178.5 |
|
|
|
|
| I2C |
16 |
2104.6 |
|
|
|
|
| I2K |
16 |
2017.9 |
|
|
|
|
| MEM |
8 |
4.7 |
8 |
24.5 |
8 |
40.2 |
| MRA |
8 |
359.2 |
8 |
743.6 |
8 |
1099.3 |
| ODU |
8 |
46.5 |
8 |
115.3 |
8 |
167.5 |
| OSU |
8 |
3.8 |
8 |
3.7 |
|
|
| PSC |
8 |
256.1 |
|
|
|
|
| SDA |
8 |
222.5 |
8 |
262.8 |
8 |
637.3 |
| TXG |
|
|
|
|
|
|
| TXS |
8 |
1.2 |
8 |
1.8 |
8 |
1.2 |
| UFL |
8 |
365.0 |
8 |
752.5 |
8 |
881.6 |
|
|
Sites not responding: 5
APN BUF
FRG NCG
TXG |
Sites not responding: 3
FRG NCG TXG |
Sites not responding: 4
FRG NCG
OSU TXG |
Top
2004 October 20
NLANR/MNA home page
|