NetQuest: Scalable and Flexible Internet Measurement

The Internet connects over 200 million hosts and nearly a billion users worldwide. How to better understand and thereby control this enormous, decentralized, and constantly evolving infrastructure is a major challenge faced by today's researchers, engineers, and network operators. As an essential means of achieving such better understanding, network measurement is becoming crucial to a variety of existing and emerging network applications, such as IP performance management, fault diagnosis, traffic engineering, content distribution, overlay routing, multihoming, and peer-to-peer applications. These applications often require the ability to continuously monitor the current network state and react to changes in a timely fashion. Despite considerable recent progress in network measurement, there are still several gaps towards fulfilling such requirements.

First, existing network monitoring techniques often focus on accurately measuring the properties of individual links or paths and therefore lack scalability. They rarely scale beyond 100 nodes. In contrast, large-scale network management applications often require the ability to monitor the whole network which may contain orders of magnitude more nodes. Second, robustness issues have not been sufficiently investigated in the context of large-scale network monitoring and inference. Large-scale Internet measurements are inherently noisy due to failures, measurement artifacts, missing data, and even malicious behaviors. How to draw meaningful conclusions based on such noisy data remains a challenging problem that must be addressed in order to make large-scale network monitoring useful in practice. Third, existing techniques are often tailored to monitor specific types of network properties, and thus lack the flexibility to accommodate applications with even slightly different requirements and different performance metrics.

Our research aims to address the above challenges, and develop innovative techniques to design measurement experiments, conduct measurement, and analyze the data. We build our research on top of solid theoretical foundation in statistics, information theory, and optimization, and also leverage considerable practical engineering experience in measuring and managing large operational IP networks.