Yang Wang, Manos Kapritsos, Lara Schmidt, Lorenzo Alvisi, and Mike Dahlin

Proceedings of the USENIX Symposium on Networked Systems Design and Implementation (NSDI) 2014.

View PDF or BibTeX.

areas
Distributed Systems

abstract
This paper presents Exalt, a library that gives back to researchers the ability to test the scalability of today’s large storage systems; an ability which, ironically, the very scale of such systems appears to have robbed them of. To that end we introduce Tardis, a data representation scheme that allows data to be identified and efficiently compressed even at low level storage layers that are not aware of the semantics and formatting used by higher levels of the system. This compression enables a high degree of node colocation, which allows large-scale experiments to be run on as few as a hundred machines. Our experience with HDFS and HBase shows that, by allowing us to run the real system code at an unprecedented scale, Exalt can help identify scalability problems that are not observable at lower scales. Resolving those problems improved the aggregate throughput of HDFS by an order of magnitude.