Performance and Reliability in Modern Storage Systems

Storage services form the platform on which widely-used cloud
services, mobile applications, data analytics engines, and
transactional databases are built. Such services are trusted with
irreplaceable personal and commercial information by users, companies,
and even governments.

The designers of storage services often have to choose between
performance and reliability. If the developer makes the system
reliable, performance is often significantly reduced. If the developer
instead maximizes performance, a crash could lead to data loss and
corruption.

In this talk, I describe how to build systems that achieve both strong
reliability and high performance. In many systems, reliability is
maintained by carefully ordering updates to storage. The key insight
is that the low-level mechanism used to enforce ordering is
overloaded: it provides durability as well as ordering. I introduce
a new primitive, osync(), that decouples ordering from
durability of writes. I present Optimistic Crash Consistency, a new
crash-recovery protocol that builds on osync() to provide
strong reliability guarantees and high performance. I implement these
techniques in the Optimistic File System (OptFS) and show that it
provides 10X increased performance for some workloads. With
researchers in Microsoft, I employ the principles of Optimistic Crash
Consistency in a distributed storage system, resulting in 2-5X
performance improvements.
Vijay Chidambaram is a Ph.D candidate in the Department of Computer
Sciences at the University of Wisconsin-Madison. His current research
focus is to ensure the reliability of applications in the rapidly
changing landscape of storage and cloud computing. Specifically, he
has contributed new reliability techniques in (local and distributed)
storage systems, and built frameworks for finding reliability bugs in
applications. His work has resulted in patent applications by Samsung
and Microsoft. He was awarded the Microsoft Research Fellowship in
2014, and the University of Wisconsin-Madison Alumni Scholarship in
2009.