Research • 8/15/2023

Building A Learning-based Storage Ecosystem

After decades of development, storage systems today have been built into complicated ecosystems, which involve the development and deployment of storage devices, storage software, and application-level data stores. In order to rapidly meet the ever-increasing performance and efficiency requirements of applications, the storage ecosystem needs to recognize workload changes and adapt quickly across the full stack in a coordinated fashion. However, it is hard to achieve this with current human-driven system-building approaches, due to the complicated storage stack and the lack of effective coordination between the storage hardware and software. I proposed systems and architecture techniques to develop a learning-based storage ecosystem. This research received NSF CAREER Award ($593K) in 2022. I am leading my research group to conduct four research activities: (1) learning-based storage device development; (2) learning-based storage deployment; (3) learning- based storage software optimizations; and (4) architecture support for building a learning-based storage ecosystem.

To accelerate the manufacturing of efficient storage devices, we proposed to enable the development of customized storage devices for specific application types with automated tuning of hardware specifications (MLSys at NeurIPS’22), considering that manufacturing techniques for storage devices have become mature, and the demands for customized storage devices are continuing to increase. Therefore, we can enable developers to identify optimal device specifications with much less time and effort (learning to drive software-defined SSDs, MICRO’23). To maximize the storage resource efficiency in real-world deployments, we proposed to develop efficient storage harvesting and elastic storage management for multi-tenant applications using learning techniques (BlockFlex-OSDI’22, NVMW Memorable Paper Award Finalist). Therefore, we can achieve both improved resource utilization and performance isolation. To enhance the learning-based storage ecosystem, we revisited the storage hardware architecture for building learning-based storage drives, and redesigned the internal storage management with learning techniques to best utilize the precious hardware resources (i.e., SSD DRAM) of storage controllers. We developed the first learning-based flash translation layer for SSDs (LeaFTL, ASPLOS’23). This work was also invited to be presented at WDC Storage 2030 AI Focus.

Here is a list of our recent publications:

Video