Scalable filesystem enumeration and metadata operations

Abstract

Systems, apparatus, and methods are disclosed for performing scalable operations in a file system, including POSIX-like file systems. Metadata entries in a namespace or directory tree are sharded across multiple file metadata servers. An enumeration operation, such as listing a directory, is parallelized across the multiple file metadata servers, while retaining standard functionality transparently to clients. Other enumeration operations include no-output operations such as changing file attributes or deleting a file, and cumulative operations such as counting total disk space usage. The parallelization is compatible with tree-level parallelization and storage-level parallelization. Disclosed technologies can be applied to other fields requiring scalable enumeration, such as database and network applications.