Stable-fs, the cutting-edge file system powering the ic-wasi-polyfill library, has just been enhanced with a new update, promising to significantly improve performance and provide developers with more flexibility. Serving as the backend for the wasi2ic tool, which adapts WASI applications to run on the Internet Computer, Stable-fs has undergone some substantial changes. The latest version, 0.7, delivers numerous improvements designed to make the file system faster, more efficient, and better suited for real-world applications.
The core architecture of Stable-fs was built to address the challenges of using file systems in the decentralised world of blockchain. Originally relying on a mix of stable cells and BTreeMap structures, the design was functional but had its limitations. Managing file systems with multiple components, such as folder structures, file metadata, and file content, required considerable effort and, at times, resulted in performance bottlenecks. The new updates to Stable-fs tackle these concerns head-on by introducing improvements to both file management and data storage.
One of the standout features of this update is the introduction of mounted memory files, which offer faster read and write speeds compared to traditional chunked file storage. Mounted memory files work by allowing users to configure specific files to act as hosts for virtual memory. Once this is set up, all file operations are redirected to the virtual memory, reducing overhead and speeding up the process. For use cases that involve large files with frequent random access – such as SQLite databases – this new feature is a game-changer. The implementation of mounted memory files ensures that users can access data at lightning speed, providing a seamless experience that was previously difficult to achieve in decentralised systems.
The flexibility of this new memory management system is another key benefit. Users can choose to transfer data from the mounted memory to chunked storage, synchronising the two for better long-term data management. Alternatively, the system allows users to mount a file and re-mount it after a canister upgrade, ensuring that no data is lost in the process. While the mounted memory comes with a few limitations, such as a restricted number of virtual memories, the performance improvements are undeniable.
For scenarios where large quantities of files need to be managed, the team behind Stable-fs has also overhauled the chunked storage system. By refining the chunking mechanism, they’ve reduced serialization overhead and search time, optimising file handling for various data access patterns. The new system introduces a BTreeMap that stores pointers to data chunks instead of the chunks themselves, drastically cutting down on memory usage and improving performance. Moreover, the implementation of direct memory access means that small portions of data can be read or written without the need for rewriting entire chunks. The data is only read or written once, ensuring greater efficiency in managing file operations.
An important aspect of the chunked memory storage improvement is the ability to configure chunk sizes. The previous default chunk size was 4KB, but with the new update, users can increase it to as much as 64KB. This change has a profound impact on performance: for instance, doubling the chunk size results in half the amount of required insertions for the same amount of data. With chunk sizes now configurable at various levels – 4KB, 8KB, 16KB, 32KB, and 64KB – users can tailor the system to their specific needs, achieving optimal speeds for their workloads.
In fact, with the 64KB chunk size setting, users can write a massive 100MB file in just under 1 billion instructions. The ability to fine-tune chunk sizes makes this update especially valuable for developers seeking to optimise their applications for different types of workloads. Whether it’s managing small files or handling enormous data sets, Stable-fs offers a flexible solution for developers working in decentralised environments.
Other new features in version 0.7 include the ability to store mounted memory within its host file, a feature that enables more efficient memory management. There’s also the added option to switch between older chunk storage formats and the newly implemented version 2 (V2) chunks. While the V2 format is expected to be the default moving forward, users can still opt to use the older format for performance estimation purposes if needed. Additionally, the system now supports sparse file functionality and allows developers to set a maximum allowed file size, providing further customisation options to suit specific project requirements.
The optimisations in version 0.7 are also noteworthy. For one, caching has been significantly improved, reducing the need for repeated metadata or chunk pointer lookups. This boosts access times, ensuring that file operations are faster and more efficient. Furthermore, look-ahead pointer caching accelerates reading and writing by reducing search times for the next available chunk. These improvements, combined with faster metadata handling and chunk storage optimisation, result in a system that is both more reliable and considerably quicker than its predecessors.
When it comes to performance, the differences between Stable-fs version 0.4 and the new 0.7 release are significant. Tests show a substantial reduction in the time required to write and read 100MB of data, with the 0.7 version performing 10 to 100 times faster in various use cases. Notably, the time it takes to write a new file has dropped from 14.51 billion instructions in version 0.4 to just 1.32 billion instructions in version 0.7. Similarly, reading files and overwriting existing data has become far more efficient, with improvements ranging from 10 to 100 times faster. The ability to handle 1KB segments across multiple files is also a strong suit of the updated system, significantly reducing the overhead when working with large datasets.
In one specific use case, benchmarking SQLite database operations revealed impressive results. For instance, creating 100,000 user records in a regular file took 2.35 billion instructions, but using mounted memory reduced that time to just 2.30 billion instructions, an improvement of 2.23%. Similarly, joint database queries and deletions were faster when using the mounted memory setup, improving by as much as 43% for certain operations. While bulk record additions didn’t show a dramatic speed-up, the real benefits of mounted memory were evident in operations requiring random memory access.
Despite these impressive performance enhancements, it’s important to note that Stable-fs is not yet production-ready. The system is still being refined, and while it has shown great promise in test environments, there are still areas for improvement. The development team recommends that the system be used for non-critical projects at this stage, with a focus on gathering feedback and identifying potential areas for further optimisation. However, the rapid pace of improvements and ongoing testing mean that Stable-fs is moving quickly towards becoming a fully production-ready solution for decentralised file storage.
The release of Stable-fs 0.7 is an important step forward for the Internet Computer ecosystem. With its new features, optimisations, and improvements, the system is better equipped than ever to handle a wide range of applications, from large-scale file storage to database management. By reducing overhead, speeding up data access, and offering greater flexibility, Stable-fs ensures that developers working within decentralised networks can build high-performance applications with confidence. With more updates expected in the near future, Stable-fs is well on its way to becoming an indispensable tool for developers in the ICP community
