Depositphotos
During Open Source Week, DeepSeek made its Fire-Flyer Fire System (3FS) parallel file system publicly available. According to the company, 3FS reaches 7.3 TB/s aggregate read speed on its own server data clusters, where it has been used since 2019.
3FS is based on Linux and designed specifically for AI-HPC operations, in which GPU nodes continuously access multiple storage servers for LLM training. 3FS differs from other file systems by prioritizing random read speeds above all else and almost completely ignoring caching.
During training of artificial intelligence models computing units need constant access to random training data, and reading this data is a one-time process. Thus, the read cache is effectively useless and is mostly not used by 3FS. In fact, using a read cache during LLM training can be potentially harmful, as repeatedly reading the same data can damage the model.
DeepSeek previously statedwhich achieved 3FS performance of 6.6 TB/s and also ran training tasks in the background, adding another 1.4 TB/s of read throughput. By comparison, the competing Ceph file system first achieved read speeds of only 1.1 TB/s in early 2024.
Everyone who wants to try to try out the Fire-Flyer file system and its features, can download everything they need on the DeepSeek page on Github. If a file system is that good, it has the potential to be a hit.
Source: Tom’s Hardware