Products

Commercial use cases for Lustre

Commercial use cases for Lustre

GUEST BLOG BY EDGAR HAREN…Lustre is a type of parallel distributed file system, generally used for large-scale cluster computing. Lustre has found its home within the high-performance computing (HPC) market segment, and has been commonly used for government-sponsored research in national labs, or by universities focused on data-intensive simulations, such as weather forecasting and climate modeling. Lustre is also used in the private sector by information-intense industries, such as defense, aeronautics and oil and gas. Because Lustre file systems have high-performance capabilities and open licensing, they have been implemented with more than 60 percent of the world’s super computers. However, with the explosion of mobile devices creating and consuming data, even commercial enterprises are looking to Lustre as a solution to help them analyze and extract value from their respective data flows.

According to IDC, data creation will grow to 40ZB (40 billion TB) by 2020 versus the 2.8ZB created in 2010. This aligns with IDC’s prediction of a doubling in smart connected devices up to 2B units by 2016. As a result of this data explosion, the growing demand for big data solutions presents an opportunity for Lustre to migrate into the commercial market and serve as a catalyst. OpenSFS, the consortium that promotes the Lustre open-source file system user community, states, “The opportunity still exists for Lustre to gain substantial adoption in commercial markets, particularly considering the dynamics surrounding big data – influenced deployments, which require I/O performance at scale beyond what organizations may have previously encountered.” In addition, Intel is helping to serve as a catalyst for this migration with the release of the Intel Enterprise Edition for Lustre software package, which enables Lustre to move beyond the lab and into the enterprise. Intel provides business customers with a commercial-grade version of Lustre optimized to address key storage and data throughput challenges of HPC-class computing in commercial business. While large-enterprise firms are seeing the value of HPC, very few small- to medium-sized firms in the US have investigated the value of HPC, and as a result, many US manufacturing firms have fallen behind their global competition due to the inability to leverage the benefits of HPC for the design and development of their products. Intel has defined this deployment — or rather the lack of deployment of HPC solutions for commercial mid-sized businesses — as the “missing middle”. 

Lenovo has supported a handful of SMB and public sector customers in creating affordable and high-performing Lustre solutions. Below is a reference architecture designed to accommodate a file system up to 250TB. The compute portion of the cluster is composed of 16 1U ThinkServer RD540 systems, while the storage portion is composed of six 2U ThinkServer RD640 systems and eight ThinkServer SA120 direct-attached storage arrays. The fabric backbone is based on Mellanox InfiniBand switches and interconnects. The software is Intel Enterprise Edition for Lustre, which provides a set of integrated management tools designed to simplify installation, deployment, monitoring and management of the cluster. This architecture has attracted interest because of its affordable nature, as we have seen up to 19 percent cost savings versus similar designs with competitive compute and storage elements.

I hope you found this blog informative, and that you consider Lenovo servers and solutions for your future Lustre deployments.

For more information on Lenovo’s servers and solutions for HPC, please contact Edgar Haren at eharen@lenovo.com. For more information on Intel Enterprise Edition for Lustre, view the Intel white paper, A New Generation of Lustre Software Expands HPC Into the Commercial Enterprise. Or you can view Intel’s recent webcast called, The Importance of Very Fast, Highly Scalable Storage for Today’s HPC.