Software-Defined Everything with OpenStack, Midokura and Ceph

Co-written by Krzysztof (Chris) Janiszewski.

In a previous blog post on OpenStack, I discussed Lenovo’s mission of enabling customers to successfully deploy their cloud solution on our infrastructure using their favorite OpenStack distribution. Here, I’ll highlight the additional work we’ve done to enable OpenStack running in real production environments requiring the highest availability and reliability of the underlying physical infrastructure as well as the cloud services hosted on it.

Businesses today are under pressure to create an agile and flexible IT infrastructure to stay on top of users’ needs and innovate to keep pace with the market. Cloud has emerged as the de facto standard to address these agility and flexibility demands over the last five or so years. People have come to recognize the business value cloud offers through “on-demand” availability of resources, flexibility and reliability in infrastructure offerings, anytime/anywhere access, utility billing models and a high degree of automation. It seems that there is virtually nothing that cloud cannot do when it comes to IT these days!

If we turn our attention briefly to what is going on inside the cloud, we notice some interesting trends. Cloud is evolving ever more rapidly! While to the end user Cloud provides an abstracted view of the infrastructure to make it look flexible (and magical!), the underlying cloud hardware inherently is not so flexible! Hardware is designed to provide specific, fixed and finite number of functions. For example, a network switch has embedded software in it (called the operating system) that provides the packet routing and other network functions. This software is very “static”, meaning the built-in routing algorithms don’t change unless they are changed via an upgrade to the next version of the switch operating system. You cannot simply reprogram the switch on-the-fly to make adjustment to its routing so you can prioritize traffic for specific user workloads. This inherent “rigidity” limits how flexible you can be with your infrastructure on the outside. But that is today. On the evolutionary path to become really flexible and really agile with infrastructure, organizations and especially cloud providers (whether private or public), are looking up to “software-defined everything” (SDE) technologies – i.e. software-defined networking (SDN), software-defined storage (SDS), software-defined data center (SDDC) and so on.

The fundamental idea of SDE is to “detach” the core function of the hardware, in other words the “software controller”, and move it outside to a standard server system. This way the hardware essentially becomes a programmable device and the “core logic” resides on another server, and it’s purely software based. The hardware itself will be built to expose the programmable interface for the external software to “plug-in” its logic. For instance, in the network switch example before, the newer switches now implement a programmable interface standard called OpenFlow, which enables an external software-based controller to implement new routing algorithms.

OpenStack and Lenovo SDE

To pick up our conversation from the previous OpenStack article, it’s evident that OpenStack will drive a significant amount of innovation in the data center in the future. The push towards open source and community-driven technologies for managing infrastructure and virtualization is fueling a variety of new projects around OpenStack, which has now become a nice test bed for SDE.

Lenovo’s Enterprise Solution Services (ESS) recently conducted a successful proof of concept (PoC) with a customer to demonstrate the work we are doing in OpenStack and SDE. The PoC was focused on a production-ready, highly available, OpenStack implementation with SDN architecture using Lenovo networking and Midokura Enterprise Midonet software and OpenStack Ceph for the SDS component.

In the paper, Lenovo and Midokura OpenStack PoC: Software-Defined Everything, Krzysztof (Chris) Janiszewski and the team from Midokura demonstrate how we solve the networking complexity issues around OpenStack using Midokura Enterprise MidoNet software, which offers a distributed architecture that’s built to scale. By using a network virtualization overlay approach, MidoNet is able to provide Layer 2-4 network services, including routing, load-balancing and NAT, at the local hypervisor. MidoNet is also able to tie into OpenStack provisioning tools to provide seamless networking integrations.

While OpenStack is a software-based solution, it still requires physical hardware to operate, while providing compute, storage and networking infrastructure. In fact, using proper hardware is critical to achieving a successful OpenStack deployment. Proper selection of hardware will help ensure that reliability and performance metrics are met while helping reduce CAPEX and OPEX around the solution. Lenovo’s hardware offerings have many clear advantages when it comes to the three key areas of physical infrastructure. Lenovo offers high-performance, Intel processor-based servers to power the virtual machines. The ThinkServer RD650 has a unique storage configuration available with nine 3.5-inch and six 2.5- inch drive bays in the front and two 2.5-inch drive bays in the back so customers can achieve the right mix between cost and performance. When it comes to networking, Lenovo offers high-performance, low-cost 10Gb and 40Gb Ethernet networking solutions to provide connectivity for networking, storage and compute.

Cloud and enterprise organizations’ data needs grow exponentially and the classic enterprise storage solutions do not suffice to meet the demand in a cost-effective manner. Moreover, the refresh cycle of the legacy storage hardware lags behind x86 commodity hardware. The viable answer to this problem is the emerging SDS approach. Ceph, one of the leading SDS solutions, provides scale-out software that runs on x86 hardware and has the ability to handle exabytes of storage. Ceph is highly reliable, self-healing, easy to manage and open source. The PoC environment utilizes eight dedicated Ceph nodes implemented with the Lenovo ThinkServer RD650 servers with over 300TB of raw storage. Each storage node is populated with 8x 6TB HDD for object storage device (OSD) and 2x 200GB supporting SSD drives for journaling. The PoC white paper provides more detail on the Ceph implementation.

While OpenStack matures as the open-source cloud operating system, it also presents unique opportunity for organizations to experiment with software-defined technologies. Lenovo is quite actively engaged in SDE, both with our organic development and through partners such as Midokura. The white paper published from a customer PoC by our Enterprise Solution Services team demonstrates the breadth of Lenovo hardware and software technologies that are available to our customers interested in implementing OpenStack-based clouds. We will continue to enhance these capabilities and we hope our customers will benefit from them.