[ Disclosure: I was invited by GestaltIT as a delegate to their Storage Field Day 19 event from Jan 22-24, 2020 in the Silicon Valley USA. My expenses, travel, accommodation and conference fees were covered by GestaltIT, the organizer and I was not obligated to blog or promote the vendors’ technologies presented at the event. The content of this blog is of my own opinions and views ]
A funny photo (below) came up on my Facebook feed a couple of weeks back. In an honest way, it depicted how a developer would think (or the lack of thinking) about the storage infrastructure designs and models for the applications and workloads. This also reminded me of how DBAs used to diss storage engineers. “I don’t care about storage, as long as it is RAID 10“. That was aeons ago 😉
The world of developers and the world of infrastructure people are vastly different. Since cloud computing birthed, both worlds have collided and programmable infrastructure-as-code (IAC) have become part and parcel of cloud native applications. Of course, there is no denying that there is friction.
Welcome to DevOps!
The Kubernetes factor
Containerized applications are quickly defining the cloud native applications landscape. The container orchestration machinery has one dominant engine – Kubernetes.
In the world of software development and delivery, DevOps has taken a liking to containers. Containers make it easier to host and manage life-cycle of web applications inside the portable environment. It packages up application code other dependencies into building blocks to deliver consistency, efficiency, and productivity. To scale to a multi-applications, multi-cloud with th0usands and even tens of thousands of microservices in containers, the Kubernetes factor comes into play. Kubernetes handles tasks like auto-scaling, rolling deployment, computer resource, volume storage and much, much more, and it is designed to run on bare metal, in the data center, public cloud or even a hybrid cloud.
The Kubernetes factor is bridging, gluing, assimilating cloud native applications to the underlying infrastructure components of compute, network and storage and more. The technology of disaggregation of these infrastructure components is maturing fast and provisioning and storage management are going to get very, very interesting in the next 1-3 years. I have written a few pieces of disaggregation and composable technology here:
The Storage Conundrum
Because of the nature of containers, the storage component for containers can be ephemeral or persistent. More and more containers are requiring persistent storage to address the growing demands of both enterprise and cloud native applications being stuffed into containers. Thus Kubernetes has to address this storage conundrum where data has to remain persistent and consistent for containerized enterprise applications. Again, I have deliberated this question in my previous blogs:
The emerging CSI (Container Storage Interface) is being widely accepted as the persistent storage volume adapter between Kubernetes and the traditional storage infrastructure piece which has the persistent properties. But honestly I still see the CSI methods as the best of the worst because there isn’t anything better out there. As cloud native applications become widespread, as they scale-out and be distributed across multi-clouds, multi-apps, into tens of thousands of access requests for persistent volume claims (PVCs), will the CSI framework be able to match the ambitions of Kubernetes and the world of containers? I have my whispering doubts about this “non-native” integration methods of persistent volume storage with Kubernetes/containers.
Fortunately, object storage is. It is native to the cloud, given its natural integration with S3 API, simple HTTP/S CRUD (create, retrieve, update, delete) access methods and newer operations such as S3 Select and Object Locking. Many newer distributed object storage technologies are already strictly consistent, fast, and secure.
MinIO and their friendly mc
MinIO is super high performance object storage platform which is also Kubernetes friendly. It was designed from ground up, as a private object storage server. Its performance is defined by unique minimalist design philosophy such as using Golang for the software and GoASM (Go Assembly Language) for its core components. It uses Highway Hashing for SIMD tasks with the Intel AVX512 and Arm Neon advanced instruction sets. It is simple, lightweight, clean and has an extremely small footprint. Anand Babu Periasamy (deduped to “AB”), the co-founder and CEO, mentioned to the Storage Field Day 19 delegates, that the MinIO server is only 44MB!
One of the MinIO philosophies is to bridge the gap between Dev and Ops. How do you make the object storage server’s functionalities friendly and attractive to both sides of the divide? How do you provide an amicable solution for the developers and yet remain loyal to the infra admins? It is their simple Unix/Linux shell-like interface called the MinIO client or mc for short.
Simple, unassuming, not flashy and almost Unix/Linux native. After a few hands-on, you might not even know they are just mc commands.
Thus MinIO, its commands and SDK, has been extremely effective in knitting both Dev and Storage Ops as one. And this has helped its proliferation and rapid adoption in many, many cloud native, enterprise and industry vertical applications. You can read its strong solution integrations here.
VMware® Project Pacific
VMware® may be the king of the hill for virtualized enterprise applications, but in a containerized, multi-cloud, cloud native application world, it is not. Naturally, VMware® wants to claim its stake in this new Kubernetes driven containerized world. Sure VMware® deep partnerships and integration with the Big 3 public cloud vendors – AWS, Azure and Google Cloud – but hybrid cloud is already the definitive model. AWS is already parachuting into the enterprise with Outposts, and Azure has been doing it with Azure Stack (Hyper-V virtualized) for a few years now. Google Anthos is making its move into the enterprise, and on the same side of the fence, Red Hat®/IBM® is challenging with Openshift™. Their threats to VMware® are real.
From a storage framework and operations perspective, VMware® response is their Cloud Native Storage (CNS) to their Project Pacific. Project Pacific, as quoted “empowers IT Operators and Developers to accelerate innovation by converging containers and VMs into VMware’s vSphere platform with native Kubernetes“. The VMware® CNS high level architecture is shown below:
[ Note: Red Hat®/IBM® Openshift™ also has a storage service called CNS (Container Native Storage). This is a containerized Gluster Storage Server and is different from the VMware® CNS ]
Within VMware® CNS, there are several storage interfaces including CSI (Container Storage Interface) and CPI (Cloud Provider Interface). But as far as object storage services are concerned, VMware has chosen MinIO as the resident object storage provider. The reasons are obvious. It is
- Fast
- Simple
- Clean
- Lightweight
- Proven integration
but the most significant reason is MinIO’s effective assimilation of Dev and Storage Ops. A perfect fit to VMware®’s Project Pacific plans. BlocksandFiles ran a piece about this integration.
[ Note: Deeper understanding about VMware® CNS can be found at Cormac Hogan’s blog here ]
Closing
Marc Andreessen of Andreessen Horowitz was right when he wrote “Why Software is Eating the World” in 2011. Today, almost a decade later, software powers everything. Programmability of the storage infrastructure, operations and management are landing on the hands of cloud applications developers now. To scale cloud services across the world, Dev and Ops have to become one.
The thoughts that go into MinIO are far reaching and yet, the company is still humble in its marketing. Perhaps, this DevOps paradigm shift with MinIO and the Kubernetes assimilation is subtle now, but the undercurrents are strong. Change is here …
Fellow Storage Field Day delegate, Penguin Punk Dan Frith was right too because in his blog, he said MinIO is not your father’s object storage platform.
Pingback: Storage Field Day 19 – Wrap-up and Link-o-rama | PenguinPunk.net
Pingback: Paradigm Shift of Dev to Storage Ops - Tech Field Day