Kubernetes Persistent Storage Managed Well

[ Disclosure: This is a StorPool Storage sponsored blog ]

StorPool Storage – Distributed Storage

There is a rapid adoption of Kubernetes in the enterprise and in the cloud. The push for digital transformation to modernize businesses for a cloud native world in the next decade has lifted both containerized applications and the Kubernetes container orchestration platform to an unprecedented level. The application landscape, especially the enterprise, is looking at Kubernetes to address these key areas:

• Scale
• High performance
• Availability and Resiliency
• Security and Compliance
• Controllable Costs
• Simplified

The Persistent Storage Question

Enterprise applications such as relational databases, email servers, and even the cloud native ones like NoSQL, analytics engines, demand a single data source of truth. Fundamentals properties such as ACID (atomicity, consistency, isolation, durability) and BASE (Basic Availability, Soft State, Eventual Consistency) have to have persistent storage as the foundational repository for the data. And thus, persistent storage have rallied under Container Storage Interface (CSI), and fast becoming a de facto standard for Kubernetes. At last count, there are more than 80 CSI drivers from 60+ storage and cloud vendors, each providing block-level storage to Kubernetes pods.

However, at this juncture, Kubernetes is still very engineering-centric. Persistent storage is equally as challenging, despite all the new developments and hype around it.

Kubernetes on-premises

Enterprises are presented with the Kubernetes dilemma. Many enterprise are not ready to do the lift-and-shift of legacy applications to the public clouds. There are enterprise applications developed on containers but faces concerns and risks of security and compliance. They are presented with choices, but these choices are limited to perhaps IBM® Red Hat® OpenShift and Ceph, or VMware® Tanzu with Cloud Foundations. Because Kubernetes takes a degree of technical expertises to scale, the talent resource pool is also limited.

Another option is to take up managed Kubernetes services such as Google Kubernetes Engine (GKE), Azure Kubernetes Service (AKS) or Amazon Elastic Kubernetes Service (EKS). However, not only does these managed K8s services are in the public cloud, their hybrid persistent storage plays are still very limited.

On-premises containerized enterprise applications running in on-premises Kubernetes pod demanding 100K IOPs with sub-milliseconds response time is almost impossible without the right persistent storage technology for Kubernetes in place.

Criteria for well managed Kubernetes Persistent Storage

As we shape the requirements for persistent storage for Kubernetes, we list several critical properties as the criteria for well managed persistent storage for Kubernetes.

• On-premises
  • Compliant with data policies and governance
  • Integration into the internal cybersecurity best practices
  • Congruent with established IT processes, practices and policies
• High performant and scalable
  • Physical bare metal to reduce layers and overheads
  • SSDs and future NVMe ready
  • Ability to scale into hundreds of TBs in capacity, and linearly in storage performance
• Durable and resilient
  • Highly available, continuous persistent storage service
  • Self-healing when a node or service goes down
• Simple
  • Minimal changes to existing server, network and storage infrastructure
  • Well known protocols like iSCSI
• Managed service
  • Closing the Kubernetes skills gap
  • Bespoke, white glove monitoring and support services
• Metered usage and cost control
  • Pay as you grow
  • Good price/performance and price/feature

The rise of managed bare metals

As Kubernetes evolve, another rapidly rising infrastructure play is running it on bare metal physical servers. The demands of higher performance to address new types of workload such AI, big and deep data analytics and machine learning, along with traditional enterprise applications are fuelling a more integrated and advanced Kubernetes platform that can take advantage of SmartNICs, GPUs and persistent memory technology such as Intel® Optane™.

Whilst there is apparent performance gain, the management and automation of Kubernetes and persistent storage on bare metal also present new and unknown challenges. Thus, using a managed service to operate, manage and monitor the persistent storage to Kubernetes on bare metal is more prudent, rather than jumping through hoops and hurdles to get it right. And the errors and mistakes in operating and supporting persistent storage on bare metal Kubernetes can be costly, especially when in-house skilled and experienced resources are extremely rare. Therefore, having plans to have managed services for persistent storage to serve bare metal Kubernetes platform may be the wisest choice to realize the benefits of scaling enterprise applications, and without the costly errors and risks.

Managing and controlling costs

At the end of all this, regardless about all the feel good points stated, the management of any organization will simply ask “How much is this going to cost over a period of time?“.

The OPEX model is definitely supplanting the traditional CAPEX model. Even so, the costs of building and operating persistent storage for Kubernees in a cloud and cloud-like model are unpredictable especially when uncertainties are abound. Removing unpredictable concerns such as capacity planning and sizing, hardware and performance testing, ensuring data recoverability and protection and many others will clear one of the most challenging obstacles for Kubernetes. This will eventually but surely translate into more controlled and manageable operating costs of persistent, giving the C-levels the best possible advantage in running the business – reducing unpredictability.

StorPool

My association with StorPool is not just for this blog. I have known of them since Storage Field Day 18 more than a year ago. I had several video chats with Boyan Ivanov, their CEO and Boyan Krosnov, CTO of StorPool after that. I have followed StorPool’s progress as the leading managed services provider for block storage. Recently, they chalked up a 40% year-on-year growth for the first half of 2020, with an impressive Net Promoter Score of 73%.

StorPool CEO and co-founder – Boyan Ivanov

StorPool already has a strong prowess in managed block storage services, and it is in its DNA to expand into the managed Kubernetes persistent storage services as well. Their deep technical roots reflect their capabilities to be one of the best in managed Kubernetes services in the business. Read about their Kubernetes offerings

• Fast Kubernetes Persistent Storage
• Persistent Storage for Scalable Bare-metal Kubernetes

The StorPool Storage Advantage

I explore the many facets of StorPool Storage, and its many features and functionalities. The criteria I highlighted earlier are well fitted to StorPool’s market offering. It is

• Fully managed storage solution with integration into Kubernetes Bare Metal
• A CAPEX Pay as You Grow model with manageable costs control
• Optimized performance, over 1 million IOPS with sub-milliseconds latency and balanced price/performance
• Easily to setup on-premises with simple network design architecture
• Native support of widely used iSCSI protocol
• 24×7 mission critical availability with durable persistent storage data service that triple replicated, and dual redundant network
• Built-in self healing and data protection policies
• Secure and meets data residency and compliance needs
• Consistency to existing enterprise IT processes and practices within the organization

StorPool Storage Persistent Storage for Kubernetes

These advantages will no doubt transform into an efficient persistent storage layer that will serve the Kubernetes pods and clusters, be it virtualized or bare metal, well. This is the StorPool advantage. It is indeed Kubernetes Persistent Storage managed well.