Data Fabric was a big buzzword going back several years. I wrote a piece talking about Data Fabric, mostly NetApp®’s, almost 7 years ago, which I titled “The Transcendence of Data Fabric“. Regardless of storage brands and technology platforms, and each has its own version and interpretations, one thing holds true. There must be a one layer of Data Singularity. But this is easier said than done.
Fast forward to present. The latest buzzword is Edge-to-Core-Cloud or Cloud-to-Core-Edge. The proliferation of Cloud Computing services, has spawned beyond to multiclouds, superclouds and of course, to Edge Computing. Data is reaching to so many premises everywhere, and like water, data has found its way.
Edge-to-Core-to-Cloud (Gratitude thanks to https://www.techtalkthai.com/dell-technologies-opens-iot-solutions-division-and-introduces-distributed-core-architecture/)
The question on my mind is can we have a single storage platform to serve the Edge-to-Core-to-Cloud paradigm? Is there a storage technology which can be the seamless singularity of data? 7+ years onwards since my Data Fabric blog, The answer is obvious. Object Storage.
The ubiquitous object storage and the S3 access protocol
For a storage technology that was initially labeled “cheap and deep”, object storage has become immensely popular with developers, cloud storage providers and is fast becoming storage repositories for data connectors. I wrote a piece called “All the Sources and Sinks going to Object Storage” over a month back, which aptly articulate how far this technology has come.
But unknown to many (Google NASD and little is found), object storage started its presence in SNIA (it was developed in Carnegie-Mellon University prior to that) in the early 90s, then known as NASD (network attached secure disk). As it is made its way into the ANSI T10 INCITS standards development, it became known as Object-based Storage Device or OSD.
- [ November 2011 ] The future is intelligent objects
The introduction of object storage services 16+ years ago by Amazon Web Services (AWS) via their Simple Storage Services (S3) further strengthened the march of object storage, solidified its status as a top tier storage platform. It was to AWS’ genius to put the REST API over HTTP/HTTPS with its game changing approach to use CRUD (create, retrieve, update, delete) operations to work with object storage. Hence the S3 protocol, which has become the de facto access protocol to object storage.
- [ October 2013 ] Has Object Storage become the everything store?
Yes, I wrote those 2 blogs 11 and 9 years ago respectively because I saw that object storage technology was a natural fit to the burgeoning new world of storage computing. It has since come true many times over.
In the beginning … and now
There were many early detractors to object storage. Myself included. I was the EMC® product manager for IP Storage products between 2007-2009. I worked on the Centera, an early beast of object storage, and troubleshooting the PEA (pool entry authorization) file was a real pain. Centera went on to become Atmos, and now Dell® ECS.
During this time, object storage was going through the era of “cheap-and-deep”. Y0u want attract very cheap $/GB? Object storage was most likely a storage platform of choice for the bottom feeders. Object storage big, clunky and less agile to the choices we have today.
For me, MinIO was a true game changer in the world of object storage. Yeah, there were many challengers to the wooly mammoths of object storage vendors in the likes of Hitachi Vantara® HCP (nee HCAP), IBM iCOS (nee Cleversafe), Dell EMC® ECS, NetApp® StorageGRID. But it was MinIO which made object storag so simple, lightweight and lightning fast, and easily available through its generous open source model. Developers began to adopt the MinIO object storage everywhere. I love MinIO.
- [ January 2022 ] Celebrating MinIO
GigaOM®, the technology analyst firm made it clear to the industry that not all object storage technology platforms are the same. They have introduced 2 separate Radar reports on object storage, and as a result placed the object storage into 2 different categories – Enterprise Object Storage and High Performance Object Storage.
- [ Subscription required ] GigaOM Radar for Enterprise Object Storage
- [ Subscription required ] GigaOM Radar for High Performance Object Storage
Thank you note: Here I would like to thank Enrico Signoretti for giving me the complimentary subscription pass to GigaOM.
Here are the 2021 GigaOM reports for the 2 object storage categories side-by-side:
GigaOM Enterprise and High Performance Object Storage Radar reports
Object storage Edge criteria for edge-to-core-cloud
I want to reiterate that not all object storage technology can fit as a singular “data fabric” in this new edge-to-core-to-cloud architecture. Between the core to the cloud and vice versa, the availability and the maturity of object storage has been well executed and defined over the decade. The challenging part is data storage and processing at the edge.
Sure there are many inherent characteristics in almost every object storage technology. They are immutability, security, cost per gigabyte, flexibility and storage management. But at the edge, the ecosystem of end point and IoT devices present a new challenge to object storage. Here several critical criteria must be given further thoughts in order to make the entire edge-to-core-cloud architecture work cohesively.
- Performant – data ingestion rates can be very high at the end points and IoT devices
- Security – with limited security in most edge devices, data must be secure from the very beginning. Object storage provides that
- Lightweight – There are limited memory and storage resources available at the end
- Use in low powered and low bandwidth devices
- Resiliency – Reliability and availability are always a challenge. Object storage must have rapid recoverability powers
- Diversity – Many types of applications exist at the edge. Object storage must be able to integrate well with many of these non-IT (information technology), notable OT (operational technology)-based type of micro-applications.
The object storage ubiquity continues
In the span of 20+ years, object storage has enveloped many segments of IT storage. Like I always said, “Data is like water” and object storage has become the storage repository of choice for edge-to-core-to-cloud. But it is not sitting on its laurels. It is already setting itself very nicely into the use case of Kubernetes DevOps world, and with the burgeoning of multiclouds, object storage can become the storage singularity across disparate public clouds once vendors agree on the lingua franca among them. Soon, one storage platform will dominated multiclouds object storage and beyond. And that is object storage.
This singularity also allows a less cluttered ways of moving data. In the edge-to-core-to-cloud architecture, data mobility to the right data processing premises is vital. Having an encumbered data delivery path from the data end points at the edge, to the core and the centralized cloud data centers democratizes the flow and the usage of data (within data protection, privacy, security and sovereignty policies), and gives the capability to build the central intelligence of a single source of truth.
Another beyond looking use case is Object Storage on Tape. Yes, tape. That is because data preservation practices have become an activity that is gaining popularity to preserve data for 100 year archiving and maintaining the safety and integrity of important data for present and future generations.
- [ Subscription required ] GigaOM Sonar Report for Object Storage on Tape
I cannot wait to see what is next …
