My dilemma of stateful storage marriage

I should be a love match maker.

I have been spending much hours in the past few months, thinking of stateful data in stateful storage containers and how they would consummate with distributed applications containers and functions-as-a-service (aka serverless, aka Lambda). It still hasn’t made much sense, and I have not solved this problem yet. Although there were bits and pieces that coming together and the jigsaw looked well enough to give a cackled reply, what I have now is still not good enough for me. I am still searching for answers, better than the ones I have now.

The CAP theorem is in center of my mind. Distributed data, distributed states of data are on my mind. And by the looks of things, the computing world is heading towards containers and serverless computing too. Both distributed applications containers and serverless computing make a lot of sense. If we were to engage a whole new world of fog computing, edge computing, IoT, autonomous systems, AI, and other real-time computing, I would say that the future belongs to decentralization. Cloud Computing and having edge systems and devices getting back to the cloud for data is too slow. The latency of micro- or even nano-seconds is just not good enough. If we rely on the present methods to access the most relevant data, we are too late.

If we were to build the next generation of distributed data systems, what would it be? It won’t be a data center for sure. Even the word data “CENTER” means centralization, the anti-matter of distributed systems and data data systems.

I have seen and read many technology vendors touting object storage as the right fit to meet the requirements of stateful data for the containers and serverless computing. Applications containers are distributed. So are server computing factories. Objects are distributed too, so I’d guess that object storage is the best answer to this. Maybe I am too naive; maybe I am too uninformed. But it seems to me that generation 1 of most object storage platforms use eventual data consistency (CAP Theorem applies), and that has become the crux of my dilemma.

Eventual data consistency is latency. Eventual data consistency is lack of relevancy. Both are, to me, complete opposites of the real-time statefulness of data, something of a misnomer to the requirements of distributed applications containers and serverless computing.

There is hope. Generation 2 multi-cloud storage systems is promising. It is bringing more intelligence and layering a new fabric over generation 1 object storage. They are touting microseconds I/O access and I have seen several of these Gen2 vendors in SuperComputing conferences in the US and in Europe in the past 6-8 months. And I love to keep learning and have deeper understandings of this wonderful dilemma.

I am still out there searching for answers, better answers. My previous trips to Storage Field Days were good. In SFD12, there were Datera and Elastifile in my books. In SFD14, it was Scality. And now, in the upcoming Storage Field Day 15, we will be expecting Hedvig and WekaIO to hand out more jigsaw pieces to the puzzle. I have always enjoyed the serendipity of Storage Field Days, and I am looking for more.

Happy Lunar New Year to all. It’s the Year of the Dog!

Tagged , , , , , , . Bookmark the permalink.

About cfheoh

I am a technology blogger with 20+ years of IT experience. I write heavily on technologies related to storage networking and data management because that is my area of interest and expertise. I introduce technologies with the objectives to get readers to *know the facts*, and use that knowledge to cut through the marketing hypes, FUD (fear, uncertainty and doubt) and other fancy stuff. Only then, there will be progress. I am involved in SNIA (Storage Networking Industry Association) and as of October 2013, I have been appointed as SNIA South Asia & SNIA Malaysia non-voting representation to SNIA Technical Council. I was previously the Chairman of SNIA Malaysia until Dec 2012. As of August 2015, I am returning to NetApp to be the Country Manager of Malaysia & Brunei. Given my present position, I am not obligated to write about my employer and its technology, but I am indeed subjected to Social Media Guidelines of the company. Therefore, I would like to make a disclaimer that what I write is my personal opinion, and mine alone. Therefore, I am responsible for what I say and write and this statement indemnify my employer from any damages.

One Response to My dilemma of stateful storage marriage

  1. Pingback: My dilemma of stateful storage marriage - Tech Field Day

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.