Open Source Storage and Data Responsibility

There was a Super Blue Moon a few days ago. It was a rare sky show. Friends of mine who are photo and moon gazing enthusiasts were showing off their digital captures online. One ignorant friend, who was probably a bit envious of the other people’s attention, quipped that his Oppo Reno 10 Pro Plus can take better pictures. Oppo Reno 10 Pro Plus claims 3x optical zoom and 120x digital zoom. Yes, 120 times!

Yesterday, a WIRED article came out titled “How Much Detail of the Moon Can Your Smartphone Really Capture?” It was a very technical article. I thought the author did an excellent job explaining the physics behind his notes. But I also found the article funny, flippant even, when I juxtaposed this WIRED article to what my envious friend was saying the other day about his phone’s camera.

Super Blue Moon 2023

Open Source storage expectations and outcomes

I work for iXsystems™. Open Source has been its DNA for over 30 years. Similarly, I have also worked on Open Source (decades before it was called open source) in my home labs ever since I entered the industry. I had SoftLanding Linux System 3.5″ diskette (Linux kernel 0.99), and I bought a boxed set of FreeBSD OS from Walnut Creek (photo below). My motivation was to learn as much as possible about information technology world because I was making my first steps into building my career (I was also quietly trying to prove my father wrong) in the IT industry.

FreeBSD Boxed Set (circa 1993)

Open source has democratized technology. It has placed the power of very innovative technology into the hands of the common people With Open Source, I see the IT landscape changing as well, especially for home labers like myself in the early years. Social media platforms, FAANG (Facebook, Apple, Amazon, Netflix, Google), etc, etc, have amplified that power (to the people). But with that great power, comes great responsibility. And some users with little technology background start to have hallucinated expectations and outcomes. Just like my friend with the “powerful” Oppo phone.

Likewise, in my world, I have plenty of anecdotes of these types of open source storage users having wild expectations, but little skills to exact the reality.

DIY is not enterprise data storage. 

Many of these expectations flow over to the company. It is a good thing when the user knows what he or she is doing. The DIY mindset is great for the small business users who are practically doing everything on their own. They had to build their experience and knowledge by fire, and they know how to fix things when the volcano erupts.

But that mindset is the wrong one to bring where data value is paramount to a larger company’s operations that serves a sizeable business ecosystem. Data compromise and data loss could cost thousands of dollars per hour, not to mention the long tail of damage control. The discipline of storing data in an open source storage solution for a DIYer is different than the discipline in an enterprise setting. It should be beyond the open source product or the software alone.

The mindset from a storage product has to shift to Data Services Platform one.  A Data Services Platform from a storage infrastructure point-of-view, as a data repository have many factors that affect the data and the inherent value to the business. There are required dependencies and competencies. There are interoperability and connectivity. There are data movements and there are data lifecycles. There are validations and certifications as well.

I have always applied my own data points into my conversations with customers. I concocted these data points and have refined them for 25 years now. I have mentioned them several times in my previous blogs. These points (A.P.P.A.R.M.S.C.) are always relevant into building a solid foundation for a data services infrastructure platform. They are:

• Availability
• Protection
• Performance
• Accessibility
• Recovery
• Management
• Security
• Compliance

An open source storage product, when applied in an enterprise setting, is no different. By applying these data consideration points, the storage architect breaks away from the DIY mentality into an enterprise one. What can I do to make my data better? How can I enhance the data value for my organization? 

Not what can I do to save more money because the software is an open source and free.

Technical Debt

I believe in technical debt. I have seen so many people cutting corners, skimping their IT infrastructure and software ecosystem with cheap solutions. The mentality is “It is not going to happen to me” has fallen flat every single time. And when technical debt bites, they bite HARD. I will give you a couple of recent stories.

One manufacturing plant has been running TrueNAS® CORE (the free open source version) for a number of years. I was called and I listened to their problem. They wanted to restore the data after a power outage. They said that their storage volumes “suddenly” had a password request to them after the outage. Naturally, I would ask for their snapshot backups to restore the data. They said they had none. They had never backed up from the very beginning because the boss believed that doing backup would eat up the available storage, leaving less capacity for production. And storage volumes do not get encrypted with a password unless someone applied a password or passphrase to them.

A large conglomerate told their storage administrators to “backup” their data to AWS S3 because the storage in the AWS cloud is “super cheap”. During the COVID time, where everyone was working from home, one data manager ignorantly opened up the access to the S3 “backup” buckets to allow 20+ data analysts to download the data from S3 to their laptops at home. And these analysts, unaware of the cloud egress menace, was uploading and downloading an average of 15-20 times a day. Per analyst. The data management discipline just went out the window by mislabelling the data as “backup”.

In both cases, the technical debt costs were heavy. As a result of “thinking cheap”, the tangible and intangible costs of managing or losing data were disastrous.

The Data Responsibility

I am fortunate to have a Computer Science background. I am happy to have retained much of my technical knowledge and skills. So it is easy to bridge this disconnect between the storage infrastructure and open source storage software mentality to the enterprise data management. Similar to what I said about Data First mentality when considering Cloud Computing, we must adopt a Data Responsibility mindset. Especially when we are considering an open source storage solution. The value of the data is paramount to the organization.

It is great if you are super technical storage guy. But being technical is just one part of the enterprise equation. The ability to connect technical, technology, operations and business excellence into a well-oiled ecosystem for the data to live is more important than being technical, or having that “superpower” mindset that there is no such thing as “technical debt”.

Thus it is best to consider the paid deployment and support services of open source software such as TrueNAS® in the form of a storage appliance. It is equally important to invest in enterprise features such as high availability and hardened security of the TrueNAS® storage appliance where they are validated to perform. Leave it to the professionals. They will give you the peace of mind. Your data is safer in their hands. That is the Data Responsibility I am talking about for the Enterprise.