The hot cold times of HCI

Hyperconverged Infrastructure (HCI) is a hot technology. It has been for the past decade since Nutanix™ took the first mover advantage from the Converged Infrastructure (CI) technology segment and made it pretty much its ownfor a while.

Hyper Converged Infrastructure

But the HCI market (not the technology) is a strange one. It is hot. It is cold. The perennial leader, Nutanix™, has yet to eke out a profitable year. VMware® is strong in the market. Cisco™, which was hot with their HyperFlex solution in 2019, was also stopped short with a dismal decline in the IDC Worldwide HCI 2Q2020 tracker below:

IDC Worldwide Hyperconverged Infrastructure Tracker – 2Q2020

dHCI = Disaggregated or discombobulated? 

dHCI is known as disaggregated HCI. The disaggregation part is disaggregated hardware, especially on the storage part. Vendors like HPE® with Nimble Storage, Hitachi Vantara, NetApp® and a few more have touted the disaggregation of the performance and capacity, the separation of storage and compute as a value proposition but through close inspection, it is just another marketing ploy to attach a SAN storage to servers. It was marketing old wine in a new bottle. As rightly pointed out by my friend, Charles Chow of Commvault® quoted in his blog

Continue reading

When you buy storage solutions on price alone

Most people won’t bat an eye buying a car. It is a status symbol for many, but the value of the work returned from the car to the cost of buying the car is a great disparity. Furthermore, the price of the car depreciates quickly, making the “investment” more like an act of losing money fast.

So the story begins. When it comes to buying a storage technology platform, the initial price on the quote more or less decide the outcome. The reply of “Too expensive!” with little consideration about the returns of certain values relative to the initial buying price is far too frequent and plenty.

There has to be more considerations about these values. Here are in buying a storage technology platform besides just the initial price.


One recent conversation was about Intel® Optane™ vs NAND Flash. An well-known online eCommerce proprietor in South East Asia decided to go against the grain, and went for the more “expensive” Optane™ instead of the getting an array of NAND Flash NVMe SSDs.

Continue reading

Layers in Storage – For better or worse

Storage arrays and storage services are built upon by layers and layers beneath its architecture. The physical components of hard disk drives and solid states are abstracted into RAID volumes, virtualized into other storage constructs before they are exposed as shares/exports, LUNs or objects to the network.

Everyone in the storage networking industry, is cognizant of the layers and it is the foundation of knowledge and experience. The public cloud storage services side is the same, albeit more opaque. Nevertheless, both have layers.

In the early 2000s, SNIA® Technical Council outlined a blueprint of the SNIA® Shared Storage Model, a framework describing layers and properties of a storage system and its services. It was similar to the OSI 7-layer model for networking. The framework helped many industry professionals and practitioners shaped their understanding and the development of knowledge in their respective fields. The layering scheme of the SNIA® Shared Storage Model is shown below:

SNIA Shared Storage Model – The layering scheme

Storage vendors layering scheme

While SNIA® storage layers were generic and open, each storage vendor had their own proprietary implementation of storage layers. Some of these architectures are simple, but some, I find a bit too complex and convoluted.

Here is an example of the layers of the Automated Volume Management (AVM) architecture of the EMC® Celerra®.

EMC Celerra AVM Layering Scheme

I would often scratch my head about AVM. Disks were grouped into RAID groups, which are LUNs (Logical Unit Numbers). Then they were defined as Celerra® dvols (disk volumes), and stripes of the dvols were consolidated into a storage pool.

From the pool, a piece of a storage capacity construct, called a slice volume, were combined with other slice volumes into a metavolume which eventually was presented as a file system to the network and their respective NAS clients. Explaining this took an effort because I was the IP Storage product manager for EMC® between 2007 – 2009. It was a far cry from the simplicity of NetApp® ONTAP 7 architecture of RAID groups and volumes, and the WAFL® (Write Anywhere File Layout) filesystem.

Another complicated layered framework I often gripe about is Ceph. Here is a look of how the layers of CephFS is constructed.

Ceph Storage Layered Framework

I work with the OpenZFS filesystem a lot. It is something I am rather familiar with, and the layered structure of the ZFS filesystem is essentially simpler.

Storage architecture mixology

Engineers are bizarre when they get too creative. They have a can do attitude that transcends the boundaries of practicality sometimes, and boggles many minds. This is what happens when they have their own mixology ideas.

Recently I spoke to two magnanimous persons who had the idea of providing Ceph iSCSI LUNs to the ZFS filesystem in order to use the simplicity of NAS file sharing capabilities in TrueNAS® CORE. From their own words, Ceph NAS capabilities sucked. I had to draw their whole idea out in a Powerpoint and this is the architecture I got from the conversation.

There are 3 different storage subsystems here just to provide NAS. As if Ceph layers aren’t complicated enough, the iSCSI LUNs from Ceph are presented as Cinder volumes to the KVM hypervisor (or VMware® ESXi) through the Cinder driver. Cinder is the persistent storage volume subsystem of the Openstack® project. The Cinder volumes/hypervisor datastore are virtualized as vdisks to the respective VMs installed with TrueNAS® CORE and OpenZFS filesystem. From the TrueNAS® CORE, shares and exports are provisioned via the SMB and NFS protocols to Windows and Linux respectively.

It works! As I was told, it worked!

A.P.P.A.R.M.S.C. considerations

Continuing from the layered framework described above for NAS, other aspects beside the technical work have to be considered, even when it can work technically.

I often use a set of diligent data storage focal points when considering a good storage design and implementation. This is the A.P.P.A.R.M.S.C. Take for instance Protection as one of the points and snapshot is the technology to use.

Snapshots can be executed at the ZFS level on the TrueNAS® CORE subsystem. Snapshots can be trigged at the volume level in Openstack® subsystem and likewise, rbd snapshots at the Ceph subsystem. The question is, which snapshot at which storage subsystem is the most valuable to the operations and business? Do you run all 3 snapshots? How do you execute them in succession in a scheduled policy?

In terms of performance, can it truly maximize its potential? Can it churn out the best IOPS, and deliver at wire speed? What is the latency we can expect with so many layers from 3 different storage subsystems?

And supporting this said architecture would be a nightmare. Where do you even start the troubleshooting?

Those are just a few considerations and questions to think about when such a layered storage architecture along. IMHO, such a design was over-engineered. I was tempted to say “Just because you can, doesn’t mean you should

Elegance in Simplicity

Einstein (I think) quoted:

Einstein’s quote on simplicity and complexity

I am not saying that having too many layers is wrong. Having a heavily layered architecture works for many storage solutions out there, where they are often masked with a simple and intuitive UI. But in yours truly point of view, as a storage architecture enthusiast and connoisseur, there is beauty and elegance in simple designs.

The purpose here is to promote better understanding of the storage layers, and how they integrate and interact with each other to deliver the data services to the network. In the end, that is how most storage architectures are built.


Discovering OpenZFS Fusion Pool

Fusion Pool excites me, but unfortunately this new key feature of OpenZFS is hardly talked about. I would like to introduce the Fusion Pool feature as iXsystems™ expands the TrueNAS® Enterprise storage conversations.

I would not say that this technology is revolutionary. Other vendors already have the similar concept of Fusion Pool. The most notable (to me) is NetApp® Flash Pool, and I am sure other enterprise storage vendors have the same. But this is a big deal (for me) for an open source file system in OpenZFS.

What is Fusion Pool  (aka ZFS Allocation Classes)?

To understand Fusion Pool, we have to understand the basics of the ZFS zpool. A zpool is the aggregation (borrowing the NetApp® terminology) of vdevs (virtual devices), and vdevs are a collection of physical drives configured with the OpenZFS RAID levels (RAID-0, RAID-1, RAID-Z1, RAID-Z2, RAID-Z3 and a few nested RAID permutations). A zpool can start with one vdev, and new vdevs can be added on-the-fly, expanding the capacity of the zpool online.

There are several types of vdevs prior to Fusion Pool, and this is as of pre-TrueNAS® version 12.0. As shown below, these are the types of vdevs available to the zpool at present.

OpenZFS zpool and vdev types – Credit: Jim Salter and Arstechnica

Fusion Pool is a zpool that integrates with a new, special type of vdev, alongside other normal vdevs. This special vdev is designed to work with small data blocks between 4-16K, and is highly efficient in handling random reading and writing of these small blocks. This bodes well with the OpenZFS file system metadata blocks and other blocks of small files. And the random nature of the Read/Write I/Os works best with SSDs (can be read or write intensive SSDs).

Continue reading

TrueNAS – The Secure Data Platform for EasiShare

The Enterprise File Sync and Share (EFSS) EasiShare presence is growing rapidly in the region, as enterprises and organizations are quickly redefining the boundaries of the new workspace. Work files and folders are no longer confined to the shared network drives within the local area network. It is going beyond to the “Work from Anywhere” phenomenon that is quickly becoming the way of life. Breaking away from the usual IT security protection creates a new challenge, but EasiShare was conceived with security baked into its DNA. With the recent release, Version 10, file sharing security and resiliency are stronger than ever.

[ Note: I have blogged about EasiShare previously. Check out the 2 links below ]

Public clouds are the obvious choice but for organizations to protect their work files, and keep data secure, services like Dropbox for Business, Microsoft® Office 365 with OneDrive and Google® Workspace are not exactly the kind of file sharing with security as their top priority. A case in point was the 13-hour disruption to Wasabi Cloud last week, where the public cloud storage provider’s domain name,, was suspended by their domain name registrar because of malware discrepancy at one of its endpoints. There were other high profile cases too.

This is where EasiShare shines, because it is a secure, private EFSS solution for the enterprise and beyond, because business resiliency is in the hands and control of the organization that owns it, not the public cloud service providers.

EasiShare unifies with TrueNAS for secure business resiliency

EasiShare is just one several key business solutions iXsystems™ in Asia Pacific Japan is working closely with, and there is a strong, symbiotic integration with the TrueNAS® platform. Both have strong security features that fortify business resiliency, especially when facing the rampant ransomware scourge.

Value of a Single Unified Data Services Platform

A storage array is not a solution. It is just a box that most vendors push to sell. A storage must be a Data Services Platform. Readers of my blog would know that I have spoken about the Data Services Platform 3 years ago and you can read about it:

Continue reading

Ransomware recovery with TrueNAS ZFS snapshots

This is really an excuse to install and play around with TrueNAS® CORE 12.0.

I had a few “self assigned homework exercises” I have to do this weekend. I was planning to do a video webcast with an EFSS vendor soon, and the theme should be around ransomware. Then one of the iXsystems™ resellers, unrelated to the first exercise, was talking about this ransomware messaging yesterday after we did a technical training with them. And this weekend is coming on a bit light as well. So I thought I could bring all these things, including checking out the TrueNAS® CORE 12.0, together in a video (using Free Cam), of which I would do for the first time as well. WOW! I can kill 4 birds with one stone! All together in one blog!

It could be Adam Brown 89 or worse

Trust me. You do not want AdamBrown89 as your friend. Or his thousands of ransomware friends.

When (not if) you are infected by ransomware, you get a friendly message like this in the screenshot below. I got this from a local company who asked for my help a few months ago.

AdamBrown89 ransomware message

AdamBrown89 ransomware message

I have written about this before. NAS (Network Attached Storage) has become a gold mine for ransomware attackers, and many entry level NAS products are heavily inflicted with security flaws and vulnerabilities. Here are a few notable articles in year 2020 alone.

Continue reading

FreeNAS 11.2 & 11.3 eBook

[ Full disclosure: I work for iXsystems™ Inc. This eBook was 3/4 completed when I joined on July 1, 2020 ]

I am releasing my FreeNAS™ eBook today. It was completed about 4 weeks ago, but I wanted the release date to be significant which is August 31, 2020.

FreeNAS logo

Why August 31st? Because today is Malaysia’s Independence Day.

Why the book?

I am an avid book collector. To be specific, IT and storage technology related books. Since I started working on FreeNAS™ several years ago, I wanted to find a book to learn. But the FreeNAS™ books in the market are based on an old version of FreeNAS™. And the FreeNAS™ documentation is a User Guide where it explains every feature without going deeper with integration of real life networking services, and situational applications such as SMB or NFS client configuration.

Since I have been doing significant amount of feature “testings” of FreeNAS™ from version 9.10 till the present version 11,3 on Virtualbox™, I have decided to fill that gap. I have decided to write a cookbook-style FreeNAS™ on Virtualbox™ that covers most of the real-life integration work with various requirements including Active Directory, cloud integration and so on. All for extending beyond the FreeNAS™ documentation.

Continue reading

The instant value of Open Source Storage

[ Full disclosure: I work for iXsystems™ . Opinions and views are mine. ]

TrueNAS Open Storage logo

The story began …

It was 2011. A friend of a friend called me out of the blue. He was rambling about his company’s storage needs. I recalled vividly that he wanted 100TB, and Dell and HP (before HPE) were hopeless doing NAS (network attached storage) in an Apple environment. They assembled a Frankenstein-ish NAS and plastered a price over MYR$100K around it.

In his environment, the Apple workstations were connected to dozens of WD Cloud Book storage (whatever it was called back then), daisy chained via Firewire to each other. I recalled one workstation had 3 WD “books” daisy chained together. They got the exploding storage needs but performance sucked. With every 2nd or 3rd user, access to files were at a snail pace, taking up to more than 2 minutes to open a file sometimes.

At that time, my old colleague at Sun was fervently talking about ZFS and OpenSolaris™. I told him about this opportunity, and so we began. It was him who used the word “crafter”. “We are not building“, he said, “we are crafting“. He was right.

OpenSolaris logo

Continue reading

Valuing the security value of NAS storage

Garmin paid, reportedly millions. Do you sleep well at night knowing that the scourge of ransomware is rampant and ever threatening your business. Is your storage safe enough or have you invested in a storage which was the economical (also to be known as cheap) to your pocket?

Garmin was hacked by ransomware

I have highlighted this before. NAS (Network Attached Storage) has become the goldmine for ransomware. And in the mire of this COVID-19 pandemic, the lackadaisical attitude of securing the NAS storage remains. Too often than not, end users and customers, especially in the small medium enterprises segment, continue to search for the most economical NAS storage to use in their business.

Is price the only factor?

Why do customers and end users like to look at the price? Is an economical capital outlay of a cheap NAS storage with 3-year hardware and shallow technical support that significant to appease the pocket gods? Some end users might decided to rent cloud file storage, Hotel California style until they counted the 3-year “rental” price.

Continue reading