Category Archives: Filesystems

Fueling the Flywheel of AWS Storage

By | | , , , , , , , , , , , , , , , , ,
Leave a comment

It was bound to happen. It happened. AWS Storage is the Number 1 Storage Company.

The tell tale signs were there when Silicon Angle reported that AWS Storage revenue was around USD$6.5-7.0 billion last year and will reach USD$10 billion at the end of 2021. That news was just a month ago. Last week, IT Brand Pulse went a step further declaring AWS Storage the Number 1 in terms of revenue. Both have the numbers to back it up.

AWS Logo

How did it become that way? How did AWS Storage became numero uno?

Flywheel juggernaut

I became interested in the Flywheel concept some years back. It was conceived in Jim Collins’ book, “Good to Great” almost 20 years ago, and since then, Amazon.com has become the real life enactment of the Flywheel concept.

Amazon.com Flywheel – How each turn becomes sturdier, brawnier.

Every turn of the flywheel requires the same amount of effort although in the beginning, the noticeable effect is minuscule. But as every turn gains momentum, the returns of each turn scales greater and greater to the fixed efforts of operating a single turn.

Continue reading →

Trusting your storage – It’s not about performance

By | | , , , , , , , , , , , , , ,
6 Comments

I have taken some downtime from my blog since late October. Part of my “hiatus” was my illness which had affected my right kidney but I am happy to announce that I am well again. During this period, I spent a lot of the time reading the loads of storage technologies announcements and their marketing calls and almost every single one of them touts Performance as if it is the single “sellable” feature of the respective storage vendor. None ever positions data integrity and the technology behind it in what I believe as the most important and most fundamental feature of any storage technology – Reading the right data exactly it was written into the storage array.

[ Note: Data integrity is even more critical in cloud storage and data corruption, especially the silent ones are even more acute in the clouds ]

Sure, this fundamental feature sounds like it is a given thing in any storage array but believe me, there are enterprise storage arrays which have failed to deliver this simple feature properly. I have end users coming to me through out my storage career that they have database corruption, or file corruption and unable to access their data in an acceptable manner. Data corruption is real folks!

Data corruption.

After several weeks of reading these stuff, I got jaded with so many storage vendors playing leapfrog announcements with their millions of IOPS boasts.

The 3 legged stool

Rewind to circa 2012, just about the time when EMC® acquired XtremIO™. XtremIO™ was a nascent All-Flash startup, and many, including yours truly, really saw the EMC® acquisition was about a high performant storage array. I was having an email conversation with Shahar Frank, one of the co-founders of XtremIO™, and expressing my views about their performance. What Shahar replied surprised me.

The fundamentals of the strength of a storage array was a like a 3-legged stool. 2 legs of the stool would be Performance, and Protection, but with 2 legs, the person sitting on the stool would fall. The 3rd leg would stabilize the balance of the stool, and this 3rd leg was Reliability. This stumped me because XtremIO™’s most sellable feature was Performance. But the wisdom of Shahar pointed to Reliability, the least exciting feature and the most dull of the 3. He was brilliant, of course and went on to found ElastiFile (acquired by Google™), but that’s another story for another day.

Continue reading →

Ransomware recovery with TrueNAS ZFS snapshots

By | | , , , , , , , , , , , , , , , , ,
3 Comments

This is really an excuse to install and play around with TrueNAS® CORE 12.0.

I had a few “self assigned homework exercises” I have to do this weekend. I was planning to do a video webcast with an EFSS vendor soon, and the theme should be around ransomware. Then one of the iXsystems™ resellers, unrelated to the first exercise, was talking about this ransomware messaging yesterday after we did a technical training with them. And this weekend is coming on a bit light as well. So I thought I could bring all these things, including checking out the TrueNAS® CORE 12.0, together in a video (using Free Cam), of which I would do for the first time as well. WOW! I can kill 4 birds with one stone! All together in one blog!

It could be Adam Brown 89 or worse

Trust me. You do not want AdamBrown89 as your friend. Or his thousands of ransomware friends.

When (not if) you are infected by ransomware, you get a friendly message like this in the screenshot below. I got this from a local company who asked for my help a few months ago.

AdamBrown89 ransomware message

AdamBrown89 ransomware message

I have written about this before. NAS (Network Attached Storage) has become a gold mine for ransomware attackers, and many entry level NAS products are heavily inflicted with security flaws and vulnerabilities. Here are a few notable articles in year 2020 alone. [ Note: This has been my journal of the security flaws of NAS devices from 2020 onwards ]

Continue reading →

OpenZFS 2.0 exciting new future

By | | , , , , , , , , , , , , , , , , , , , , , , , , , ,
2 Comments

The OpenZFS (virtual) Developer Summit ended over a weekend ago. I stayed up a bit (not much) to listen to some of the talks because it started midnight my time, and ran till 5am on the first day, and 2am on the second day. Like a giddy schoolboy, I was excited, not because I am working for iXsystems™ now, but I have been a fan and a follower of the ZFS file system for a long time.

History wise, ZFS was conceived at Sun Microsystems in 2005. I started working on ZFS reselling Nexenta in 2009 (my first venture into business with my company nextIQ) after I was professionally released by EMC early that year. I bought a Sun X4150 from one of Sun’s distributors, and started creating a lab server. I didn’t like the workings of NexentaStor (and NexentaCore) very much, and it was priced at 8TB per increment. Later, I started my second company with a partner and it was him who showed me the elegance and beauty of ZFS through the command lines. The creed of ZFS as a volume and a file system at the same time with the CLI had an effect on me. I was in love.

OpenZFS Developer Summit 2020 Logo

OpenZFS Developer Summit 2020 Logo

Exciting developments

Among the many talks shared in the OpenZFS Developer Summit 2020 , there were a few ideas and developments which were exciting to me. Here are 3 which I liked and I provide some commentary about them.

  • Block Reference Table
  • dRAID (declustered RAID)
  • Persistent L2ARC

Continue reading →

A Paean to NFS

By | | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
Leave a comment

It is certainly encouraging to see both NAS protocols, NFS and SMB, featured well in the latest VMware® vSAN 7 Update 1 release. The NFS v3 and v4.1 support was already in vSAN 7.0 when it was earlier announced as part of its Native File Services for vSAN. But some years ago, NFS was not always the primary storage protocol of choice. SAN protocols, Fibre Channel and iSCSI, were almost always designated to serve enterprise applications. At the client side, Windows became prominent, and the SMB/CIFS protocol dominated the landscape of the desktop. This further pushed NFS into the back closet.

NFS or Network File System has its naysayers. The venerable, but often maligned distributed network file protocol is 36 years today. In storage vendors such as NetApp®, VAST Data, Pure Storage FlashBlade, and Dell EMC Isilon, NFS is still positioned as the primary file protocol for manufacturing testers on the shop floor, EDA/eCAD applications, seismic and subsurface applications in Oil & Gas and many more. In another development, just like its presence in the vSAN Native Services,, NFS has also quietly embedded itself into many storage platforms to serve the data platform services within the respective framework itself.

And I have experienced NFS from the client side to the enterprise applications and more, and I take this opportunity to pay tribute.

NFS (Network File System) client server network

NFS (Network File System) client server network

Continue reading →

Give back or no give

By | | , , , , , , , , , , , , , , , , , , , , ,
Leave a comment

[ Disclosure: I work for iXsystems™ Inc. Views and opinions are my own. ]

If my memory served me right, I recalled the illustrious leader of the Illumos project, Garrett D’Amore ranting about companies, big and small, taking OpenZFS open source codes and projects to incorporate into their own technology but hardly ever giving back to the open source community. That was almost 6 years ago.

My thoughts immediately go back to the days when open source was starting to take off back in the early 2000s. Oracle 9i database had just embraced Linux in a big way, and the book by Eric S. Raymond, “The Cathedral and The Bazaar” was a big hit.

The Cathedral & The Bazaar by Eric S. Raymond

Since then, the blooming days of proprietary software world began to wilt, and over the next twenty plus year, open source software has pretty much taken over the world. Even Microsoft®, the ruthless ruler of the Evil Empire caved in to some of the open source calls. The Microsoft® “I Love Linux” embrace definitely gave the victory feeling of the Rebellion win over the Empire. Open Source won.

Open Source bag of worms

Even with the concerted efforts of the open source communities and projects, there were many situations which have caused frictions and inadvertently, major issues as well. There are several open source projects licenses, and they are not always compatible when different open source projects mesh together for the greater good.

On the storage side of things, 2 “incidents” caught the attention of the masses. For instance, Linus Torvalds, Linux BDFL (Benevolent Dictator for Life) and emperor supremo said “Don’t use ZFS” partly due to the ignorance and incompatibility of Linux GPL (General Public License) and ZFS CDDL (Common Development and Distribution License). That ruffled some feathers amongst the OpenZFS community that Matt Ahrens, the co-creator of the ZFS file system and OpenZFS community leader had to defend OpenZFS from Linus’ comments.

Continue reading →

FreeNAS 11.2 & 11.3 eBook

By | | , , , , , , , , , , , , , , , , , , , , ,
Leave a comment

[ Full disclosure: I work for iXsystems™ Inc. This eBook was 3/4 completed when I joined on July 1, 2020 ]

I am releasing my FreeNAS™ eBook today. It was completed about 4 weeks ago, but I wanted the release date to be significant which is August 31, 2020.

FreeNAS logo

Why August 31st? Because today is Malaysia’s Independence Day.

Why the book?

I am an avid book collector. To be specific, IT and storage technology related books. Since I started working on FreeNAS™ several years ago, I wanted to find a book to learn. But the FreeNAS™ books in the market are based on an old version of FreeNAS™. And the FreeNAS™ documentation is a User Guide where it explains every feature without going deeper with integration of real life networking services, and situational applications such as SMB or NFS client configuration.

Since I have been doing significant amount of feature “testings” of FreeNAS™ from version 9.10 till the present version 11,3 on Virtualbox™, I have decided to fill that gap. I have decided to write a cookbook-style FreeNAS™ on Virtualbox™ that covers most of the real-life integration work with various requirements including Active Directory, cloud integration and so on. All for extending beyond the FreeNAS™ documentation.

Continue reading →

The instant value of Open Source Storage

By | | , , , , , , , , , , , ,
1 Comment

[ Full disclosure: I work for iXsystems™ . Opinions and views are mine. ]

TrueNAS Open Storage logo

The story began …

It was 2011. A friend of a friend called me out of the blue. He was rambling about his company’s storage needs. I recalled vividly that he wanted 100TB, and Dell and HP (before HPE) were hopeless doing NAS (network attached storage) in an Apple environment. They assembled a Frankenstein-ish NAS and plastered a price over MYR$100K around it.

In his environment, the Apple workstations were connected to dozens of WD Cloud Book storage (whatever it was called back then), daisy chained via Firewire to each other. I recalled one workstation had 3 WD “books” daisy chained together. They got the exploding storage needs but performance sucked. With every 2nd or 3rd user, access to files were at a snail pace, taking up to more than 2 minutes to open a file sometimes.

At that time, my old colleague at Sun was fervently talking about ZFS and OpenSolaris™. I told him about this opportunity, and so we began. It was him who used the word “crafter”. “We are not building“, he said, “we are crafting“. He was right.

OpenSolaris logo

Continue reading →

Intel is still a formidable force

By | | , , , , , , , , , , , , , , , , , , , , , , , , , , ,
1 Comment

It is easy to kick someone who is down. Bad news have stronger ripple effects than the good ones. Intel® is going through a rough patch, and perhaps the worst one so far. They delayed their 7nm manufacturing process, one which could have given Intel® the breathing room in the CPU war with rival AMD. And this delay has been pushed back to 2021, possibly 2022.

Intel Apple Collaboration and Partnership started in 2005

Their association with Apple® is coming to an end after 15 years, and more security flaws surfaced after the Spectre and Meltdown debacle. Extremetech probably said it best (or worst) last month:

If we look deeper (and I am sure you have), all these negative news were related to their processors. Intel® is much, much more than that.

Their Optane™ storage prowess

I have years of association with the folks at Intel® here in Malaysia dating back 20 years. And I hardly see Intel® beating it own drums when it comes to storage technologies but they are beginning to. The Optane™ revolution in storage, has been a game changer. Optane™ enables the implementation of persistent memory or storage class memory, a performance tier that sits between DRAM and the SSD. The speed and more notable the latency of Optane™ are several times faster than the Enterprise SSDs.

Intel pyramid of tiers of storage medium

If you want to know more about Optane™’s latency and speed, here is a very geeky article from Intel®:

The list of storage vendors who have embedded Intel® Optane™ into their gears is long. Vast Data, StorOne™, NetApp® MAX Data, Pure Storage® DirectMemory Modules, HPE 3PAR and Nimble Storage, Dell Technologies PowerMax, PowerScale, PowerScale and many more, cement Intel® storage prowess with Optane™.

3D Xpoint, the Phase Change Memory technology behind Optane™ was from the joint venture between Intel® and Micron®. That partnership was dissolved in 2019, but it has not diminished the momentum of next generation Optane™. Alder Stream and Barlow Pass are going to be Gen-2 SSD and Persistent Memory DC DIMM respectively. A screenshot of the Optane™ roadmap appeared in Blocks & Files last week.

Intel next generation Optane roadmap

Continue reading →

Valuing the security value of NAS storage

By | | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,
1 Comment

Garmin paid, reportedly millions. Do you sleep well at night knowing that the scourge of ransomware is rampant and ever threatening your business. Is your storage safe enough or have you invested in a storage which was the economical (also to be known as cheap) to your pocket?

Garmin was hacked by ransomware

I have highlighted this before. NAS (Network Attached Storage) has become the goldmine for ransomware. And in the mire of this COVID-19 pandemic, the lackadaisical attitude of securing the NAS storage remains. Too often than not, end users and customers, especially in the small medium enterprises segment, continue to search for the most economical NAS storage to use in their business.

Is price the only factor?

Why do customers and end users like to look at the price? Is an economical capital outlay of a cheap NAS storage with 3-year hardware and shallow technical support that significant to appease the pocket gods? Some end users might decided to rent cloud file storage, Hotel California style until they counted the 3-year “rental” price.

Continue reading →