IDC Archives - Storage Gaga

Computational Storage embodies Data Velocity and Locality

By cfheoh | March 21, 2022 - 8:00 am |March 31, 2022 Analytics, Appliance, Artificial Intelligence, Big Data, Blockchain, Cloud, Composable Infrastructure, compression, Containers, Data Management, Data Security, Deduplication, Deep Learning, Edge Computing, IDC, Machine Learning, NVMe, PCIe, SNIA, Solid State Devices

1 Comment

I have been earnestly observing the growth of Computational Storage for a number of years now. It was known by several previous names, with the name “in-situ data processing” stuck with me the most. The Computational Storage nomenclature became more cohesive when SNIA® put together the CMSI (Compute Memory Storage Initiative) some time back. This initiative is where several standards bodies, the major technology players and several SIGs (special interest groups) in SNIA® collaborated to advance Computational Storage segment in the storage technology industry we know of today.

The use cases for Computational Storage are burgeoning, and the functional implementations of Computational Storage are becoming vital to tackle the explosive data tsunami. In 2018 IDC, in its Worldwide Global Datasphere Forecast 2021-2025 report, predicted that the world will have 175 ZB (zettabytes) of data. That number, according to hearsay, has been revised to a heady figure of 250ZB, given the superlative rate data is being originated, spawned and more.

Computational Storage driving factors

If we take the Computer Science definition of in-situ processing, Computational Storage can be distilled as processing data where it resides. In a nutshell, “Bring Compute closer to Storage“. This means that there is a processing unit within the storage subsystem which does not require the host CPU to perform processing. In a very simplistic manner, a RAID card in a storage array can be considered a Computational Storage device because it performs the RAID functions instead of the host CPU. But this new generation of Computational Storage has much more prowess than just the RAID function in a RAID card.

There are many factors in Computational Storage that make a lot sense. Here are a few:

Voluminous data inundate the centralized architecture of the cloud platforms and the enterprise systems today. Much of the data come from end point devices – mobile devices, sensors, IoT, point-of-sales, video cameras, et.al. Pre-processing the data at the origin data points can help filter the data, reduce the size to be processed centrally, and secure the data before they are ingested into the central data processing systems
Real-time processing of the data at the moment the data is received gives the opportunity to create the Velocity of Data Analytics. Much of the data do not need to move to a central data processing system for analysis. Often in use cases like autonomous vehicles, fraud detection, recommendation systems, disaster alerts etc require near instantaneous responses. Performing early data analytics at the data origin point has tremendous advantages.
Moore’s Law is waning. The CPU (central processing unit) is no longer the center of the universe. We are beginning to see CPU offloading technologies to augment the CPU’s duties such as compression, encryption, transcoding and more. SmartNICs, DPUs (data processing units), VPUs (visual processing units), GPUs (graphics processing units), etc have come forth to formulate a new computing paradigm.
Freeing up central resources with Computational Storage also accelerates the overall distributed data processing in the whole data architecture. The CPU and the adjoining memory subsystem are less required to perform context switching caused by I/O interrupts as in most of the compute/storage architecture today. The total effect relieves the CPU and giving back more CPU cycles to perform higher processing tasks, resulting in faster performance overall.
The rise of memory interconnects is enabling a more distributed computing fabric of data processing subsystems. The rising CXL (Compute Express Link™) interconnect protocol, especially after the Gen-Z annex, has emerged a force to be reckoned with. This rise of memory interconnects will likely strengthen the testimony of Computational Storage in the fast approaching future.

Computational Storage Deployment Models

SNIA Computational Storage Universe in 2019

Continue reading →

The burgeoning world of NVMe

By cfheoh | November 29, 2021 - 8:00 am |November 28, 2021 100Gigabit Ethernet, Cloud, Composable Infrastructure, CXL, Disks, Fibre Channel, IDC, Infiniband, iSCSI, Memory Cloud, NVMe, Performance Benchmark, Performance Caching, Software-defined Datacenter, Solid State Devices, Storage Area Network, Storage Optimization, Virtualization

1 Comment

When I wrote this article “Let’s smoke this storage peace pipe” 5 years ago, I quoted:

“NVMe® and NVM®eF‰, as it evolves, can become the Great Peacemaker and bringing both divides and uniting them into a single storage fabric.“

I envisioned NVMe® and NVMe®oF™ setting the equilibrium at the storage architecture level, finishing the great storage fabric into one. This balance in the storage ecosystem at the storage interface specifications and language-protocol level has rapidly unifying storage today, and we are already seeing the end-to-end NVMe paths directly from the PCIe bus of one host to another, via networks over Ethernet (with RoCE, iWARP, and TCP flavours) and Fibre Channel™. Technically we can have an end point device, example a tablet, talking the same NVMe language to its embedded storage as well as a cloud NVMe storage in an exascale storage far, far away. In the past, there were just too many bridges, links, viaducts, aqueducts, bypasses, tunnels, flyovers to cross just to deliver a storage command, or a data in a formats, encased and encoded (and decoded) in so many different ways.

Colours in equilibrium, like the rainbow

Simple basics of NVMe®

SATA (Serial Attached ATA) and SAS (Serial Attached SCSI) are not optimized for solid state devices. besides legacy stuff like AHCI (Advanced Host Controller Interface) in SATA, and archaic SCSI-3 primitives in SAS, NVM® has so much to offer. It can achieve very high bandwidth and support 65,535 I/O queues, each with a queue depth of 65,535. The queue depth alone is a massive jump compared to SAS which has a queue depth limit of 256.

A big part of this is how NVMe® handles I/O processing. It has a submission queue (SQ) and a completion queue (CQ), and together they are know as a Queue Pair (QP). The NVMe® controller handles tens of thousands at I/Os (reads and writes) simultaneously, alerted to switch between each SQ and CQ very quickly using the MSI or MSI-X interrupt. Think of MSI and MSI-X as a service bell, a hardware register that informs the NVM® controller when there are requests in the SQ, and informs the hosts that there are completed requests in the CQ. There will be plenty of “dings” by the MSI-X service register but the NVMe® controller can perform it very well, with some smart interrupt coalescing.

NVMe I/O processing

NVMe® 1.1, as I recalled, used to be have 3 admin commands and 10 base commands, which made it very lightweight compared to SCSI-3. However, newer commands were added to NVMe® 2.0 specifications included command sets fo key-value operations and zoned named space.

Continue reading →

The hot cold times of HCI

By cfheoh | March 8, 2021 - 9:00 am |March 6, 2021 Appliance, Cisco, Citrix, Cloud, Composable Infrastructure, Containers, Datacore, Datrium, DellEMC, Edge Computing, Hitachi Vantara, HPE, HPE Simplivity, Hyperconvergence, IDC, NetApp, Nimble Storage, Nutanix, ROBO, Server SAN, Simplivity, Software Defined Storage, Software-defined Datacenter, Starwind, VDI, VMware

dHCI = Disaggregated or discombobulated?

dHCI is known as disaggregated HCI. The disaggregation part is disaggregated hardware, especially on the storage part. Vendors like HPE® with Nimble Storage, Hitachi Vantara, NetApp® and a few more have touted the disaggregation of the performance and capacity, the separation of storage and compute as a value proposition but through close inspection, it is just another marketing ploy to attach a SAN storage to servers. It was marketing old wine in a new bottle. As rightly pointed out by my friend, Charles Chow of Commvault® quoted in his blog

Why would anyone then consider a stack that is ‘new’ that looks a whole lot like ‘old’?

Continue reading →

Is General Purpose Object Storage disenfranchised?

By cfheoh | December 23, 2019 - 5:40 pm |January 14, 2020 100Gigabit Ethernet, Amazon Web Services, Analytics, API, Artificial Intelligence, Big Data, BYOD, Ceph, Cloud, Cloudian, Clusters, Deep Learning, DellEMC, Docker, Dropbox, Edge Computing, Filesystems, Flash, Gartner, Hadoop, HDS, High Performance Computing, Hitachi Vantara, IDC, Industry 4.0, IoT, Lustre, Machine Learning, Mellanox Technologies, Minio, NetApp, Object Storage, OpenIO, Openstack, Performance Benchmark, Reliability, Scale-out architecture, Software Defined Storage, Storage Field Day, Storage Market Share, swiftstack, Tape storage, Tech Field Day

6 Comments

[Disclosure: I am invited by GestaltIT as a delegate to their Storage Field Day 19 event from Jan 22-24, 2020 in the Silicon Valley USA. My expenses, travel, accommodation and conference fees will be covered by GestaltIT, the organizer and I am not obligated to blog or promote the vendors’ technologies to be presented at this event. The content of this blog is of my own opinions and views]

This is NOT an advertisement for coloured balls.

This is the license to brag for the vendors in the next 2 weeks or so, as we approach the 2020 new year. This, of course, is the latest 2019 IDC Marketscape for Object-based Storage, released last week.

My object storage mentions

I have written extensively about Object Storage since 2011. With different angles and perspectives, here are some of them:

The Future is Intelligent Objects (2011)
What should be Cloud Storage? (2011)
APIs that stick in Storage (2012)
Has Object Storage become the Everything Store? (2013)
Of Object Storage, Filesystems and Multicloud (2017)
My Dilemma of Stateful Storage Marriage (2018)
The Malaysian Openstack Storage Conundrum (2018)
Sleepless in Malaysia with Object Storage (2019)
The Waning Light of Openstack Swift (2019)

Continue reading →

Sleepless in Malaysia with Object Storage

By cfheoh | January 22, 2019 - 1:42 pm |January 22, 2019 Amazon, Amazon Web Services, Analytics, API, Big Data, Cloud, Cloudian, Clusters, Data Management, Deep Learning, DellEMC, Dropbox, Filesystems, Google, Hadoop, Hadoop Clusters, HDS, IDC, IoT, iSCSI, Linux, Machine Learning, Microsoft, Minio, NAS, NFS, Object Storage, OpenIO, Redhat, Security, swiftstack

Object Storage? What’s that?

For the past couple of months, I have been speaking with a few parties in Malaysia about object storage technology. And I was fairly surprised with the responses.

The 2 reports

For a start, I did not set out to talk about object storage. It kind of fell onto my lap. 2 recent Hitachi Vantara reports revealed that countries like Australia, Hong Kong and even South East Asian countries were behind in their understanding of what object storage was, and the benefits it brought to the new generation of web scale and enterprise applications.

In the first report, an IDC survey sponsored by Hitachi Vantara, mentioned that 41% of the enterprises in Australia are not aware of object storage technology. In a similar survey, this one pointing towards Hong Kong and China, the percentages were 38% and 35% respectively. I would presume that the percentages for countries in South East Asia would not fall too far from the apple tree.

How is Malaysia doing?

However, I worry that the percentage number could be far more dire in Malaysia. In the past 2 months, responses from several conversations painted a darker hue about object storage technology with the companies in Malaysia. These included a reasonable sized hosting company, a well-established systems integrator, a software development company, several storage practitioners in Openstack and a DellEMC’s regional consultant for unstructured data. The collective conclusion was object storage technology was relatively unknown (probably similar to the percentages to the IDC/Hitachi Vantara reports), but it appeared to be shunned at this juncture. In web scale applications, Redhat Ceph block and files appeared popular in contrast to Openstack Swift. In enterprise applications, it was a toss of iSCSI and NFS.

Image from https://zdnet4.cbsistatic.com/hub/i/r/2018/04/24/c79e9dfb-b4a9-46bb-b831-f2c57fdf8a1d/resize/470xauto/5e4846d1bc7a034c382baf6dcbb612ed/cloud-storage.jpg

Continue reading →

It’s all about executing the story

By cfheoh | March 22, 2013 - 7:33 am |March 22, 2013 Data, Dell, EMC, Fujitsu, HP, IBM, IDC, NetApp, Oracle, Storage Market Share

4 Comments

I have been in hibernation mode, with a bit of “writer’s block”.

I woke up in Bangalore in India at 3am, not having adjusted myself to the local timezone. Plenty of things were on my mind but I can’t help thinking about what’s happening in the enterprise storage market after the Gartner Worldwide External Controller-Based report for 4Q12 came out last night. Below is the consolidated table from Gartner:

Just a few weeks ago, it was IDC with its Worldwide Disk Storage Tracker and below is their table as well:

Continue reading →

Is there no one to challenge EMC?

By cfheoh | December 11, 2012 - 6:36 am |December 11, 2012 Data, Dell, EMC, Fibre Channel, Fujitsu, HDS, HP, IBM, IDC, iSCSI, NetApp, NFS, Object Storage, Oracle, Storage Market Share, Unified Storage

5 Comments

It’s been a busy, busy month for me.

And when the IDC Worldwide Quarterly Disk Storage Systems Tracker for 3Q12 came out last week, I was reading in awe how impressive EMC was at the figures that came out. But most impressive of all is how the storage market continue to grow despite very challenging and uncertain business conditions. With the Eurozone crisis, China experiencing lower economic growth numbers and the uncertainty in the US economic sectors, it is unbelievable that the storage market grew 24.4% y-o-y. And for the first time, 7,104PB was shipped! Yes folks, more than 7 exabytes was shipped during that period!

In the Top 5 external disk storage market based on revenue, only EMC and HDS recorded respectable growth, recording 8.7% and 13.8% respectively. NetApp, my “little engine that could” seems to be running out of steam, earning only 0.9% growth. The rest of the field, IBM and HP, recorded negative growth. Here’s a look at the Top 5 and the rest of the pack:

HP -11% decline is shocking to me, and given the woes after woes that HP has been experiencing, HP has not seen the bottom yet. Let’s hope that the new slew of HP storage products and technologies announced at HP Discover 2012 will lift them up. It also looked like a total rebranding of the HP storage products as well, with a big play on the word “Store”. They have names like StoreOnce, StoreServ, StoreAll, StoreVirtual, StoreEasy and perhaps more coming.

The Open SAN market, which includes iSCSI has EMC again at Number 1, with 29.8%, followed by IBM (14%), HDS (12.2%) and HP (11.8%). When combined with NAS numbers, the NAS + Open SAN market, EMC has 33.5% while NetApp is 13.7%.

Of course, it is just not about external storage because the direct-attached storage numbers count too. With that, the server vendors of IBM, HP and Dell are still placed behind EMC. Here’s a look at that table from IDC:

There’s a highlight of Dell in the table above. Dell actually grew by 4.0% compared to decline in HP and IBM, gaining 0.1%. However, their numbers seem too tepid and led to the exit of Darren Thomas, Dell’s storage group head honco. News of Darren’s exit was on TheRegister.

I also want to note that NAS growth numbers actually outpaced Open SAN numbers including iSCSI.

This leads me to say that there is a dire need for NAS technical and technology expertise in the local storage market. As the adoption of NFSv4 under way and SMB 2.0 and 3.0 coming into the picture, I urge all storage networking professionals who are more pro-SAN to step out of their comfort zone and look into NAS as well. The world is changing and it is no longer SAN vs NAS anymore. And NFSv4.1 is blurring the lines even more with the concepts of layout.

But back to the subject to storage market, is there no one out there challenging EMC in a big way? NetApp was, some years ago, recorded double digit growth and challenging EMC neck-and-neck, but that mantle seems to be taken over by HDS. But both are long way to go to get close to EMC.

Kudos to the EMC team for damn good execution!

The reports are out!

By cfheoh | June 10, 2012 - 7:43 am |June 22, 2012 Apple, Dell, EMC, Gartner, HP, IBM, IDC, NetApp, Oracle, Steve Jobs, Storage Market Share, Unified Storage

3 Comments

It’s another quarter and both Gartner and IDC reports on disk storage market are out.

What does it take to slow down EMC, who is like a behemoth beast mowing down its competition? EMC, has again tops both the charts. IDC Worldwide Disk Storage Tracker for Q1 of 2012 puts EMC at 29.0% of the market share, followed by NetApp at 14.1%, and IBM at 11.4%. In fourth place is HP with 10.2% and HDS is placed fifth with 9.4%.

In the Gartner report, EMC has the lead of 32.5%, followed by NetApp at 12.7% and IBM with 11.0%. HDS held fourth place at 9.5% and HP is fifth with 9.0%. Continue reading →

IDC 4Q11 Tracker numbers are in

By cfheoh | March 2, 2012 - 11:22 pm |March 3, 2012 IDC, Storage Market Share

2 Comments

It was a challenging 2011 but the tremendous growth of data continues to spur the growth of storage. According to IDC in its latest Worldwide Quarterly Disk Storage Systems Tracker, the storage market grew a healthy 7.7% in factory revenues, and the total disk storage capacity shipped was 6,279 petabytes, up 22.4% year-on-year! What Greg Schulz once said was absolutely true. “There is no recession in storage”

Let’s look at the numbers. Overall, the positions of the storage vendors did not change much, but to me, the more exciting part is the growth quarter over quarter.

EMC and NetApp continue to post double digit growth perennially, with 25.9% and 16.6% respectively. Once again, taking market share from HP and others. HP, with the upheaval that is going on right now throughout the organization, got hit badly with a decline of 3.8%, while IBM held ground of 0.0%. And a data growth of 0.0% when the data growth is at more than 20% is not good, not good at all.

HDS, continuing its momentum with a good story, took a decent 11.6% and a fantastic number from HDS’s perspective. I have been out with my HDS buddies and I can feel their excitement and energy that I have never seen before. And that is a good indicator of the innovation and new technology that is coming out from HDS. They just need to work on their marketing and tell the industry more about what they are doing. Japanese can be so modest.

From the report, 2 things peeved me.

IDC reports that NetApp and HP are *tied* at 3rd. This does not make any sense at all! How can they be tied when NetApp has double digit growth, 11.2% market share and a revenue of USD$734 million while HP has negative growth, 10.3% market share and USD$677 million revenue? The logic boggles my mind!
They lumped Dell and Oracle into others! And others had a -1.4% growth. I am eager to find out how these 2 companies are doing, especially Dell who has been touting superb growth with their storage story.

Meanwhile, in TOTAL, here’s the table for the Total Worldwide Disk Storage Market share for 4Q11.

Numbers don’t lie. HP and IBM, in both 2nd and 3rd place respectively, are not in good shape. Negative growth in an upward trending market spells more trouble in the long run, and they had better buck up.

In this table, Dell gained and went up to #4 ahead of NetApp and from the look of things, Dell is doing all the right things to make sure that their storage market story is gelling together. Kudos! In fact, NetApp’s position and perception in the last 2-3 quarters have been shaky with Dell and HDS breathing down its neck. There isn’t likely to be significant dent to NetApp by HDS or Dell at this point in time, but having been the “the little engine that could” (that’s what I used to call NetApp) for the last few years, NetApp seems to be losing a bit of the extra “ooomph” that has excited the market in the past.

Lastly, I just want to say that my comments are based on the facts and figures in the tables published by IDC. I remember the last time I commented with the same approach, buddies of mine in the industry disagreed with me, saying that each of them are doing great in the Malaysian or South East Asian market.

Sorry guys, I blog I was see and I welcome you to take me to your sessions to let me know how well you are doing here. I would be glad to write more about it. (Hint, hint).

IDC Worldwide Storage Software QView 3Q11

By cfheoh | December 24, 2011 - 8:37 am |October 27, 2012 EMC, IBM, IDC, Storage Market Share, Uncategorized

Category Archives: IDC