AoE – All about Ethernet!

This is long overdue.

A reader of my blog asked if I could do a piece on Coraid. Coraid who?

This name is probably a name not many people heard of in Malaysia. Even most the storage guys that I talk to never heard of it.

I have known about Coraid for a few years now (thanks to my incessant reading habits), looking at it from nonchalant point of view.  But when the reader asked about Coraid, I contacted Kevin Brown, CEO of Coraid, whom I am not exactly sure how I was connected through LinkedIn. Kevin was very responsive and got one of their Directors to contact me. Kaushik Shirhatti was his name and he was very passionate to share their Coraid technology with me. Thanks Kevin and Kaushik!

That was months ago but the thought of writing this blog post has been lingering. I had to scratch the itch. 😉

So, what’s up with Coraid? I can tell that they are different but seems to me that their entire storage architecture is so simple that it takes a bit of time for even storage guys to wrap their head around it. Why do I say that?

For storage guys (like me), we are used to layers. One of the memorable movie quotes I recalled was from Shrek: “Orges are like onions! Onions have layers!“.

When we dissect the storage networking topology, we have the RAID layer, the abstraction into LUNs layer, the network layer, the session layer, the zoning layer, the masking and mapping layers, and so on, and so forth. You know what I mean. Those are the constructs of a typical SAN architecture. When iSCSI came along more than a decade ago, that layering gotten thicker with more layers, but somehow the customers adopted it. iSCSI, love it or loathe it, has been eating up the Fibre Channel SAN market in the past few years.

Why? I believe the familiarity of “local disk” kept the customers feeling good, within their comfort zone. “It looks and feels like a D: drive”. iSCSI runs on the Ethernet network, not that complex Fibre Channel stuff.

And this is helped by applications such as databases which want and demand local ownership of the storage space. So, the local disk phenomenon continues, despite our usual grouches and complaints about iSCSI performance and security, among others.

But those layers bothers us. Between the compute layer and the storage layer, there are just too many “layers” of ensuring data delivery, consistency, protection, etc, etc. The storage layer is too far apart from the compute layer and in this era of virtualization and cloud, the storage layer must be as close as possible to the compute layer. No wonder we storage guys get blamed for not delivering fast I/O to the compute layer. We have too many layers in between. Maybe we should rope in the network guys to be blamed as well because that’s the layer in between!

No wonder some vendors (VMware included with its VSA architecture) are beginning to reignite the DAS (direct-attached storage) architecture again, coupling it with scale-out and global namespace features.

Well, Coraid is a very good fit into the requirements SAN guys might love. Here are a few I can think of:

• Lesser layers resulting in higher data delivery throughput and lower latency in response time
• Higher security because IP (Internet Protocol) is less involved. We all know the proliferation of IP, but it is also where the exposure to routing that is the bane of security concerns when it comes to networked storage
• Removing traditional SAN implementation such as multipathing, zoning, firewalls etc will inadvertently reduce cost of managing the storage architecture
• Local disk flavour reduces fear and still dances within the customers’ comfort zone

AoE Ethernet type is 0x88A2.

The whole Ethernet SAN architecture, as prescribed by Coraid, is still pretty much the initiator and target concepts, with no architectural change to the Ethernet network. Coraid supplies the HBA (which I was informed is a reflashed Gigabit Ethernet NIC), the AoE driver (easy to install and available for Windows and Linux platforms), and the Coraid EtherDrive SRX storage array. The SRX supports SATA, SAS and SSD drives mixed within the appliance’s chassis.

We know well enough we should never bet against Ethernet, so the future looks bright as the Ethernet speed is moving towards 40Gbit/sec and 100Gbit/sec mainstream is not too far away.

There is no need for multipathing as the AoE driver at the host and OS level installs the AoE mapping layer, which essentially dices all data blocks into 8KB chunks and tracks the sending, the acknowledgment and the receiving of these chunks between the AoE HBA and the SRX storage array. It uses “configuration strings” to ensure consistency and implements a “host-based cooperative locking“.

According to the description from Wikipedia, when more than 1 initiator is accessing the AoE target (the SRX appliance), the AoE target acts as an arbitrator to record and keeps track of which host is accessing the storage blocks at a particular point in time. If there is a conflict, only one access is accepted and the other host is informed of the conflict. The AoE mapping layer will then handle the retransmit if required.

Each segmented 8KB blocks are sent over multiple Ethernet frames, through all available AoE HBA ports as a unicast link. This method ensures the highest possible I/O throughput to the SRX storage array. Coraid claims that their SRX appliance can deliver up to 1,800 MB/sec throughput and 200,000 IOPS in a single 4U appliance.

But my reader wanted to know about Coraid replication technology. Coraid implements remote data copy (which can be viewed as a replication or mirroring technology) in 2 possible ways (excluding host-based replication/mirroring) at their technology’s level.

One of the methods is to introduce the Coraid EtherDrive VSX. Here’s a high level look at the VSX architecture implemented with SRX (courtesy of Coraid):

The EtherDrive VSX appliance integrates multiple EtherDrive SRX appliances and allows block-level synchronous mirroring and asynchronous remote replication. In addition to that, the VSX is also responsible for implementing storage virtualization, snapshots, cloning and thin provisioning.

Hence, their architecture is rather different than other more traditional storage vendors where all these technologies such as replication, snapshots and thin provisioning are all integrated into their storage array. By separating the VSX and the SRX, I can only assume that this will give Coraid the ability to adapt their architecture in a much more flexible manner, leaving the EtherDrive SRX appliance to do what it does best – which is to deliver local drive storage in the high performing manner with the lowest latency through the Ethernet architecture.

The world is changing and the demands of delivering unified storage is on everybody’s lips. So, in July 2012, if I am not wrong about the date, Coraid has licensed ZFS to deliver high-performance NAS with their Coraid ZX appliance. Here’s a high level diagram I got from Coraid’s website.

As you know, I am big fan of ZFS. (read my blog post here). I think by introducing ZFS into their offering, Coraid has completed their portfolio of storage solutions. Of course, ZFS also adds another data replication method. This is the second way that Coraid can deliver remote replication in a disaster recovery situation. Kaushik also shared this slide with me:

Coupled with the acquisition of Yunteq in late 2011, Coraid is integrating their Storage Manager management interface into EtherCloud, their all-encompassing storage management interface. According to IDC, in which I quote:

“Coraid EtherCloud is designed to make every aspect of storage management and
control programmable using a REST API. Using the API, administrators can easily
automate workflows related to storage provisioning and management, and integrate
the management of storage with the rest of the infrastructure.

EtherCloud consists of a policy engine that allows storage to be dynamically allocated
and managed according to application requirements and business policies with endto-
end visibility and control. This allows IT administrators to reduce infrastructure
provisioning times significantly and provide a programmable platform to integrate
storage resources with networking and computing resources. EtherCloud also allows
application owners to deploy storage with one-click provisioning in a multitenant
environment without having to deal with the complexity of underlying storage
operations.”

The overall Coraid architecture is shown below:

Coraid is indeed evolving moving into quite a complete storage solution. As Kevin Brown, Coraid CEO once quoted “We use raw Ethernet for a SAN”.

Indeed AoE is All About Ethernet.

p/s: I just want it clear that I am not paid write this. I do not owe any favour to anyone but I wanted to share with my readers (especially in Malaysia) that there is an interesting storage technology and we should not fear learning about it.